{nagham.omar, rmaya, ym}@campus.technion.ac.il

avigal@technion.ac.il

Abstract

Training strategy, namely whether to retrain from scratch or fine-tune from the previous checkpoint, is an overlooked decision variable in active learning. We show that this choice has exploitable structure: retraining is most useful in early rounds, when each batch can substantially reshape the labeled distribution, while fine-tuning becomes safer once the model trajectory stabilizes. We propose HybridAL, an adaptive training schedule that monitors an online stabilization signal and switches from retraining to fine-tuning after sustained stabilization. Two complementary signals, spectral exponent change $\Delta \alpha$ (weight-based) and accuracy change $\Delta$Acc (validation-based), span different points on the time–calibration trade-off. Across three encoder backbones and six text-classification tasks (five seeds each), HybridAL keeps endpoint macro-F1 non-inferior to retraining and fine-tuning at a $0.010$ margin, saves up to $49\%$ of retraining time, and recovers a substantial fraction of retraining’s calibration advantage as measured by negative log-likelihood (NLL). Compared with schedules that switch at a pre-committed round, HybridAL obtains lower NLL at moderate additional cost, showing that trajectory-dependent switching provides a stronger time--calibration trade-off than fixed early switching¹¹ 1 The implementation is available at: https://github.com/naghamo/hybridAL.

Accepted to EMNLP 2026 Main Conference.

1 Introduction

媒体内容 · 前往原文查看

Active learning (AL) reduces annotation costs by iteratively selecting informative unlabeled examples for labeling, rather than annotating a large dataset in a single pass (Settles, 2009). Large language models (LLMs) have been proposed as scalable annotators and judges across NLP and beyond (Tan et al., 2024; Li et al., 2025), and surpass crowd workers on some annotation tasks (Gilardi et al., 2023). However, their annotations remain task-dependent and can exhibit systematic biases (Chen et al., 2024; Ashktorab et al., 2025; Calderon et al., 2025; Szymanski et al., 2025). These limitations mean that AL remains necessary for deciding which examples to annotate under a limited budget (Ren et al., 2021). At the same time, as LLM-based annotation reduces per-sample cost, updating the model each round becomes the dominant bottleneck in the AL loop, a cost that grows with each acquisition step (Scala et al., 2025); as practitioners can afford more rounds, training-time savings become increasingly valuable.

While much AL research optimizes which samples to acquire  (Settles, 2009; Ash et al., 2019), the choice of how to update the model after each round remains underexplored (Munagala et al., 2022). Two strategies dominate practice: Retrain, which reinitializes from the original pre-trained weights (or from random initialization when no pre-trained backbone is used) and trains on all accumulated labeled data, and FineTune, which continues from the previous checkpoint. Neither is uniformly preferable. Retrain is robust but computationally redundant as the model matures, whereas FineTune is efficient but can suffer from warm-starting degradation in early, high-variance acquisition rounds (Ash and Adams, 2020). This suggests a natural asymmetry: early rounds benefit from retraining, while later rounds can often be handled by fine-tuning once the model trajectory stabilizes.

We address this gap with HybridAL, an adaptive training schedule for pool-based AL that switches from full retraining to incremental fine-tuning after sustained stabilization. The motivation is illustrated in Figure 1: Retrain produces well-calibrated predictions, as reflected by low test negative log-likelihood (NLL), but is slow; FineTune is efficient but incurs a calibration penalty. This penalty matters because uncertainty-based acquisition functions, the most widely used family in AL (Settles, 2009; Ren et al., 2021), rank candidates by predicted probabilities, so probability quality during training may affect which examples are queried. We treat this as motivation for retraining early, not as an established effect. No single strategy dominates both speed and calibration. HybridAL therefore monitors a switching signal after each round and switches when the model trajectory enters a low-change regime.

A central question is which signal best detects stabilization. We evaluate eight candidates spanning performance-based metrics (e.g., accuracy change) and model-based metrics (e.g., spectral exponent change (Martin and Mahoney, 2021) and representation similarity (Kornblith et al., 2019)). Two signals emerge as complementary operating points: the spectral exponent change ($\Delta \alpha$), a weight-based signal that favors time savings, and the validation accuracy change ($\Delta$Acc), which favors calibration. Both maintain comparable final F1.

1. We identify training strategy as an overlooked decision variable in AL and propose HybridAL (Algorithm 1), an adaptive schedule that switches from Retrain to FineTune once stabilization is detected.

2. We introduce stabilization detection (Definition 1), a general online criterion over model-trajectory signals, and identify two complementary signals: $\Delta \alpha$ (weight-based, fastest, no additional validation pass) and $\Delta$Acc (validation-based, best calibration).

3. We evaluate HybridAL across three encoder backbones and six text-classification tasks (5 seeds each) and show that endpoint F1 is non-inferior to both single-strategy baselines at a $0.010$ margin, roughly three quarters of the seed-to-seed standard deviation (two one-sided tests, TOST), that it saves up to $49\%$ of retraining time, and that it achieves a stronger time–calibration trade-off than schedules that switch at a pre-committed round, establishing that adaptive timing, not switching itself, drives the calibration gain.

2 Related Work

Existing training regimes. Retraining a model from scratch each round is often recommended for robust generalization, though it remains computationally expensive (Beck et al., 2021). Conversely, fine-tuning is computationally efficient but frequently degrades generalization due to warm-start bias (Ash and Adams, 2020). While some configurations attempt to train exclusively on newly acquired data, this strategy risks catastrophic forgetting (Munagala et al., 2022; Das et al., 2023) unless mitigated by replay-based methods that interleave a small buffer of previously labeled examples during updates (Rolnick et al., 2019).

Despite these trade-offs, existing active learning pipelines apply a single training strategy uniformly across all selection rounds, ignoring a fundamental asymmetry: early rounds operate in a high-information, high-variance regime where each batch drastically reshapes the data distribution. Fine-tuning prematurely in this phase induces a severe loss of plasticity, permanently degrading the network’s capacity to absorb new concepts (Dohare et al., 2024). Conversely, later rounds provide only marginal refinements to an already-stable model. Once a model’s internal representations geometrically mature and stabilize, phenomena observable via spectral self-regularization (Martin and Mahoney, 2021) and neural collapse (Papyan et al., 2020), fine-tuning becomes safer and more efficient.

Adaptive methods and motivation for performance-based signals. Prior AL efficiency literature focuses mainly on other aspects of the pipeline. Recent advancements have introduced adaptive frameworks that dynamically switch between acquisition strategies mid-process, using multi-armed bandits (Zhang et al., 2023), deep imitation learning (Liu et al., 2018), or budget-aware heuristics that transition from typicality to uncertainty sampling as the labeled pool grows (Hacohen et al., 2022; Hacohen and Weinshall, 2023). Other works utilize dynamic performance signals to alter the AL pipeline mid-stream. For instance, performance plateaus and confidence metrics are frequently used as stopping criteria to terminate the AL loop (Vlachos, 2008; Zhu et al., 2008). Similarly, some work has used performance deltas as reward signals for reinforcement learning-based acquisition (Fang et al., 2017), and in stream-based AL, concept drift has been used to trigger model ensemble updates (Han et al., 2024). Current Green AI frameworks borrow AL-inspired iterative sampling and utilize adaptive performance signals, such as tracking loss stagnation to dynamically trigger shifts in the training regimen, to reduce computational costs on already fully labeled datasets (Scala et al., 2024; Scala et al., 2025). While the latter methods alter training to facilitate data pruning when all labels are available, they do not inherently operate in an environment where labels are acquired iteratively.

To our knowledge, prior pool-based AL work has not treated the choice between Retrain and FineTune as an online decision variable. HybridAL targets this gap by adapting the training strategy while keeping the acquisition protocol fixed.

3 HybridAL: Adaptive Training Strategy Switching

We present HybridAL, an adaptive training method for pool-based AL that switches from full retraining to incremental fine-tuning by detecting when the model has stabilized. We define the switching problem (§3.1), introduce a stabilization detection mechanism (§3.2), and present the complete algorithm (§3.3).

3.1 Model & Problem Definition

Let $𝒟={\left\{(x_i,y_i)\right\}}_{i=1}^N$ denote a dataset over input space $𝒳$ and label space $𝒴=\left\{1,\dots ,C\right\}$ for $C$-class classification. We consider a standard pool-based AL setting (Settles, 2009) with initial labeled and unlabeled pools ${ℒ}_0$ and ${𝒰}_0$. AL proceeds for $T$ rounds under a fixed labeling budget $\left|{ℒ}_0\right|+nT$, where $n$ is the acquisition batch size. At each round $t$, a model $f_{{\theta }_t}:𝒳\to 𝒴$ with parameters ${\theta }_t$ is trained on ${ℒ}_{t-1}$ according to a strategy $s_t$. The model then selects a batch ${𝒬}_t\subset {𝒰}_{t-1}$ of size $n$ via an acquisition function. The pools are then updated as ${ℒ}_t={ℒ}_{t-1}\cup {𝒬}_t$ and ${𝒰}_t={𝒰}_{t-1}\setminus {𝒬}_t$.

In this work, we address the challenge of choosing $s_t$ as a function of history up to round $t$. Standard paradigms typically restrict $s_t$ to a constant strategy across all rounds. Retrain reinitializes from pre-trained weights and trains on all of ${ℒ}_{t-1}$, producing robust generalization (Ash and Adams, 2020; Beck et al., 2021) at growing cumulative cost. FineTune continues from ${\theta }_{t-1}$, reducing per-round cost through warm-starting, but inheriting biases from previous checkpoints that can degrade generalization (Ash and Adams, 2020), particularly in early rounds (Beck et al., 2021). A third approach (which we do not consider in our solution but include as a baseline) is NewOnly, which trains only on the newly acquired batch, discarding historical data and risking catastrophic forgetting (Munagala et al., 2022; Das et al., 2023).

To combine the early-stage robustness of Retrain with the late-stage efficiency of FineTune, we consider schedules that switch once from Retrain to FineTune. We first state the switching objective as an offline problem, then explain why it must be approximated online.

Problem 1(Training Strategy Switching).

Find a switching point $t^{\ast }\in \left\{1,\dots ,T+1\right\}$ that defines

$s_{\tau }\left(t^{\ast }\right)=\{\begin{array}{cc}\text{Retrain}& \text{if }\tau <t^{\ast },\\ \text{FineTune}& \text{if }\tau \ge t^{\ast }.\end{array}$

(1)

Here $t^{\ast }=T+1$ recovers pure Retrain, and $t^{\ast }=1$ recovers pure FineTune. The ideal switch improves the time–calibration trade-off while preserving endpoint classification performance:

$\underset{t\in \left\{1,\dots ,T+1\right\}}{\min }\mathrm{Time}\left(t\right)+\lambda \mathrm{Calib}\left(f_{{\theta }_T}^{\left(t\right)}\right)$

(2)

subject to

$\mathrm{Perf}\left(f_{{\theta }_T}^{\left(t\right)}\right)\ge \max ({\mathrm{Perf}}_{\text{Retrain}},{\mathrm{Perf}}_{\text{FineTune}})-\delta ,$

(3)

where $f_{{\theta }_T}^{\left(t\right)}$ is the final model obtained by switching at round $t$, $\mathrm{Time}\left(t\right)$ is the cumulative training time across all $T$ rounds under switch point $t$, $\mathrm{Calib}(\cdot )$ is a calibration error measure, $\mathrm{Perf}(\cdot )$ is a task performance metric (e.g., macro-F1), $\lambda \ge 0$ controls the time–calibration trade-off, and $\delta \ge 0$ is an allowed performance tolerance.

Problem 1 depends on endpoint quantities that are known only in post-hoc analysis. Evaluating $\mathrm{Perf}\left(f_{{\theta }_T}^{\left(t\right)}\right)$, $\mathrm{Calib}\left(f_{{\theta }_T}^{\left(t\right)}\right)$, or even $\mathrm{Time}\left(t\right)$ for a candidate switch point $t$ would require running the full $T$-round AL loop under that choice. HybridAL therefore approximates this objective online using the stabilization criterion in Definition 1 as a tractable proxy.

3.2 Stabilization Detection as a Switching Signal

Early rounds operate with small pools where each batch constitutes a $n/\left|{ℒ}_t\right|$ distributional shift; in this regime, warm-starting degrades generalization (Ash and Adams, 2020), and the penalty compounds across rounds. As the pool grows, the per-round shift shrinks, checkpoint quality improves, and the gap between strategies vanishes. This asymmetry motivates switching from Retrain to FineTune, rather than the reverse. The remaining question is when.

The stabilization hypothesis.

AL exhibits a regime transition: learning dynamics shift from rapid exploration (high information gain, large distributional shifts, unstable representations) to gradual refinement (diminishing returns, converged representations). The transition point varies by task and dataset complexity, so a fixed switching round cannot suit all settings. We formalize the detection of this transition as follows.

Definition 1(Stabilization Point).

Let $S\left(f_{{\theta }_t}\right)$ denote a switching signal evaluated after round $t$. The signal change is $\Delta S_t=\left|S\left(f_{{\theta }_t}\right)-S\left(f_{{\theta }_{t-1}}\right)\right|$ for $t\ge 1$. The stabilization point is the earliest round $t^{\ast }$ at which $\Delta S_t$ remains below threshold $\epsilon$ for $k$ consecutive rounds:

$t^{\ast }=\min \left\{t\ge k\right|\underset{i\in \left[t-k+1,t\right]}{\max }\Delta S_i<\epsilon \}$

(4)

where $\epsilon >0$ is the sensitivity threshold and $k\ge 1$ the patience parameter.

The threshold $\epsilon$ controls how much round-to-round change is tolerated before declaring stabilization: smaller values require the signal to flatten more before switching.

When $\Delta S_t$ is large, the model trajectory is still changing substantially and retraining remains safer. When $\Delta S_t$ stays below $\epsilon$, the trajectory has entered a low-change regime: the warm-starting penalty is less likely to dominate, and fine-tuning becomes a more efficient update. Crucially, $\Delta S_t$ is a relative measure of change, not an absolute performance level, making it less sensitive to task difficulty.

The role of patience.

A single low $\Delta S_t$ may result from noise: an uninformative batch, a temporary plateau, or class sampling imbalance (Ren et al., 2021). The patience parameter $k$ requires $k$ consecutive sub-threshold rounds before switching, filtering transient fluctuations. This corresponds exactly to the ${\max }_{i\in \left[t-k+1,t\right]}\Delta S_i<\epsilon$ condition: a single excursion above $\epsilon$ within the window resets the counter. HybridAL thus monitors the signal online and switches only when stabilization is confirmed, adapting to each task’s trajectory.

3.3 The HybridAL Algorithm

Algorithm 1 presents the complete procedure with three state variables: current strategy $s$, stabilization counter stable_count, and previous signal value $S_{\text{prev}}$.

媒体内容 · 前往原文查看

0:  Unlabeled pool ${𝒰}_0$, labeled pool ${ℒ}_0$, validation set $𝒱$, rounds $T$, batch size $n$, acquisition function $A$, threshold $\epsilon$, patience $k$

0:  Final model $f_{{\theta }_T}$

1:  $s\leftarrow \text{Retrain}$;  $\text{stable\_count}\leftarrow 0$;  $S_{\text{prev}}\leftarrow 0$

2:  for $t=1$ to $T$ do

3:   if $s=\text{Retrain}$ then

4:    $f_{{\theta }_t}\leftarrow \text{TrainFromScratch}\left({ℒ}_{t-1}\right)$

5:   else

6:    $f_{{\theta }_t}\leftarrow \text{FineTune}(f_{{\theta }_{t-1}},{ℒ}_{t-1})$

7:   end if

8:   ${𝒬}_t\leftarrow A(f_{{\theta }_t},{𝒰}_{t-1},n)$

9:   ${ℒ}_t\leftarrow {ℒ}_{t-1}\cup {𝒬}_t$;  ${𝒰}_t\leftarrow {𝒰}_{t-1}\setminus {𝒬}_t$

10:   $S_{\text{curr}}\leftarrow S\left(f_{{\theta }_t}\right)$

11:   $\Delta S_t\leftarrow \left|S_{\text{curr}}-S_{\text{prev}}\right|$;  $S_{\text{prev}}\leftarrow S_{\text{curr}}$

12:   if $\Delta S_t<\epsilon$ then

13:    $\text{stable\_count}\leftarrow \text{stable\_count}+1$

14:    if $\text{stable\_count}\ge k$ and $s=\text{Retrain}$ then

15:     $s\leftarrow \text{FineTune}$ {Permanent switch}

16:    end if

17:   else

18:    $\text{stable\_count}\leftarrow 0$

19:   end if

20:  end for

21:  return $f_{{\theta }_T}$

At each round, HybridAL trains $f_{{\theta }_t}$ according to the current strategy (Lines 3–6), selects and labels a batch (Lines 7–8), and computes the switching signal and its change (Lines 9–10). Performance-based signals (e.g., $\Delta$Acc) require a forward pass on $𝒱$. Model-based signals (e.g., $\Delta \alpha$) are computed directly from the weights. The algorithm permanently switches to FineTune after $k$ consecutive rounds with $\Delta S_t<\epsilon$ (Line 14).

Key properties.

The switch is irreversible: once $s_t=\text{FineTune}$, the algorithm never reverts, ensuring monotonically decreasing per-round cost. This design is deliberate: after switching, the model is updated via warm-starting rather than re-initialization, which changes the optimization dynamics. Under these new dynamics, signal values can fluctuate even when the model remains performant. A reversible variant would misinterpret such fluctuations as instability and repeatedly revert to Retrain, losing the cost guarantee without improving endpoint performance (Appendix B.4).

Stabilization of classification decisions does not guarantee stabilization of the full probability distribution; HybridAL mitigates calibration drift by retraining during the early rounds, when probability estimates are most sensitive to the training data composition. This motivates the design, since uncertainty-based acquisition ranks candidates by predicted probabilities and post-hoc recalibration (Guo et al., 2017) applies only to the final model, leaving acquisition decisions already made during training unchanged (Appendix B.5). Empirically, acquisition quality is maintained after the switch (Appendix D); whether improved calibration yields better acquisition utility remains open.

The time savings arise because FineTune converges in fewer epochs than Retrain under early stopping. The validation set $𝒱$ is held fixed, shared by all strategies, and does not consume labeling budget. HybridAL introduces two hyperparameters: $\epsilon$ and $k$, whose selection is described in §4.

4 Experiments

We evaluate HybridAL against single-strategy baselines and non-adaptive schedules that switch at a fixed pre-committed round, across three encoder backbones and six text-classification datasets spanning binary and multi-class regimes, with five seeds per cell. After describing the protocol (§4.1), the remainder of this section establishes three claims:

• HybridAL is non-inferior to both single-strategy baselines across backbones and task difficulties (§4.2).

• HybridAL retains most of Retrain’s calibration while capturing the bulk of FineTune’s training-time savings (§4.3).

• HybridAL achieves a stronger time–calibration trade-off than fixed early-switch schedules by adapting the switch point to the model trajectory (§4.4).

4.1 Experimental Setup

Datasets.

We use six English text-classification benchmarks (Table 1): three binary (IMDb, Jigsaw, SST-2) and three multi-class (TweetEval, AG News, Yahoo Answers). Yahoo Answers is stratified-downsampled to $60$k (uniform across classes). Per-split sizes are in Appendix A.

媒体内容 · 前往原文查看

Dataset	Domain	C	Size
Binary
IMDb (Maas et al., 2011)	sentiment analysis	2	50,000
SST-2 (Socher et al., 2013)	sentiment analysis	2	68,221
Jigsaw (Wulczyn et al., 2017)	toxicity detection	2	159,571
Multi-class
TweetEval (Barbieri et al., 2020)	sentiment analysis	3	59,899
AG News (Zhang et al., 2015)	topic classification	4	127,600
Yahoo (Zhang et al., 2015)	topic (Q&A)	10	60,000

Models.

We evaluate DistilBERT (Sanh et al., 2019) ($\sim$66M), BERT-base (Devlin et al., 2019) ($\sim$110M), and RoBERTa-base (Liu et al., 2019) ($\sim$125M). DistilBERT is the default for ablations; all three appear in the main results.

Active learning protocol.

Each run starts from a class-stratified pool of $\left|{ℒ}_0\right|=200$ and proceeds for $T=25$ rounds, acquiring $n=32$ examples per round (final budget $\mathrm{1,000}$). A stratified per-dataset validation set $𝒱$ ($477$ to $\mathrm{1,596}$ labels) is held fixed across rounds, separate from the labeling budget and AL pools, and is shared by all methods for early stopping and per-round evaluation. Validation-label assumptions and a size-sensitivity study are in Appendix C. Each configuration is repeated over 5 seeds ($42$–$46$). Sensitivity to $\left|{ℒ}_0\right|$, $n$, and the entropy pre-filter size $N=\mathrm{1,000}$ is in Appendix G.

Acquisition functions.

Entropy (Settles, 2009) is the default; ablations with Random and BADGE (Ash et al., 2019) are in Appendix G.5.

Switching signals.

We evaluate eight signals capturing round-to-round model change (Table 2): four performance-based, measured on $𝒱$: macro-F1 change $\Delta$F1, accuracy change $\Delta$Acc, cross-entropy change $\Delta$Loss, and gradient ${\ell }_2$ norm; and four model-based: spectral exponent change $\Delta \alpha$ (Martin and Mahoney, 2021), the mean power-law tail exponent of each layer’s eigenvalue spectrum differenced between rounds; ${\ell }_2$ weight distance ${\Vert {\theta }_t-{\theta }_{t-1}\Vert }_2$; representational similarity $1-\mathrm{CKA}$ (Kornblith et al., 2019); and within-class feature concentration change $\Delta$NC (Papyan et al., 2020). Of the model-based signals, $\Delta \alpha$ and weight distance operate on weight matrices alone; CKA and $\Delta$NC require a forward pass on $𝒱$ to extract representations. All strategies use $𝒱$ for early stopping and per-round evaluation regardless of signal choice; the distinction is whether the signal adds an extra pass each round. Based on a preliminary signal comparison on DistilBERT (Table 2), the main results use $\Delta \alpha$ and $\Delta$Acc: they have the two highest fire rates ($97\%$ and $93\%$), each is Pareto-optimal on (F1, time) within its family (model-based and performance-based, respectively), and the two are mutually uncorrelated ($\rho \approx 0$), capturing complementary information. Signal normalization and full per-signal results are in Appendix E.

媒体内容 · 前往原文查看

Signal	Type	Extra val. pass?	Fire rate	Mean $t^{\star }$
$\Delta$Acc⋆	performance	yes	93%	9.1
$\Delta$F1	performance	yes	90%	10.0
$\Delta$Loss	performance	yes	43%	13.4
Grad. norm	performance	yes	40%	8.5
$\Delta {\alpha }^{\star }$	model-based	no	97%	7.9
${\ell }_2$ distance	model-based	no	0%	✗
$\Delta$NC	model-based	yes	67%	12.4
CKA	model-based	yes	33%	5.4

Methods.

Three single-strategy baselines: Retrain, FineTune, and NewOnly (trains only on the new batch). Two HybridAL variants: $\Delta \alpha$ and $\Delta$Acc (Algorithm 1). Four non-adaptive ablations: FixedSwitch@$k$ for $k\in \left\{3,5,7,10\right\}$, spanning the range around the mean switch rounds of the two selected signals (Table 2), which switch unconditionally at a pre-committed round, isolating whether the gain comes from switching itself or from adaptive timing.

Hyperparameters.

We tune $(\epsilon ,k)$ on IMDb and AG News (one binary, one multi-class; 3 seeds, 15 rounds) by selecting the cell within $0.5\%$ of top validation F1 that minimises a normalised time–NLL score (Appendix F): $({\epsilon }^{\ast },k^{\ast })=({10}^{-4},3)$ for $\Delta \alpha$ and $(5\times {10}^{-3},2)$ for $\Delta$Acc, applied without retuning. The two-order-of-magnitude $\epsilon$ gap reflects different signal units, not sensitivity (F1 varies $\phantom{}\le 1.1$ percentage points (pp) across the grid).

Training and evaluation.

All strategies use AdamW (Loshchilov and Hutter, 2017) (lr$\phantom{}=2\times {10}^{-5}$, weight decay$\phantom{}={10}^{-3}$), batch size $16$, up to $10$ epochs with early stopping (patience $2$). FineTune converges in $3.4$ epochs on average vs. $5.5$ for Retrain (Appendix G.1). We report macro-F1, test NLL, and wall-clock time; for HybridAL variants we additionally report the mean switch round $t^{\star }$ and the switch rate. Significance is assessed via paired two-sided $t$-test at $\alpha =0.05$; non-inferiority is assessed by TOST. Experiments ran on two NVIDIA RTX 2080 Ti GPUs with PyTorch 2.6 (Paszke et al., 2019) and HuggingFace Transformers (Wolf et al., 2020). Full pairwise results are in Appendix B.

媒体内容 · 前往原文查看

媒体内容 · 前往原文查看

4.2 Preserving F1

Figure 2(a) shows endpoint test F1 distributions per task family. Both HybridAL variants remain close to Retrain and FineTune on every backbone: pooled across the six datasets, their means lie within $0.5$–$0.9$ pp of one another. We test this formally with TOST on the paired differences over all $90$ (backbone, dataset, seed) cells, anchoring the margin to Retrain’s mean seed-to-seed F1 standard deviation of $0.0131$. At $\delta =0.010$ both variants are non-inferior to Retrain and FineTune individually and to the per-cell better of the two; at $\delta =0.005$ three of the four hybrid–baseline pairs pass, the exception being $\Delta$Acc vs. FineTune ($p=0.051$). Per-cell tests and full bounds are in Appendix B.2. Thus, the main effect of the training schedule is not endpoint F1, but the time–calibration trade-off analyzed below. Pooled across all six datasets, NewOnly’s mean trails by $\phantom{}\approx 1.7$ pp on DistilBERT and $2.7$ pp on BERT relative to the strongest non-NewOnly method, with the deficit statistically significant on Yahoo Answers across all backbones (paired $t$-test, $p<0.05$; Appendix B.1). The gap concentrates on the hardest multi-class tasks relative to Retrain: TweetEval ($-8.2$ pp DistilBERT, $-5.8$ pp BERT) and Yahoo Answers ($-2.9$ pp DistilBERT, $-5.9$ pp BERT); on RoBERTa it vanishes ($\phantom{}\le 0.2$ pp pooled).

Figure 2(b) confirms these conclusions hold throughout training. Round-by-round, the mean validation F1 of Retrain, FineTune, HybridAL($\Delta \alpha$), and HybridAL($\Delta$Acc) is nearly identical from the first acquisition round onward; the insets (rounds 15–25) show the four curves stay within $\phantom{}\approx 1$ pp of each other at convergence. The two HybridAL variants therefore track the single-strategy baselines at every round, and F1 shows no inflection at HybridAL’s mean switch round $t^{\star }$ (stars), so the Retrain$\to$FineTune handoff does not disrupt learning. NewOnly, by contrast, converges visibly slower on both task families and especially on multi-class, and even at round $T$ remains below the other four methods.

媒体内容 · 前往原文查看

媒体内容 · 前往原文查看

4.3 HybridAL Trades Time for Calibration

Preserving F1 alone is insufficient: the entropy sampler ranks candidates by predicted probability, so calibration during the loop is what those decisions rest on. We therefore turn to the time–calibration trade-off, measured through NLL, which has a clear structure (Figure 3(a)). Retrain achieves the lowest test NLL on every backbone ($0.498$ RoBERTa to $0.532$ DistilBERT), but is the slowest method ($838$–$\mathrm{1,599}$ s). FineTune is $33$–$41$% faster, but incurs $44$–$47$% higher NLL. On DistilBERT and RoBERTa, both HybridAL variants lie between these extremes. On BERT, HybridAL($\Delta \alpha$) Pareto-dominates FineTune: it is faster ($819$ s vs. $936$ s) and better calibrated (NLL $0.710$ vs. $0.753$). Across all backbones, HybridAL($\Delta$Acc) saves $15$–$32$% of Retrain’s time at only $18$–$28$% higher NLL, reclaiming $39$–$59$% of FineTune’s raw NLL gap; HybridAL($\Delta \alpha$) saves $12$–$49$% of Retrain’s time at $32$–$36$% higher NLL. Thus, $\Delta$Acc is the safer default for calibration, while $\Delta \alpha$ favors speed. A temperature scaling analysis confirms that this ordering reflects training-time overconfidence inherited from warm-starting (Appendix B.5). Both variants lie on the substantive Pareto frontier, excluding NewOnly, a region not reached by any single-strategy pool-trained method.

Figure 3(b) shows that this ordering is stable throughout training: from approximately round $5$ onward, validation NLL follows Retrain $<$ HybridAL($\Delta$Acc) $<$ HybridAL($\Delta \alpha$) $<$ FineTune, with FineTune climbing late. NewOnly is the cheapest method and appears on the all-methods Pareto frontier, but it is not directly comparable to the pool-trained strategies: it discards the accumulated labeled set and trains only on the newest batch. This makes its endpoint NLL misleading for AL, since it begins as the worst-NLL method and only catches up late, so early acquisition decisions are made from poorly calibrated predictions. Together with its F1 deficit on hard multi-class tasks with smaller backbones (§4.2), this is why we exclude NewOnly from the substantive frontier.

4.4 Adaptive Switching

A natural question is whether HybridAL’s gains require adaptive timing or simply result from switching at any round. We compare against FixedSwitch schedules that switch unconditionally at round $k\in \left\{3,5,7,10\right\}$. Pooled mean F1 agrees within $0.010$ on every backbone across all methods (Appendix B.1); the difference lies in calibration. FixedSwitch commits to an early switch and is faster, but Figure 4 shows its NLL clusters near FineTune’s ($\phantom{}\approx 0.75$), while both HybridAL variants move closer to Retrain’s ($\phantom{}\approx 0.52$), with HybridAL($\Delta$Acc) the lowest non-Retrain method. HybridAL’s empirical switch lands later (mean $t^{\star }\approx 9$–$12$; range $3$–$25$ across cells), using the extra retraining rounds to improve calibration. The switch also varies by dataset (e.g., $t^{\star }\approx 6$ on TweetEval vs. $11$ on Yahoo Answers for $\Delta \alpha$), confirming that no fixed schedule reproduces this adaptation. The stabilization criterion (Definition 1) thus adapts to each task without per-dataset tuning, achieving a stronger time–calibration trade-off than any pre-committed schedule. Full per-dataset results are in Appendix B.

5 Conclusion

Training strategy is a decision variable that AL pipelines often leave fixed. We introduced HybridAL, an adaptive schedule that uses retraining while the model trajectory is changing rapidly and switches to fine-tuning after sustained stabilization. Across 9 methods, 3 encoder backbones, 6 text-classification benchmarks, and 5 seeds, HybridAL keeps endpoint test F1 non-inferior to the pool-trained baselines at a $0.010$ margin, substantially reduces wall-clock training time relative to full retraining, and recovers much of Retrain’s NLL-based calibration advantage. Compared with fixed early-switch schedules, HybridAL obtains a stronger time–calibration trade-off by adapting the switch point to the model trajectory, consistent with prior evidence that warm-starting costs are concentrated in early rounds (Ash and Adams, 2020; Beck et al., 2021). The two HybridAL variants offer complementary operating points: HybridAL($\Delta$Acc) favors lower NLL when a validation set is available, while HybridAL($\Delta \alpha$) favors speed and uses weight statistics without an additional validation forward pass. Both share the same switching mechanism, so practitioners can choose the signal according to the desired time–calibration trade-off.

Future work could switch to a damped per-round controller that selects the training strategy at each acquisition step while avoiding the oscillation of naïve reversibility (Appendix B.4). Replay-based extensions (Rolnick et al., 2019; Das et al., 2023) could also combine naturally with HybridAL’s switching logic, especially for stronger backbones where NewOnly’s F1 deficit already narrows.

Limitations

Irreversible switching.

HybridAL’s switch is irreversible: once it moves to FineTune, it cannot revert to Retrain even if the model later exhibits large round-over-round drift (e.g., from a batch introducing a previously rare class). The irreversibility guarantees monotonically decreasing per-round cost but bounds the calibration we can recover. Empirically, post-switch signal values frequently re-cross $\epsilon$ on every dataset (Appendix B.4), so a reversible variant would oscillate between strategies rather than settle. The irreversible design prevents this instability, but it means HybridAL cannot recover if a genuinely novel regime emerges late in training. Designing a damped or hysteresis-based reversal mechanism that avoids oscillation remains future work.

Residual calibration drift.

HybridAL leaves residual calibration drift relative to Retrain. Even HybridAL($\Delta$Acc) remains $18$–$28\%$ above Retrain’s NLL on every backbone (§4.3); applications that demand strict endpoint probability calibration (e.g., selective prediction with hard thresholds) may still require full retraining each round, potentially combined with post-hoc recalibration (Guo et al., 2017). However, for the common AL setting where calibration quality during acquisition matters most, HybridAL retains more of Retrain’s calibration through the early rounds (§4.3).

Acquisition-utility evidence is indirect.

We do not isolate the effect of calibration quality on acquisition utility. After the switch, the examples HybridAL acquires overlap little with those the Retrain run selects, yet class balance and endpoint F1 are maintained (Appendix D); this shows acquisition quality is not degraded, not that better calibration improves it. Isolating the effect would require refitting a temperature ${\tau }_t$ before acquisition at every round and comparing against native probabilities, which we leave to future work (Appendix B.5).

Structural lower bound of pool-based AL.

The same lower bound surfaces in NewOnly, which trains only on the newly acquired batch and is the cheapest method we test. NewOnly is well-calibrated at convergence but begins AL as the worst-calibrated method (§4.3). This is a structural property of pool-based AL: cumulative-data training is what supplies early-round calibration, and no schedule that omits it can match Retrain in the first several rounds. HybridAL stays in Retrain until stabilization, but it cannot remove the underlying constraint.

No formal stabilization guarantee.

The stabilization detection in Definition 1 is empirical: we have no formal guarantee on when the switch fires for an unseen dataset, nor a closed-form bound on the calibration loss it incurs. The hyperparameters $(\epsilon ,k)$ were tuned on a two-dataset subset (§4.1) and applied without retuning to every main-results experiment; a task whose stabilization profile differs substantially from our six benchmarks may need fresh tuning.

Generalization beyond encoders.

Our results cover encoder-based text classification with three backbones under $150$M parameters. Two aspects of HybridAL are tied to that setting. First, $\Delta \alpha$ is a spectral statistic of the weight matrices, so its scale depends on architecture and depth; the tuned $(\epsilon ,k)$ would not carry over to models with different spectra, and §4.1’s thresholds would need refitting. Second, the time savings come from FineTune converging in fewer epochs under early stopping, which assumes full-parameter updates; under parameter-efficient tuning the per-round cost gap that HybridAL exploits is much smaller. Whether the stabilization transition itself appears in decoder-based models, substantially larger backbones, or tasks beyond classification remains for future work.

Ethical Considerations

Data.

All six datasets are public research benchmarks used for their original classification tasks; we redistribute neither data nor annotations. The Jigsaw corpus (Wulczyn et al., 2017) contains toxic user comments, and automated toxicity classifiers are known to exhibit demographic biases (Sap et al., 2019); HybridAL is a training schedule and does not mitigate them.

Models.

We build on BERT, DistilBERT, and RoBERTa, which inherit biases from web-scale pretraining (Bender et al., 2021). Our results characterize average behavior across seeds and datasets, not worst-case behavior on specific subpopulations.

Compute.

The full set of experiments (1,626 runs across main results, ablations, and sensitivity studies) took approximately $280$ GPU-hours on two RTX 2080 Ti cards. HybridAL itself reduces per-cycle training time by $12$–$49\%$ relative to full retraining (§4.3), partially offsetting the cost of pool-based AL.

Scope.

No human subjects were involved. HybridAL is intended for research use; we do not recommend it for safety-critical applications where miscalibrated probabilities carry direct welfare consequences.

Acknowledgments

This work is partially supported by the Lando Kravetz Fund, Technion’s Grant 2073351.

References

• Ash and Adams (2020) J. Ash and R. P. Adams On warm-starting neural network training. Advances in neural information processing systems 33, pp. 3884–3894. Cited by: §1, §2, §3.1, §3.2, §5.
• Ash et al. (2019) J. T. Ash, C. Zhang, A. Krishnamurthy, J. Langford, and A. Agarwal Deep batch active learning by diverse, uncertain gradient lower bounds. arXiv preprint arXiv:1906.03671. Cited by: §G.5, §1, §4.1.
• Ashktorab et al. (2025) Z. Ashktorab, M. Desmond, Q. Pan, J. M. Johnson, M. Santillán Cooper, E. M. Daly, R. Nair, T. Pedapati, H. J. Do, and W. Geyer EvalAssist: insights on task-specific evaluations and ai-assisted judgment strategy preferences. In Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology, pp. 1–23. Cited by: §1.
• Barbieri et al. (2020) F. Barbieri, J. Camacho-Collados, L. E. Anke, and L. Neves TweetEval: unified benchmark and comparative evaluation for tweet classification. In Findings of the association for computational linguistics: EMNLP 2020, pp. 1644–1650. Cited by: Appendix A, Table 1.
• Beck et al. (2021) N. Beck, D. Sivasubramanian, A. Dani, G. Ramakrishnan, and R. Iyer Effective evaluation of deep active learning on image classification tasks. arXiv preprint arXiv:2106.15324. Cited by: §2, §3.1, §5.
• Bender et al. (2021) E. M. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell On the dangers of stochastic parrots: can language models be too big?. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pp. 610–623. Cited by: Models..
• Calderon et al. (2025) N. Calderon, R. Reichart, and R. Dror The alternative annotator test for llm-as-a-judge: how to statistically justify replacing human annotators with llms. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 16051–16081. Cited by: §1.
• Chen et al. (2024) G. H. Chen, S. Chen, Z. Liu, F. Jiang, and B. Wang Humans or llms as the judge? a study on judgement bias. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 8301–8327. Cited by: §1.
• Das et al. (2023) A. Das, G. Bhatt, M. Bhalerao, V. Gao, R. Yang, and J. Bilmes Accelerating batch active learning using continual learning techniques. arXiv preprint arXiv:2305.06408. Cited by: §2, §3.1, §5.
• Devlin et al. (2019) J. Devlin, M. Chang, K. Lee, and K. Toutanova Bert: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pp. 4171–4186. Cited by: §4.1.
• Dohare et al. (2024) S. Dohare, J. F. Hernandez-Garcia, Q. Lan, P. Rahman, A. R. Mahmood, and R. S. Sutton Loss of plasticity in deep continual learning. Nature 632 (8026), pp. 768–774. Cited by: §2.
• Fang et al. (2017) M. Fang, Y. Li, and T. Cohn Learning how to active learn: a deep reinforcement learning approach. In Proceedings of the 2017 conference on empirical methods in natural language processing, pp. 595–605. Cited by: §2.
• Gilardi et al. (2023) F. Gilardi, M. Alizadeh, and M. Kubli ChatGPT outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences 120 (30), pp. e2305016120. Cited by: §1.
• Guo et al. (2017) C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger On calibration of modern neural networks. In International conference on machine learning, pp. 1321–1330. Cited by: §B.5, §3.3, Residual calibration drift..
• Hacohen et al. (2022) G. Hacohen, A. Dekel, and D. Weinshall Active learning on a budget: opposite strategies suit high and low budgets. In International Conference on Machine Learning, pp. 8175–8195. Cited by: §2.
• Hacohen and Weinshall (2023) G. Hacohen and D. Weinshall How to select which active learning strategy is best suited for your specific problem and budget. Advances in Neural Information Processing Systems 36, pp. 13395–13407. Cited by: §2.
• Han et al. (2024) M. Han, C. Li, F. Meng, F. He, and R. Zhang An adaptive active learning method for multiclass imbalanced data streams with concept drift. Applied Sciences 14 (16), pp. 7176. Cited by: §2.
• Kornblith et al. (2019) S. Kornblith, M. Norouzi, H. Lee, and G. Hinton Similarity of neural network representations revisited. In International conference on machine learning, pp. 3519–3529. Cited by: §1, §4.1.
• Li et al. (2025) D. Li, B. Jiang, L. Huang, A. Beigi, C. Zhao, Z. Tan, A. Bhattacharjee, Y. Jiang, C. Chen, T. Wu, et al. From generation to judgment: opportunities and challenges of llm-as-a-judge. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 2757–2791. Cited by: §1.
• Liu et al. (2018) M. Liu, W. Buntine, and G. Haffari Learning how to actively learn: a deep imitation learning approach. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1874–1883. Cited by: §2.
• Liu et al. (2019) Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov Roberta: a robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. Cited by: §4.1.
• Loshchilov and Hutter (2017) I. Loshchilov and F. Hutter Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. Cited by: §4.1.
• Maas et al. (2011) A. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, and C. Potts Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pp. 142–150. Cited by: Appendix A, Table 1.
• Martin and Mahoney (2021) C. H. Martin and M. W. Mahoney Implicit self-regularization in deep neural networks: evidence from random matrix theory and implications for learning. Journal of Machine Learning Research 22 (165), pp. 1–73. Cited by: §1, §2, §4.1.
• Munagala et al. (2022) S. A. Munagala, S. Subramanian, S. Karthik, A. Prabhu, and A. Namboodiri Clactive: episodic memories for rapid active learning. In Conference on Lifelong Learning Agents, pp. 430–440. Cited by: §1, §2, §3.1.
• Papyan et al. (2020) V. Papyan, X. Han, and D. L. Donoho Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences 117 (40), pp. 24652–24663. Cited by: §2, §4.1.
• Paszke et al. (2019) A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. Pytorch: an imperative style, high-performance deep learning library. Advances in neural information processing systems 32. Cited by: §4.1.
• Ren et al. (2021) P. Ren, Y. Xiao, X. Chang, P. Huang, Z. Li, B. B. Gupta, X. Chen, and X. Wang A survey of deep active learning. ACM computing surveys (CSUR) 54 (9), pp. 1–40. Cited by: §1, §1, §3.2.
• Rolnick et al. (2019) D. Rolnick, A. Ahuja, J. Schwarz, T. Lillicrap, and G. Wayne Experience replay for continual learning. Advances in neural information processing systems 32. Cited by: §2, §5.
• Sanh et al. (2019) V. Sanh, L. Debut, J. Chaumond, and T. Wolf DistilBERT, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108. Cited by: §4.1.
• Sap et al. (2019) M. Sap, D. Card, S. Gabriel, Y. Choi, and N. A. Smith The risk of racial bias in hate speech detection. In Proceedings of the 57th annual meeting of the association for computational linguistics, pp. 1668–1678. Cited by: Data..
• Scala et al. (2024) F. Scala, S. Flesca, and L. Pontieri Play it straight: an intelligent data pruning technique for green-ai. In International conference on discovery science, pp. 69–85. Cited by: §2.
• Scala et al. (2025) F. Scala, S. Flesca, and L. Pontieri An efficient model training framework for green ai. Machine Learning 114 (12), pp. 275. Cited by: §1, §2.
• Schuirmann (1987) D. J. Schuirmann A comparison of the two one-sided tests procedure and the power approach for assessing the equivalence of average bioavailability. Journal of pharmacokinetics and biopharmaceutics 15 (6), pp. 657–680. Cited by: §B.2.
• Settles (2009) B. Settles Active learning literature survey. Technical report Technical Report 1648, University of Wisconsin-Madison Department of Computer Sciences. Cited by: §1, §1, §1, §3.1, §4.1.
• Socher et al. (2013) R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Y. Ng, and C. Potts Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1631–1642. Cited by: Appendix A, Table 1.
• Szymanski et al. (2025) A. Szymanski, N. Ziems, H. A. Eicher-Miller, T. J. Li, M. Jiang, and R. A. Metoyer Limitations of the llm-as-a-judge approach for evaluating llm outputs in expert knowledge tasks. In Proceedings of the 30th international conference on intelligent user interfaces, pp. 952–966. Cited by: §1.
• Tan et al. (2024) Z. Tan, D. Li, S. Wang, A. Beigi, B. Jiang, A. Bhattacharjee, M. Karami, J. Li, L. Cheng, and H. Liu Large language models for data annotation and synthesis: a survey. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 930–957. Cited by: §1.
• Vlachos (2008) A. Vlachos A stopping criterion for active learning. Computer Speech & Language 22 (3), pp. 295–312. Cited by: §2.
• Wolf et al. (2020) T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, et al. Transformers: state-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pp. 38–45. Cited by: §4.1.
• Wulczyn et al. (2017) E. Wulczyn, N. Thain, and L. Dixon Ex machina: personal attacks seen at scale. In Proceedings of the 26th international conference on world wide web, pp. 1391–1399. Cited by: Appendix A, Table 1, Data..
• Zhang et al. (2023) J. Zhang, S. Shao, S. Verma, and R. Nowak Algorithm selection for deep active learning with imbalanced datasets. Advances in Neural Information Processing Systems 36, pp. 9614–9647. Cited by: §2.
• Zhang et al. (2015) X. Zhang, J. Zhao, and Y. LeCun Character-level convolutional networks for text classification. Advances in neural information processing systems 28. Cited by: Appendix A, Appendix A, Table 1, Table 1.
• Zhu et al. (2008) J. Zhu, H. Wang, and E. Hovy Multi-criteria-based strategy to stop active learning for data annotation. In Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), pp. 1129–1136. Cited by: §2.

Appendix A Dataset Details

We briefly describe each dataset below; per-class distributions for the splits used in our experiments are in Table 3.

IMDb (Maas et al., 2011).

50,000 movie reviews from the Internet Movie Database, labelled positive (review score $\phantom{}\ge 7$) or negative ($\phantom{}\le 4$). Train and test splits are class-balanced 50/50; we use an 80/20 train/test partition with a $1\%$ validation holdout from train.

Jigsaw (Wulczyn et al., 2017).

159,571 comments from English Wikipedia talk pages, originally annotated for six types of toxicity (toxic, severely toxic, obscene, threatening, insulting, and identity-hate). We binarize using the primary toxicity label, yielding a heavily imbalanced binary task ($\phantom{}\approx 9.6\%$ positive class).

SST-2 (Socher et al., 2013).

Movie-review snippets from the Stanford Sentiment Treebank with binary positive/negative sentiment labels. We use the GLUE formulation, whose held-out validation split (used here as the test set) is class-balanced at $\phantom{}\approx 49/51\%$.

TweetEval (Barbieri et al., 2020).

The sentiment subtask of TweetEval. Tweets are labelled negative, neutral, or positive. The official train/test class distributions differ substantially (Table 3), making this dataset a useful stress test under label-distribution shift.

AG News (Zhang et al., 2015).

A balanced four-class news-topic classification corpus drawn from the AG news collection. Classes are World, Sports, Business, and Sci/Tech, with $\mathrm{30,000}$ training examples per class in the original release ($\mathrm{29,700}$ in our AL pool after the $1\%$ validation holdout).

Yahoo Answers (Zhang et al., 2015).

A 10-class question-topic classification corpus derived from the Yahoo! Answers comprehensive Q&A dataset. The original release contains 1.4M training and 60k test examples; for computational tractability we stratified-downsample to $\mathrm{6,000}$ documents per class ($\mathrm{4,950}$ train, $50$ validation held out from train, $\mathrm{1,000}$ test), preserving the uniform class distribution.

媒体内容 · 前往原文查看

Dataset	Class	Train ($n$)	Train (%)	Val. ($n$)	Val. (%)	Test ($n$)	Test (%)
IMDb	negative	19,750	50.00	250	50.00	5,000	50.00
	positive	19,750	50.00	250	50.00	5,000	50.00
	Total	39,500	—	500	—	10,000	—
Jigsaw	non-toxic	113,978	90.42	1,443	90.41	28,856	90.42
	toxic	12,082	9.58	153	9.59	3,059	9.58
	Total	126,060	—	1,596	—	31,915	—
SST-2	negative	29,482	44.22	298	44.21	428	49.08
	positive	37,193	55.78	376	55.79	444	50.92
	Total	66,675	—	674	—	872	—
TweetEval	negative	7,331	15.55	74	15.51	3,972	32.33
	neutral	21,326	45.24	216	45.28	5,937	48.33
	positive	18,481	39.21	187	39.20	2,375	19.33
	Total	47,138	—	477	—	12,284	—
AG News	World	29,700	25.00	300	25.00	1,900	25.00
	Sports	29,700	25.00	300	25.00	1,900	25.00
	Business	29,700	25.00	300	25.00	1,900	25.00
	Sci/Tech	29,700	25.00	300	25.00	1,900	25.00
	Total	118,800	—	1,200	—	7,600	—
Yahoo Answers	Society & Culture	4,950	10.00	50	10.00	1,000	10.00
	Science & Mathematics	4,950	10.00	50	10.00	1,000	10.00
	Health	4,950	10.00	50	10.00	1,000	10.00
	Education & Reference	4,950	10.00	50	10.00	1,000	10.00
	Computers & Internet	4,950	10.00	50	10.00	1,000	10.00
	Sports	4,950	10.00	50	10.00	1,000	10.00
	Business & Finance	4,950	10.00	50	10.00	1,000	10.00
	Entertainment & Music	4,950	10.00	50	10.00	1,000	10.00
	Family & Relationships	4,950	10.00	50	10.00	1,000	10.00
	Politics & Government	4,950	10.00	50	10.00	1,000	10.00
	Total	49,500	—	500	—	10,000	—

Appendix B Main Results: Full Breakdown

This appendix provides the complete per-cell breakdown of the main results: test F1 (§B.1), non-inferiority tests (§B.2), training time and test NLL (§B.3), post-switch signal stability (§B.4), and the temperature analysis behind the calibration differences (§B.5).

B.1 Per-dataset Test F1

Table 4 reports test F1 for every (method $\times$ backbone $\times$ dataset) cell across 5 seeds. The four FixedSwitch variants ($k\in \left\{3,5,7,10\right\}$) and the two HybridAL variants land within the same envelope as Retrain and FineTune: no cell shows a gap larger than $0.035$ F1 across these eight methods (mean spread $0.015$), confirming that switching the training strategy at any round leaves test F1 within the same envelope; formal non-inferiority tests are in §B.2. HybridAL($\Delta \alpha$) is statistically indistinguishable from both Retrain and FineTune on $16/18$ (backbone, dataset) cells (paired $t$-test, $p>0.05$); HybridAL($\Delta$Acc) is indistinguishable from Retrain on $17/18$ cells and from FineTune on $15/18$ cells. NewOnly’s deficit on DistilBERT and BERT is localized to the hard multi-class datasets (TweetEval, Yahoo Answers) and disappears on the easier binary tasks (IMDb, Jigsaw, SST-2); the gap closes on RoBERTa (§4.2).

媒体内容 · 前往原文查看

DistilBERT
Method	IMDb	AGNews	Jigsaw	SST-2	TwtEv	Yahoo
Retrain	0.819 $\pm$ 0.021	0.900 $\pm$ 0.004	0.881 $\pm$ 0.003	0.835 $\pm$ 0.021	0.634 $\pm$ 0.023	0.667 $\pm$ 0.004‡§¶⋆∙∘
FineTune	0.823 $\pm$ 0.008¶	0.904 $\pm$ 0.005	0.883 $\pm$ 0.004	0.858 $\pm$ 0.018	0.626 $\pm$ 0.021	0.655 $\pm$ 0.006§
NewOnly	0.823 $\pm$ 0.006	0.899 $\pm$ 0.001	0.882 $\pm$ 0.007	0.854 $\pm$ 0.009	0.552 $\pm$ 0.117	0.637 $\pm$ 0.009
HybridAL ($\Delta \alpha$)	0.814 $\pm$ 0.014	0.902 $\pm$ 0.007	0.881 $\pm$ 0.004∘	0.844 $\pm$ 0.022	0.627 $\pm$ 0.025	0.656 $\pm$ 0.004§
HybridAL ($\Delta$Acc)	0.827 $\pm$ 0.006	0.905 $\pm$ 0.005	0.880 $\pm$ 0.005	0.857 $\pm$ 0.012	0.629 $\pm$ 0.018	0.654 $\pm$ 0.005§
FixedSwitch (3)	0.824 $\pm$ 0.006	0.906 $\pm$ 0.004§	0.880 $\pm$ 0.004	0.852 $\pm$ 0.023	0.621 $\pm$ 0.022	0.655 $\pm$ 0.005§
FixedSwitch (5)	0.820 $\pm$ 0.014	0.902 $\pm$ 0.006	0.878 $\pm$ 0.006	0.860 $\pm$ 0.015	0.622 $\pm$ 0.008	0.657 $\pm$ 0.006§⋆
FixedSwitch (7)	0.833 $\pm$ 0.006∙	0.904 $\pm$ 0.003§	0.877 $\pm$ 0.005	0.850 $\pm$ 0.020	0.621 $\pm$ 0.023	0.655 $\pm$ 0.007§
FixedSwitch (10)	0.828 $\pm$ 0.005	0.904 $\pm$ 0.007	0.883 $\pm$ 0.005	0.857 $\pm$ 0.016	0.628 $\pm$ 0.019	0.658 $\pm$ 0.008§
BERT
Method	IMDb	AGNews	Jigsaw	SST-2	TwtEv	Yahoo
Retrain	0.826 $\pm$ 0.013	0.906 $\pm$ 0.004§	0.870 $\pm$ 0.014	0.875 $\pm$ 0.021	0.608 $\pm$ 0.013	0.667 $\pm$ 0.002§¶⋄
FineTune	0.840 $\pm$ 0.010†⋆	0.905 $\pm$ 0.004	0.880 $\pm$ 0.007	0.885 $\pm$ 0.007⋆	0.622 $\pm$ 0.014	0.666 $\pm$ 0.005¶⋆⋄
NewOnly	0.836 $\pm$ 0.012	0.899 $\pm$ 0.004	0.876 $\pm$ 0.008	0.876 $\pm$ 0.009	0.550 $\pm$ 0.090	0.608 $\pm$ 0.045
HybridAL ($\Delta \alpha$)	0.834 $\pm$ 0.012	0.903 $\pm$ 0.006§	0.884 $\pm$ 0.006§	0.895 $\pm$ 0.011§⋆	0.633 $\pm$ 0.019	0.658 $\pm$ 0.006
HybridAL ($\Delta$Acc)	0.826 $\pm$ 0.013	0.908 $\pm$ 0.002§	0.883 $\pm$ 0.009	0.868 $\pm$ 0.013	0.608 $\pm$ 0.013	0.662 $\pm$ 0.007
FixedSwitch (3)	0.845 $\pm$ 0.007†⋆	0.906 $\pm$ 0.005	0.883 $\pm$ 0.006§	0.881 $\pm$ 0.017	0.624 $\pm$ 0.021	0.662 $\pm$ 0.013
FixedSwitch (5)	0.838 $\pm$ 0.010	0.903 $\pm$ 0.005	0.878 $\pm$ 0.016	0.887 $\pm$ 0.006	0.621 $\pm$ 0.010	0.665 $\pm$ 0.007
FixedSwitch (7)	0.844 $\pm$ 0.006†⋆	0.908 $\pm$ 0.005§	0.883 $\pm$ 0.005§	0.880 $\pm$ 0.017	0.622 $\pm$ 0.024	0.654 $\pm$ 0.008
FixedSwitch (10)	0.839 $\pm$ 0.011	0.906 $\pm$ 0.003§	0.880 $\pm$ 0.011	0.889 $\pm$ 0.010	0.639 $\pm$ 0.009†‡⋆	0.656 $\pm$ 0.012
RoBERTa
Method	IMDb	AGNews	Jigsaw	SST-2	TwtEv	Yahoo
Retrain	0.862 $\pm$ 0.033	0.911 $\pm$ 0.005	0.877 $\pm$ 0.013	0.889 $\pm$ 0.015	0.647 $\pm$ 0.022	0.672 $\pm$ 0.006
FineTune	0.878 $\pm$ 0.005	0.904 $\pm$ 0.008	0.882 $\pm$ 0.007	0.906 $\pm$ 0.009	0.659 $\pm$ 0.015	0.668 $\pm$ 0.009
NewOnly	0.887 $\pm$ 0.003‡¶⋆∘	0.907 $\pm$ 0.003	0.880 $\pm$ 0.008	0.904 $\pm$ 0.008	0.647 $\pm$ 0.026	0.663 $\pm$ 0.014
HybridAL ($\Delta \alpha$)	0.873 $\pm$ 0.006	0.912 $\pm$ 0.005	0.880 $\pm$ 0.012	0.901 $\pm$ 0.013	0.652 $\pm$ 0.020	0.668 $\pm$ 0.007
HybridAL ($\Delta$Acc)	0.882 $\pm$ 0.004¶	0.914 $\pm$ 0.007	0.883 $\pm$ 0.004	0.895 $\pm$ 0.019	0.646 $\pm$ 0.021	0.672 $\pm$ 0.007§
FixedSwitch (3)	0.872 $\pm$ 0.015	0.911 $\pm$ 0.006	0.882 $\pm$ 0.005	0.905 $\pm$ 0.010	0.644 $\pm$ 0.016	0.672 $\pm$ 0.005
FixedSwitch (5)	0.877 $\pm$ 0.005	0.910 $\pm$ 0.004	0.885 $\pm$ 0.004	0.910 $\pm$ 0.010	0.652 $\pm$ 0.012	0.672 $\pm$ 0.005
FixedSwitch (7)	0.882 $\pm$ 0.007‡	0.912 $\pm$ 0.006	0.882 $\pm$ 0.004	0.901 $\pm$ 0.012	0.662 $\pm$ 0.010	0.675 $\pm$ 0.007¶
FixedSwitch (10)	0.882 $\pm$ 0.008	0.912 $\pm$ 0.004	0.886 $\pm$ 0.004	0.907 $\pm$ 0.010†	0.657 $\pm$ 0.015∙	0.671 $\pm$ 0.003

B.2 Non-Inferiority of Endpoint F1

Table 5 reports one-sided $95\%$ lower confidence bounds on the mean paired F1 difference between each HybridAL variant and each reference method. A non-significant paired $t$-test does not establish equivalence, so we assess non-inferiority directly with the two one-sided tests procedure (Schuirmann, 1987). For each comparison we form the paired difference $d=F1\left(\text{HybridAL}\right)-F1\left(\text{reference}\right)$ over all $90$ (backbone, dataset, seed) cells and test $H_0:E\left[d\right]\le -\delta$ against $H_1:E\left[d\right]>-\delta$ at $\alpha =0.05$, equivalently a paired $t$-test with the null shifted by the margin $\delta$. We anchor $\delta$ to the noise floor of the experiment: Retrain’s seed-to-seed F1 standard deviation averages $0.0131$ across the $18$ (backbone, dataset) cells (median $0.013$, max $0.033$), so $\delta =0.005$ is about one third of that floor and $\delta =0.010$ about three quarters.

媒体内容 · 前往原文查看

Variant	Reference	Lower bound	$\delta =0.005$	$\delta =0.010$
$\Delta \alpha$	Retrain	$+0.0005$	✓	✓
$\Delta \alpha$	FineTune	$-0.0045$	✓	✓
$\Delta$Acc	Retrain	$-0.0001$	✓	✓
$\Delta$Acc	FineTune	$-0.0050$	✗	✓
$\Delta \alpha$	per-cell best	$-0.0086$	✗	✓
$\Delta$Acc	per-cell best	$-0.0091$	✗	✓

Both variants are non-inferior to Retrain and FineTune individually at $\delta =0.010$, and to the per-cell better of the two baselines (whichever is higher in each individual cell) at the same margin. At the tighter $\delta =0.005$, three of the four hybrid–baseline pairs pass; HybridAL($\Delta$Acc) vs. FineTune misses with $p=0.051$ and passes at $\delta =0.010$. HybridAL($\Delta \alpha$) vs. Retrain has a positive lower bound and is therefore non-inferior at any $\delta \ge 0$. Running TOST separately within each cell would use $n=5$ seeds and is underpowered by construction, so the pooled test is the appropriate instrument for a macro-level claim about endpoint F1; all per-cell mean differences lie below $0.025$ F1, so no single cell contributes a heterogeneous effect large enough to change the conclusion.

B.3 Per-dataset Training Time and Test NLL

Table 6 reports mean test NLL and Table 7 mean training time in seconds, for every (method $\times$ backbone $\times$ dataset) cell across 5 seeds. Both metrics are lower-is-better; per-column heatmap shading within each backbone (greener $=$ lower); superscripts mark paired-$t$-test significance ($p<0.05$, two-sided) per the caption legend.

The breakdown confirms the per-backbone trade-off discussed in §4.3: Retrain reaches the lowest NLL on essentially every cell but is the slowest; FineTune carries the highest NLL among pool-trained methods. The four FixedSwitch variants are consistently faster than HybridAL on every backbone. On BERT, HybridAL($\Delta \alpha$) is also faster than FineTune (mean $819$ s vs. $936$ s), driven by early switching on several datasets. FixedSwitch NLL clusters near FineTune’s, while both HybridAL variants pull substantially closer to Retrain (§4.4). NewOnly is the cheapest method overall and converges to NLL between Retrain’s and FineTune’s, but its F1 deficit on hard multi-class tasks (§4.2) keeps it off the joint Pareto frontier.

媒体内容 · 前往原文查看

Method	IMDb	AGNews	Jigsaw	SST-2	TwtEv	Yahoo	Mean
DistilBERT
Retrain	0.426 $\pm$ 0.039‡¶∙∘⋄△	0.320 $\pm$ 0.008‡§¶⋆∙∘⋄△	0.133 $\pm$ 0.008‡⋆∙⋄	0.391 $\pm$ 0.029	0.814 $\pm$ 0.066‡¶⋆∙∘⋄△	1.108 $\pm$ 0.018‡§¶⋆∙∘⋄△	0.532 $\pm$ 0.336‡§¶⋆∙∘⋄△
FineTune	0.614 $\pm$ 0.075	0.448 $\pm$ 0.042	0.153 $\pm$ 0.016	0.460 $\pm$ 0.053	1.379 $\pm$ 0.075	1.609 $\pm$ 0.030	0.777 $\pm$ 0.540
NewOnly	0.501 $\pm$ 0.036‡∙∘⋄△	0.362 $\pm$ 0.013‡⋆∙∘⋄△	0.133 $\pm$ 0.007‡⋆∙⋄	0.435 $\pm$ 0.041	0.880 $\pm$ 0.105‡∙∘⋄△	1.265 $\pm$ 0.046‡⋆∙∘⋄△	0.596 $\pm$ 0.382‡¶⋆∙∘⋄△
HybridAL ($\Delta \alpha$)	0.641 $\pm$ 0.140	0.408 $\pm$ 0.059	0.150 $\pm$ 0.023∘	0.412 $\pm$ 0.038△	1.288 $\pm$ 0.321	1.441 $\pm$ 0.192	0.724 $\pm$ 0.508∘
HybridAL ($\Delta$Acc)	0.457 $\pm$ 0.079‡∙∘⋄△	0.433 $\pm$ 0.023	0.156 $\pm$ 0.008	0.393 $\pm$ 0.032△	1.163 $\pm$ 0.246	1.490 $\pm$ 0.175	0.682 $\pm$ 0.498‡∙∘⋄△
FixedSwitch (3)	0.628 $\pm$ 0.051	0.414 $\pm$ 0.027	0.164 $\pm$ 0.020	0.481 $\pm$ 0.102	1.398 $\pm$ 0.071	1.579 $\pm$ 0.033	0.777 $\pm$ 0.535
FixedSwitch (5)	0.665 $\pm$ 0.088	0.444 $\pm$ 0.017	0.167 $\pm$ 0.030	0.493 $\pm$ 0.157	1.427 $\pm$ 0.113	1.612 $\pm$ 0.078	0.801 $\pm$ 0.547
FixedSwitch (7)	0.610 $\pm$ 0.021△	0.429 $\pm$ 0.028	0.174 $\pm$ 0.027	0.496 $\pm$ 0.098	1.369 $\pm$ 0.093	1.551 $\pm$ 0.065	0.772 $\pm$ 0.519
FixedSwitch (10)	0.668 $\pm$ 0.029	0.426 $\pm$ 0.030	0.156 $\pm$ 0.020	0.520 $\pm$ 0.095	1.325 $\pm$ 0.111∘	1.549 $\pm$ 0.039‡	0.774 $\pm$ 0.509
BERT
Retrain	0.398 $\pm$ 0.027‡¶∙∘⋄△	0.317 $\pm$ 0.016‡§¶⋆∙∘⋄△	0.132 $\pm$ 0.007‡⋆∙⋄△	0.338 $\pm$ 0.070∙	0.835 $\pm$ 0.042‡¶∙∘⋄△	1.117 $\pm$ 0.019‡§¶⋆∙∘⋄△	0.523 $\pm$ 0.348‡§¶⋆∙∘⋄△
FineTune	0.600 $\pm$ 0.095	0.398 $\pm$ 0.014⋄	0.157 $\pm$ 0.017	0.389 $\pm$ 0.033	1.434 $\pm$ 0.074	1.539 $\pm$ 0.063∙⋄	0.753 $\pm$ 0.547
NewOnly	0.445 $\pm$ 0.037‡∙∘⋄△	0.383 $\pm$ 0.033	0.136 $\pm$ 0.007∙	0.374 $\pm$ 0.018	0.888 $\pm$ 0.056‡¶∙∘⋄△	1.357 $\pm$ 0.118∙∘⋄△	0.597 $\pm$ 0.417‡¶∙∘⋄△
HybridAL ($\Delta \alpha$)	0.554 $\pm$ 0.137	0.405 $\pm$ 0.026	0.150 $\pm$ 0.017	0.393 $\pm$ 0.052	1.300 $\pm$ 0.110	1.460 $\pm$ 0.206	0.710 $\pm$ 0.509
HybridAL ($\Delta$Acc)	0.447 $\pm$ 0.088‡∘△	0.430 $\pm$ 0.046	0.154 $\pm$ 0.012	0.438 $\pm$ 0.080	0.835 $\pm$ 0.042‡¶∙∘⋄△	1.403 $\pm$ 0.189	0.618 $\pm$ 0.419‡¶∙∘⋄△
FixedSwitch (3)	0.525 $\pm$ 0.037∘	0.411 $\pm$ 0.049	0.164 $\pm$ 0.015	0.415 $\pm$ 0.065	1.345 $\pm$ 0.117	1.586 $\pm$ 0.060	0.741 $\pm$ 0.541
FixedSwitch (5)	0.635 $\pm$ 0.091	0.416 $\pm$ 0.027	0.158 $\pm$ 0.014	0.387 $\pm$ 0.046	1.335 $\pm$ 0.097	1.550 $\pm$ 0.046	0.747 $\pm$ 0.527
FixedSwitch (7)	0.527 $\pm$ 0.088∘	0.427 $\pm$ 0.028	0.150 $\pm$ 0.009	0.400 $\pm$ 0.052	1.426 $\pm$ 0.153	1.616 $\pm$ 0.070	0.758 $\pm$ 0.568
FixedSwitch (10)	0.589 $\pm$ 0.049	0.403 $\pm$ 0.020	0.159 $\pm$ 0.022	0.379 $\pm$ 0.058	1.278 $\pm$ 0.076‡	1.553 $\pm$ 0.077	0.727 $\pm$ 0.520
RoBERTa
Retrain	0.366 $\pm$ 0.099‡∙∘	0.285 $\pm$ 0.015‡§¶⋆∙∘⋄△	0.127 $\pm$ 0.016‡⋆∘△	0.290 $\pm$ 0.043∙	0.823 $\pm$ 0.070‡∙∘⋄△	1.094 $\pm$ 0.018‡¶∙∘⋄△	0.498 $\pm$ 0.352‡§¶⋆∙∘⋄△
FineTune	0.490 $\pm$ 0.071	0.439 $\pm$ 0.042	0.163 $\pm$ 0.022	0.331 $\pm$ 0.056	1.370 $\pm$ 0.115	1.610 $\pm$ 0.107	0.734 $\pm$ 0.563
NewOnly	0.358 $\pm$ 0.036‡∙∘⋄△	0.327 $\pm$ 0.009‡⋆∙∘⋄△	0.130 $\pm$ 0.009⋆∙∘△	0.328 $\pm$ 0.032	0.867 $\pm$ 0.088‡∙∘⋄△	1.145 $\pm$ 0.066‡¶∙∘⋄△	0.526 $\pm$ 0.365‡¶⋆∙∘⋄△
HybridAL ($\Delta \alpha$)	0.381 $\pm$ 0.093∙	0.369 $\pm$ 0.046‡	0.131 $\pm$ 0.018⋆△	0.313 $\pm$ 0.039∙	1.225 $\pm$ 0.318	1.523 $\pm$ 0.078	0.657 $\pm$ 0.545‡∙∘
HybridAL ($\Delta$Acc)	0.396 $\pm$ 0.080∘	0.410 $\pm$ 0.044	0.161 $\pm$ 0.017	0.312 $\pm$ 0.043	1.085 $\pm$ 0.334∙	1.247 $\pm$ 0.210‡¶∙∘⋄△	0.602 $\pm$ 0.443‡∙∘⋄△
FixedSwitch (3)	0.471 $\pm$ 0.044	0.404 $\pm$ 0.041	0.178 $\pm$ 0.028	0.363 $\pm$ 0.031	1.528 $\pm$ 0.202	1.631 $\pm$ 0.098	0.763 $\pm$ 0.602
FixedSwitch (5)	0.523 $\pm$ 0.069	0.413 $\pm$ 0.027	0.157 $\pm$ 0.019	0.325 $\pm$ 0.043	1.469 $\pm$ 0.069	1.562 $\pm$ 0.064	0.742 $\pm$ 0.570
FixedSwitch (7)	0.474 $\pm$ 0.039	0.402 $\pm$ 0.019	0.166 $\pm$ 0.037	0.335 $\pm$ 0.038	1.287 $\pm$ 0.130∘	1.586 $\pm$ 0.080	0.708 $\pm$ 0.543∙∘
FixedSwitch (10)	0.437 $\pm$ 0.056‡∘	0.382 $\pm$ 0.039	0.167 $\pm$ 0.014	0.338 $\pm$ 0.045	1.305 $\pm$ 0.099∘	1.560 $\pm$ 0.040	0.698 $\pm$ 0.542‡∙∘

媒体内容 · 前往原文查看

Method	IMDb	AGNews	Jigsaw	SST-2	TwtEv	Yahoo	Mean
DistilBERT
Retrain	526 $\pm$ 19	690 $\pm$ 33	719 $\pm$ 41	834 $\pm$ 39	886 $\pm$ 22	1375 $\pm$ 44	838 $\pm$ 272
FineTune	367 $\pm$ 11†¶⋆⋄△	407 $\pm$ 17†¶⋆∘⋄△	459 $\pm$ 12†⋆⋄△	621 $\pm$ 14†¶⋆∘⋄△	564 $\pm$ 13†⋆∘⋄△	675 $\pm$ 19†¶⋆∘⋄△	515 $\pm$ 115†¶⋆∘⋄△
NewOnly	153 $\pm$ 6†‡¶⋆∙∘⋄△	324 $\pm$ 15†‡¶⋆∙∘⋄△	374 $\pm$ 13†‡¶⋆∙∘⋄△	122 $\pm$ 3†‡¶⋆∙∘⋄△	94 $\pm$ 7†‡¶⋆∙∘⋄△	162 $\pm$ 12†‡¶⋆∙∘⋄△	205 $\pm$ 107†‡¶⋆∙∘⋄△
HybridAL ($\Delta \alpha$)	478 $\pm$ 84	564 $\pm$ 102	572 $\pm$ 115†	780 $\pm$ 97	687 $\pm$ 126†	1061 $\pm$ 258†	690 $\pm$ 234†
HybridAL ($\Delta$Acc)	510 $\pm$ 70	512 $\pm$ 42†	523 $\pm$ 21†	727 $\pm$ 84†	765 $\pm$ 102	1000 $\pm$ 189†	673 $\pm$ 204†
FixedSwitch (3)	365 $\pm$ 11†¶⋆⋄△	420 $\pm$ 18†¶⋆∘⋄△	457 $\pm$ 9†⋆⋄△	617 $\pm$ 13†¶⋆∘⋄△	567 $\pm$ 9†⋆∘⋄△	674 $\pm$ 30†¶⋆∘⋄△	517 $\pm$ 113†¶⋆∘⋄△
FixedSwitch (5)	370 $\pm$ 8†¶⋆⋄△	452 $\pm$ 13†⋆⋄△	476 $\pm$ 15†⋆⋄△	644 $\pm$ 8†¶△	601 $\pm$ 19†⋆⋄△	746 $\pm$ 30†¶⋆⋄△	548 $\pm$ 130†¶⋆⋄△
FixedSwitch (7)	415 $\pm$ 10†⋆	491 $\pm$ 21†	512 $\pm$ 25†△	656 $\pm$ 16†¶△	607 $\pm$ 15†⋆△	777 $\pm$ 26†△	576 $\pm$ 122†¶⋆△
FixedSwitch (10)	415 $\pm$ 16†⋆	505 $\pm$ 16†	538 $\pm$ 9†	680 $\pm$ 3†	662 $\pm$ 18†	866 $\pm$ 13†	611 $\pm$ 149†¶⋆
BERT
Retrain	960 $\pm$ 51	1442 $\pm$ 155	1296 $\pm$ 61	1560 $\pm$ 57	1581 $\pm$ 62	2752 $\pm$ 185	1599 $\pm$ 574
FineTune	651 $\pm$ 22†⋆⋄△	774 $\pm$ 28†⋆∙∘⋄△	820 $\pm$ 33†¶⋆∙∘⋄△	1127 $\pm$ 15†	1042 $\pm$ 32†⋆	1202 $\pm$ 24†	936 $\pm$ 205†⋆
NewOnly	242 $\pm$ 10†‡¶⋆∙∘⋄△	528 $\pm$ 28†‡¶⋆∙∘⋄△	621 $\pm$ 28†‡¶⋆∙∘⋄△	236 $\pm$ 22†‡¶⋆∙∘⋄△	168 $\pm$ 25†‡¶⋆∙∘⋄△	269 $\pm$ 14†‡⋆∙∘⋄△	344 $\pm$ 172†‡¶⋆∙∘⋄△
HybridAL ($\Delta \alpha$)	760 $\pm$ 98†⋆	994 $\pm$ 200†	926 $\pm$ 48†	621 $\pm$ 73†‡	549 $\pm$ 32†‡⋆	1066 $\pm$ 821†	819 $\pm$ 372†⋆
HybridAL ($\Delta$Acc)	947 $\pm$ 55†	972 $\pm$ 69†	964 $\pm$ 60†	994 $\pm$ 570	1584 $\pm$ 60	1062 $\pm$ 190†	1087 $\pm$ 323†
FixedSwitch (3)	625 $\pm$ 21†¶⋆⋄△	813 $\pm$ 33†⋆⋄△	858 $\pm$ 28†⋆⋄△	542 $\pm$ 8†‡⋄△	505 $\pm$ 11†‡¶⋆∘△	584 $\pm$ 34†‡⋆∘⋄△	654 $\pm$ 138†‡¶⋆∘⋄△
FixedSwitch (5)	658 $\pm$ 21†⋆⋄△	837 $\pm$ 28†⋆⋄△	883 $\pm$ 23†△	553 $\pm$ 29†‡△	562 $\pm$ 6†‡⋆	682 $\pm$ 16†‡⋆△	696 $\pm$ 129†‡⋆⋄△
FixedSwitch (7)	697 $\pm$ 4†⋆△	948 $\pm$ 41†	924 $\pm$ 36†	562 $\pm$ 8†‡△	526 $\pm$ 23†‡¶⋆∘△	711 $\pm$ 28†‡⋆△	728 $\pm$ 166†‡⋆△
FixedSwitch (10)	724 $\pm$ 9†⋆	987 $\pm$ 87†	965 $\pm$ 13†	618 $\pm$ 19†‡	574 $\pm$ 14†‡⋆	815 $\pm$ 35†‡	781 $\pm$ 165†‡⋆
RoBERTa
Retrain	950 $\pm$ 59	1072 $\pm$ 42	1176 $\pm$ 82	749 $\pm$ 17	693 $\pm$ 28	873 $\pm$ 39	919 $\pm$ 178
FineTune	620 $\pm$ 20†⋆∘⋄△	704 $\pm$ 20†¶⋆∘⋄△	838 $\pm$ 46†¶△	535 $\pm$ 16†⋆∘⋄△	491 $\pm$ 21†⋆∘⋄△	529 $\pm$ 14†¶⋆∘⋄△	619 $\pm$ 124†¶⋆∙∘⋄△
NewOnly	219 $\pm$ 9†‡¶⋆∙∘⋄△	432 $\pm$ 9†‡¶⋆∙∘⋄△	538 $\pm$ 40†‡¶⋆∙∘⋄△	203 $\pm$ 10†‡¶⋆∙∘⋄△	138 $\pm$ 4†‡¶⋆∙∘⋄△	204 $\pm$ 18†‡¶⋆∙∘⋄△	289 $\pm$ 148†‡¶⋆∙∘⋄△
HybridAL ($\Delta \alpha$)	836 $\pm$ 175	995 $\pm$ 111	1154 $\pm$ 168	654 $\pm$ 104	583 $\pm$ 115	637 $\pm$ 44†⋆	810 $\pm$ 241†
HybridAL ($\Delta$Acc)	803 $\pm$ 110†	823 $\pm$ 70†¶	909 $\pm$ 34†¶△	703 $\pm$ 87	645 $\pm$ 80	815 $\pm$ 108	783 $\pm$ 117†
FixedSwitch (3)	627 $\pm$ 17†⋆∘⋄△	736 $\pm$ 47†¶△	840 $\pm$ 29†¶⋆⋄△	549 $\pm$ 23†⋆∘⋄△	495 $\pm$ 24†⋆∘△	561 $\pm$ 33†¶⋆⋄△	635 $\pm$ 124†¶⋆∘⋄△
FixedSwitch (5)	696 $\pm$ 25†⋄	768 $\pm$ 27†¶△	860 $\pm$ 39†¶⋄△	602 $\pm$ 25†⋆	524 $\pm$ 14†⋆	567 $\pm$ 13†¶⋆⋄△	669 $\pm$ 122†¶⋆⋄△
FixedSwitch (7)	711 $\pm$ 19†	794 $\pm$ 46†¶	899 $\pm$ 34†¶△	592 $\pm$ 17†	520 $\pm$ 9†⋆△	640 $\pm$ 20†⋆	693 $\pm$ 131†¶⋆△
FixedSwitch (10)	699 $\pm$ 16†	862 $\pm$ 53†	993 $\pm$ 35†	596 $\pm$ 16†⋆	552 $\pm$ 22†⋆	668 $\pm$ 13†⋆	728 $\pm$ 158†¶⋆

B.4 Post-Switch Signal Stability

Tables 8 and 9 report post-switch dynamics and per-cell $t^{\star }$. After switching, both signals frequently re-cross their thresholds ($31\%$ to $46\%$ of post-switch rounds for $\Delta \alpha$; $36\%$ to $83\%$ for $\Delta$Acc), and $t^{\star }$ itself spans $4.0$ to $23.0$ across cells. Endpoint F1 is non-inferior (§4.2), so the excursions reflect post-switch optimization dynamics rather than failed stabilization. A reversible variant would treat them as instability and oscillate, breaking the monotonic cost guarantee; hence the irreversibility in Algorithm 1. Figures 5 and 6 show representative per-round trajectories with $\epsilon$ and $t^{\star }$ marked: a fast switcher, a slow switcher, a run whose sub-threshold dips are filtered by the patience parameter, and a run that never fires. Figure 7 aggregates this behaviour across all $90$ runs per signal: $73\%$ ($\Delta \alpha$) and $66\%$ ($\Delta$Acc) of below-$\epsilon$ stretches are shorter than $k$ and are therefore filtered rather than triggering a switch.

媒体内容 · 前往原文查看

媒体内容 · 前往原文查看

媒体内容 · 前往原文查看

媒体内容 · 前往原文查看

媒体内容 · 前往原文查看

媒体内容 · 前往原文查看

媒体内容 · 前往原文查看

媒体内容 · 前往原文查看

媒体内容 · 前往原文查看

媒体内容 · 前往原文查看

媒体内容 · 前往原文查看

	$\Delta \alpha$ ($\epsilon ={10}^{-4}$)			$\Delta$Acc ($\epsilon =5\times {10}^{-3}$)
Dataset	$t^{\star }$	max post	%$\phantom{}>\epsilon$	$t^{\star }$	max post	%$\phantom{}>\epsilon$
IMDb	8.4	4.2e​-​4	43	17.8	3.8e​-​2	83
AG News	10.7	4.4e​-​4	46	8.0	2.5e​-​2	54
Jigsaw	6.9	2.9e​-​4	31	9.0	1.6e​-​2	36
SST-2	8.4	4.6e​-​4	33	10.8	3.9e​-​2	67
TweetEval	6.3	4.5e​-​4	41	12.5	6.7e​-​2	80
Yahoo	10.7	4.6e​-​4	40	15.2	5.4e​-​2	79

媒体内容 · 前往原文查看

	HybridAL ($\Delta \alpha$)				HybridAL ($\Delta$Acc)
Dataset	DistilBERT	BERT	RoBERTa	Mean	DistilBERT	BERT	RoBERTa	Mean
IMDb	12.3	8.5	4.5	8.4	15.5	23.0	14.8	17.8
AG News	11.3	8.8	12.0	10.7	8.6	8.0	7.4	8.0
Jigsaw	11.2	5.4	4.0	6.9	7.4	9.6	10.0	9.0
SST-2	7.5	9.6	8.0	8.4	9.8	9.0	13.5	10.8
TweetEval	7.5	6.6	4.7	6.3	14.0	✗	11.0	12.5
Yahoo	11.8	10.8	9.4	10.7	13.4	15.8	16.5	15.2
Mean	10.3	8.3	7.1	8.6	11.4	13.1	12.2	12.2

B.5 Calibration Mechanism: Temperature Analysis

To understand the mechanism behind the calibration differences in §4.3, we fit a single temperature parameter $\tau$ per run on the validation set (Guo et al., 2017) and examine the fitted values (Table 10).

The fitted temperatures reveal a clear pattern: Retrain is intrinsically well-calibrated ($\tau =0.97$, requiring almost no correction), while FineTune is severely overconfident ($\tau =2.29$). Both HybridAL variants fall between these extremes ($\tau \approx 1.8$–$2.0$), consistent with the design intent: retraining during the early high-variance rounds produces less overconfident predictions than switching to warm-starting immediately. The monotonic ordering Retrain $<$ HybridAL($\Delta$Acc) $<$ HybridAL($\Delta \alpha$) $<$ FineTune in both $\tau$ and raw NLL confirms that later switching preserves more of Retrain’s calibration, as predicted by the stabilization hypothesis (§3.2).

After temperature scaling, all pool-trained methods converge to similar NLL ($0.517$–$0.533$) and ECE ($2.5$–$3.0\%$), confirming that they can reach the same calibration ceiling given a held-out validation set for post-hoc fitting. This is expected: all pool-trained methods reach similar F1 (§4.2), so their learned representations carry similar discriminative information; the differences lie in how well-scaled the probabilities are during training, when uncertainty-based acquisition functions use them to select examples. NewOnly is the exception: even after scaling, it retains the highest NLL ($0.560$) and ECE ($3.3\%$), indicating that discarding historical data harms representation quality, not just probability scaling. This provides additional evidence for excluding NewOnly from the substantive Pareto frontier (§4.3). Acquisition in this work uses the model’s native probabilities, so probability quality during training affects which examples are queried. Applying per-round temperature scaling before acquisition is possible in principle but would require refitting ${\tau }_t$ at every round; we leave this controlled comparison to future work. In low-resource AL settings where the validation set itself is expensive, HybridAL’s intrinsic calibration ($\tau \approx 1.8$–$2.0$ vs. FineTune’s $2.29$) may be the only available option.

媒体内容 · 前往原文查看

Method	$\tau$	NLL	NLL ($\tau$)	ECE	ECE ($\tau$)
Retrain	0.97	0.532	0.517	5.4%	2.6%
FineTune	2.29	0.777	0.533	13.7%	2.5%
NewOnly	1.20	0.596	0.560	8.0%	3.3%
HybridAL ($\Delta \alpha$)	1.97	0.722	0.532	12.1%	2.8%
HybridAL ($\Delta$Acc)	1.80	0.682	0.525	11.8%	3.0%

Appendix C Validation-Label Assumptions and Size Sensitivity

$𝒱$ is fixed, held out, and disjoint from ${ℒ}_t$ and ${𝒰}_t$; it is shared by all methods for early stopping and per-round evaluation, so it is not a cost specific to HybridAL. Its size ranges from $477$ to $\mathrm{1,596}$ labels (Table 3), comparable to or larger than the $\mathrm{1,000}$-label acquisition budget. The two signals differ in how they use it: $\Delta$Acc reads its value from the early-stopping forward pass and adds no extra computation, while $\Delta \alpha$ is computed from weight matrices alone and depends on $𝒱$ only through early stopping. Replacing early stopping with a fixed-epoch schedule would remove that dependence entirely, making $\Delta \alpha$ validation-free, but would forfeit the time savings, which come from FineTune converging in $3.4$ epochs vs. Retrain’s $5.5$ under early stopping (§G.1).

Table 11 reports a focused study of how much validation data $\Delta$Acc actually needs. We subsample $𝒱$ to $\left|{𝒱}_{\text{sig}}\right|\in \left\{25,50,100,200,\text{full}\right\}$ labels on DistilBERT with IMDb and AG News, 3 seeds per cell, at the tuned $(\epsilon ,k)=(5\times {10}^{-3},2)$; subsampling is class-stratified, performed once at initialisation, and deterministic per seed. The subsampled set drives both early stopping and the $\Delta$Acc signal, and endpoint F1 is scored on the untouched test set. Endpoint F1 varies by at most $0.007$ within each dataset across all sizes (spread $0.0040$ on AG News, $0.0069$ on IMDb), inside the $0.013$ seed-noise floor of §B.2, so a few dozen validation labels suffice for the signal.

媒体内容 · 前往原文查看

Dataset	$\left|{𝒱}_{\text{sig}}\right|$	Fire rate	$t^{\star }$	Test F1
AG News	25	100%	$8.0$ $\pm$ $2.6$	$0.9010$ $\pm$ $0.0024$
	50	100%	$10.0$ $\pm$ $8.9$	$0.9044$ $\pm$ $0.0010$
	100	67%	$10.5$ $\pm$ $9.2$	$0.9022$ $\pm$ $0.0001$
	200	33%	$11$	$0.9004$ $\pm$ $0.0061$
	full ($\mathrm{1,200}$)	100%	$7.7$ $\pm$ $2.5$	$0.9041$ $\pm$ $0.0065$
IMDb	25	33%	$11$	$0.8236$ $\pm$ $0.0086$
	50	33%	$23$	$0.8305$ $\pm$ $0.0037$
	100	33%	$17$	$0.8237$ $\pm$ $0.0173$
	200	0%	—	$0.8295$ $\pm$ $0.0032$
	full ($500$)	33%	$24$	$0.8262$ $\pm$ $0.0077$

Fire rate should not be read as a trend in this table. It is non-monotonic in $\left|{𝒱}_{\text{sig}}\right|$, and at $\left|{𝒱}_{\text{sig}}\right|=25$ validation accuracy is quantised in steps of $1/25=0.04$, so $\Delta$Acc is either exactly $0$ or at least $0.04$, well above $\epsilon =5\times {10}^{-3}$: sub-threshold movement is numerically unobservable, and the signal can fire because every reading below $0.04$ registers as zero rather than because the trajectory has stabilised. IMDb’s lower fire rate persists at full $𝒱$ and matches its largest post-switch re-crossing rate in Table 8, so it is a property of the dataset rather than of validation size.

Appendix D Post-Switch Batch Composition

This appendix asks what changes about acquisition after the switch. We log per-round selected indices, acquired-batch entropies, and full validation logits for Retrain, HybridAL($\Delta \alpha$), and HybridAL($\Delta$Acc) on DistilBERT with IMDb and AG News, seeds $42$–$44$ (18 runs). This is a focused study on two datasets and one backbone, not the full grid.

D.1 Batch Overlap

Table 12 reports the Jaccard overlap between each HybridAL run’s acquired batch and that of the paired Retrain run at the same round, pooled over the six (dataset, seed) cells. Before the switch the two are identical by construction, since both are retraining from the same initialisation on the same pool. After the switch the overlap collapses to under $0.02$: the model selects almost entirely different examples. Endpoint F1 is nonetheless non-inferior (§B.2), so a different batch is not a worse batch at this budget.

媒体内容 · 前往原文查看

Comparison vs. Retrain	Pre-switch	Post-switch
HybridAL ($\Delta \alpha$)	$1.0000$ $\pm$ $0.0000$	$0.0174$ $\pm$ $0.0427$
HybridAL ($\Delta$Acc)	$0.9992$ $\pm$ $0.0071$	$0.0197$ $\pm$ $0.0515$

D.2 Class Balance

Table 13 gives the class distribution of acquired batches before and after the switch. No class collapses: the largest shift is $8.0$ pp on AG News Sci/Tech for $\Delta \alpha$, and the binary IMDb splits stay within $5$ pp of even. Low overlap therefore does not come from the sampler concentrating on a single class.

媒体内容 · 前往原文查看

Signal	Dataset	Pre-switch	Post-switch
$\Delta \alpha$	IMDb	$(47.6,52.4)$	$(51.4,48.6)$
$\Delta \alpha$	AG News	$(29.0,11.9,33.1,26.1)$	$(25.0,9.8,31.1,34.1)$
$\Delta$Acc	IMDb	$(52.5,47.5)$	$(51.5,48.5)$
$\Delta$Acc	AG News	$(30.6,13.5,29.2,26.7)$	$(24.6,11.0,32.8,31.5)$

D.3 Acquired-Batch Entropy

Table 14 reports the mean predictive entropy of the acquired batch. Post-switch entropy falls by $0.09$ ($\Delta \alpha$) and $0.08$ ($\Delta$Acc) relative to pre-switch. This is consistent with the sharper post-fine-tuning softmax documented in §B.5 ($\tau$ up to $2.29$), which compresses all entropies toward zero without reordering them, so top-$n$ selection would be unaffected. We note that this is an interpretation of the temperature evidence rather than a direct measurement: we did not compute rank correlation between pre- and post-switch entropy orderings.

媒体内容 · 前往原文查看

Method	Pre-switch	Post-switch	Change
HybridAL ($\Delta \alpha$)	$0.8455$ $\pm$ $0.1665$	$0.7547$ $\pm$ $0.0907$	$-0.0909$
HybridAL ($\Delta$Acc)	$0.8392$ $\pm$ $0.1835$	$0.7624$ $\pm$ $0.0884$	$-0.0768$
Retrain	$0.8419$ $\pm$ $0.1617$	—	—

D.4 Per-Round Calibration

Table 15 tracks 15-bin validation ECE through the AL loop. HybridAL matches Retrain in the early rounds, where the pool is small and each batch reshapes the labeled distribution most, and drifts above it later: over the last five rounds the gap averages $2.90$ pp for $\Delta \alpha$ and $5.06$ pp for $\Delta$Acc. Endpoint F1 is unchanged over the same window (§B.2). Late calibration drift therefore does not translate into a measurable endpoint cost here, and it is correctable at inference by temperature scaling (§B.5); but the pattern also means these analyses cannot support a claim that better calibration yields better acquisition, which we do not make.

媒体内容 · 前往原文查看

Round	Retrain	$\Delta \alpha$	$\Delta$Acc
5	$6.09$ $\pm$ $2.44$	$5.90$ $\pm$ $2.47$	$6.58$ $\pm$ $2.85$
10	$5.28$ $\pm$ $2.37$	$7.27$ $\pm$ $3.35$	$7.60$ $\pm$ $4.23$
15	$4.28$ $\pm$ $1.31$	$6.43$ $\pm$ $2.91$	$7.15$ $\pm$ $4.95$
20	$5.43$ $\pm$ $4.01$	$7.36$ $\pm$ $3.95$	$9.87$ $\pm$ $4.27$
24	$4.70$ $\pm$ $1.03$	$8.47$ $\pm$ $4.01$	$10.06$ $\pm$ $4.11$
Last 5	$5.11$ $\pm$ $3.08$	$8.01$ $\pm$ $3.76$	$10.17$ $\pm$ $3.87$

Appendix E Signal Ablation

We evaluated eight candidate switching signals (§4.1) on DistilBERT with the entropy sampler, across all six datasets and $5$ seeds. Because the raw signals have different units and magnitudes, we normalise each per-round value by the signal’s maximum over rounds $2$–$5$ of a calibration run, producing unit-free values that share a common threshold $\epsilon =0.5$ with patience $k=3$. Under this normalisation, a switch fires when the round-to-round change $\Delta S_t$ (Definition 1) drops below $\epsilon =0.5$ for $k$ consecutive rounds. This shared configuration is intended for ranking signals against each other; the absolute time and F1 numbers are not directly comparable to the per-signal tuned hyperparameters used in the main results (Appendix F). Table 16 reports per-(signal $\times$ dataset) test F1, test NLL, total training time, and switching rate (fraction of seeds where the signal triggered a switch within $T=25$ rounds); Figure 9 summarises the same data as a parallel-coordinates profile across five normalised axes (switch rate, speed, F1, calibration, overall rank).

Among the firing signals, $\Delta \alpha$ achieves the highest mean test F1 ($0.793$) at the lowest training cost ($424$ s), and $\Delta$Acc is the closest performance-based runner-up (F1 $\phantom{}=0.791$, time $438$ s) while requiring only a standard validation forward pass; the two signals dominate the eight on overall rank in Figure 9. We select $\Delta$Acc over the nearly redundant $\Delta$F1 (Spearman $\rho =0.8$ between per-round signal values, Figure 8) for its computational simplicity: accuracy is a single scalar, whereas macro-F1 requires per-class aggregation. $\Delta \alpha$ is orthogonal to every other signal ($\rho \approx 0$, Figure 8), confirming it captures complementary information: weight spectral statistics rather than task-level performance. We therefore adopt both as independent HybridAL variants in the main results (§4). Figure 10 confirms that all firing signals track Retrain’s per-round validation F1 throughout training, with $\Delta \alpha$ switching earliest.

Test NLL exposes the calibration cost of switching: the firing performance-based signals ($\Delta$Acc, $\Delta$F1, $\Delta$Loss, $\Delta$NC) average $0.57$–$0.70$ vs. Retrain’s $0.53$, and the model-based $\Delta \alpha$ averages $0.74$. Even the best signal ($\Delta$Loss at $0.57$) does not match Retrain on NLL, consistent with the calibration-time trade-off in §4.3; we adopt $\Delta \alpha$ and $\Delta$Acc despite this trade-off because their (F1, time) positions are Pareto-dominant among the eight, and calibration is then traded against time via the choice of signal. Although $\Delta$Loss achieves the best NLL among firing signals ($0.574$), it fires on only $43\%$ of cells and fails entirely on IMDb, defaulting to Retrain’s cost on non-firing datasets. Finally, ${\ell }_2$ weight distance never fires on any of the six datasets, CKA fires only on AG News and Yahoo, and gradient norm fires only on IMDb, SST-2, and TweetEval; the corresponding cells in Table 16 sit within one seed std of Retrain on every dataset, confirming that running HybridAL with a non-firing signal is statistically indistinguishable from Retrain.

媒体内容 · 前往原文查看

媒体内容 · 前往原文查看

媒体内容 · 前往原文查看

Test $F_1$ $\uparrow$
Signal	IMDb	AGNews	Jigsaw	SST-2	TwtEv	Yahoo	Mean
Retrain	0.819 $\pm$ 0.021	0.900 $\pm$ 0.004	0.881 $\pm$ 0.003	0.835 $\pm$ 0.021	0.634 $\pm$ 0.023	0.667 $\pm$ 0.004	0.789 $\pm$ 0.105
$\Delta$Acc	0.825 $\pm$ 0.009	0.904 $\pm$ 0.003	0.881 $\pm$ 0.007	0.847 $\pm$ 0.023	0.633 $\pm$ 0.009	0.657 $\pm$ 0.006	0.791 $\pm$ 0.109
$\Delta \alpha$	0.826 $\pm$ 0.006	0.901 $\pm$ 0.005	0.880 $\pm$ 0.007	0.861 $\pm$ 0.010	0.633 $\pm$ 0.012	0.657 $\pm$ 0.004	0.793 $\pm$ 0.109
$\Delta$F1	0.830 $\pm$ 0.011	0.904 $\pm$ 0.003	0.880 $\pm$ 0.005	0.847 $\pm$ 0.023	0.628 $\pm$ 0.011	0.656 $\pm$ 0.006	0.791 $\pm$ 0.111
$\Delta$Loss	0.804 $\pm$ 0.022	0.901 $\pm$ 0.005	0.883 $\pm$ 0.005	0.853 $\pm$ 0.020	0.636 $\pm$ 0.006	0.662 $\pm$ 0.005	0.790 $\pm$ 0.107
$\Delta$NC	0.812 $\pm$ 0.024	0.903 $\pm$ 0.004	0.879 $\pm$ 0.006	0.863 $\pm$ 0.011	0.628 $\pm$ 0.013	0.653 $\pm$ 0.009	0.790 $\pm$ 0.112
GradNorm	0.831 $\pm$ 0.008	0.905 $\pm$ 0.002	0.883 $\pm$ 0.008	0.860 $\pm$ 0.010	0.601 $\pm$ 0.039	0.661 $\pm$ 0.007	0.790 $\pm$ 0.119
${\ell }_2$	0.823 $\pm$ 0.010	0.902 $\pm$ 0.003	0.879 $\pm$ 0.005	0.862 $\pm$ 0.008	0.630 $\pm$ 0.022	0.664 $\pm$ 0.006	0.793 $\pm$ 0.109
CKA	0.823 $\pm$ 0.010	0.901 $\pm$ 0.009	0.879 $\pm$ 0.005	0.862 $\pm$ 0.008	0.630 $\pm$ 0.022	0.659 $\pm$ 0.004	0.792 $\pm$ 0.110
Test NLL $\downarrow$
Signal	IMDb	AGNews	Jigsaw	SST-2	TwtEv	Yahoo	Mean
Retrain	0.426 $\pm$ 0.039	0.320 $\pm$ 0.008	0.133 $\pm$ 0.008	0.391 $\pm$ 0.029	0.814 $\pm$ 0.066	1.108 $\pm$ 0.018	0.532 $\pm$ 0.336
$\Delta$Acc	0.599 $\pm$ 0.137	0.432 $\pm$ 0.028	0.147 $\pm$ 0.015	0.451 $\pm$ 0.076	1.104 $\pm$ 0.261	1.470 $\pm$ 0.216	0.700 $\pm$ 0.477
$\Delta \alpha$	0.658 $\pm$ 0.052	0.446 $\pm$ 0.018	0.147 $\pm$ 0.016	0.448 $\pm$ 0.063	1.321 $\pm$ 0.202	1.435 $\pm$ 0.190	0.743 $\pm$ 0.494
$\Delta$F1	0.583 $\pm$ 0.119	0.432 $\pm$ 0.028	0.141 $\pm$ 0.014	0.451 $\pm$ 0.076	1.025 $\pm$ 0.298	1.486 $\pm$ 0.222	0.686 $\pm$ 0.475
$\Delta$Loss	0.435 $\pm$ 0.037	0.418 $\pm$ 0.025	0.158 $\pm$ 0.020	0.376 $\pm$ 0.037	0.875 $\pm$ 0.163	1.184 $\pm$ 0.176	0.574 $\pm$ 0.364
$\Delta$NC	0.463 $\pm$ 0.081	0.435 $\pm$ 0.028	0.148 $\pm$ 0.017	0.427 $\pm$ 0.087	1.085 $\pm$ 0.262	1.462 $\pm$ 0.210	0.670 $\pm$ 0.479
GradNorm	0.537 $\pm$ 0.094	0.313 $\pm$ 0.013	0.129 $\pm$ 0.015	0.431 $\pm$ 0.062	1.414 $\pm$ 0.162	1.114 $\pm$ 0.022	0.656 $\pm$ 0.469
${\ell }_2$	0.414 $\pm$ 0.037	0.323 $\pm$ 0.012	0.134 $\pm$ 0.011	0.339 $\pm$ 0.008	0.805 $\pm$ 0.045	1.117 $\pm$ 0.019	0.522 $\pm$ 0.341
CKA	0.414 $\pm$ 0.037	0.449 $\pm$ 0.027	0.134 $\pm$ 0.011	0.339 $\pm$ 0.008	0.805 $\pm$ 0.045	1.585 $\pm$ 0.069	0.621 $\pm$ 0.484
Training time (s) $\downarrow$
Signal	IMDb	AGNews	Jigsaw	SST-2	TwtEv	Yahoo	Mean
Retrain	533 $\pm$ 23	690 $\pm$ 34	720 $\pm$ 39	372 $\pm$ 18	402 $\pm$ 10	655 $\pm$ 21	562 $\pm$ 141
$\Delta$Acc	497 $\pm$ 90	457 $\pm$ 12	530 $\pm$ 10	329 $\pm$ 63	335 $\pm$ 66	478 $\pm$ 130	438 $\pm$ 104
$\Delta \alpha$	410 $\pm$ 38	478 $\pm$ 45	509 $\pm$ 46	312 $\pm$ 18	314 $\pm$ 51	520 $\pm$ 134	424 $\pm$ 106
$\Delta$F1	499 $\pm$ 85	457 $\pm$ 12	545 $\pm$ 39	330 $\pm$ 63	378 $\pm$ 52	465 $\pm$ 132	445 $\pm$ 100
$\Delta$Loss	595 $\pm$ 27	607 $\pm$ 88	632 $\pm$ 34	401 $\pm$ 51	397 $\pm$ 32	613 $\pm$ 141	541 $\pm$ 123
$\Delta$NC	528 $\pm$ 89	482 $\pm$ 48	651 $\pm$ 88	375 $\pm$ 29	382 $\pm$ 34	450 $\pm$ 149	478 $\pm$ 123
GradNorm	448 $\pm$ 116	797 $\pm$ 31	865 $\pm$ 54	398 $\pm$ 72	294 $\pm$ 27	692 $\pm$ 16	582 $\pm$ 224
${\ell }_2$	606 $\pm$ 28	776 $\pm$ 17	826 $\pm$ 44	438 $\pm$ 12	445 $\pm$ 16	713 $\pm$ 23	634 $\pm$ 156
CKA	604 $\pm$ 28	451 $\pm$ 18	817 $\pm$ 43	431 $\pm$ 11	441 $\pm$ 16	404 $\pm$ 65	525 $\pm$ 152
Switching rate (%)
Signal	IMDb	AGNews	Jigsaw	SST-2	TwtEv	Yahoo	Mean
$\Delta$Acc	100	100	100	100	60	100	93
$\Delta \alpha$	100	100	100	100	100	80	97
$\Delta$F1	100	100	100	100	40	100	90
$\Delta$Loss	0	100	100	20	20	20	43
$\Delta$NC	20	100	80	60	60	80	67
GradNorm	80	0	0	60	100	0	40
${\ell }_2$	0	0	0	0	0	0	0
CKA	0	100	0	0	0	100	33

Appendix F Hyperparameter Tuning

Table 17 and Figure 11 give the $(\epsilon ,k)$ sensitivity for $\Delta \alpha$ and $\Delta$Acc, whose informative $\epsilon$ regimes differ by two orders of magnitude. We sweep $\left\{5\times {10}^{-5},{\mathrm{\hspace{0.17em}10}}^{-4},\mathrm{\hspace{0.17em}1.5}\times {10}^{-4},\mathrm{\hspace{0.17em}2}\times {10}^{-4}\right\}$ for $\Delta \alpha$ and $\left\{5\times {10}^{-3},{\mathrm{\hspace{0.17em}10}}^{-2},\mathrm{\hspace{0.17em}1.5}\times {10}^{-2},\mathrm{\hspace{0.17em}2}\times {10}^{-2}\right\}$ for $\Delta$Acc, each anchored so the smallest $\epsilon$ fires within the first five rounds and the largest within the budget; $k\in \left\{2,3,5,7\right\}$ is shared.

We pick $({\epsilon }^{\ast },k^{\ast })$ in two stages. First, restrict to cells whose mean validation F1 is within $0.5\%$ of the grid top:

$𝒞=\{(\epsilon ,k):F1(\epsilon ,k)\ge 0.995F1{}_{\max }{}^{}\}.$

(5)

This tolerance sits well below the per-cell seed standard deviation ($\phantom{}\sim 5$ pp), so $𝒞$ contains every cell statistically tied with the F1 best.

Second, among cells in $𝒞$ we need to trade off time against calibration, since the fastest cell is not always the best-calibrated. We minimise a combined score of wall-clock training time and validation NLL, each normalised by the in-set minimum:

$\mathrm{score}(\epsilon ,k)=\frac{\mathrm{Time}(\epsilon ,k)}{{\mathrm{Time}}_{\min }^{𝒞}}+\lambda \cdot \frac{\mathrm{NLL}(\epsilon ,k)}{{\mathrm{NLL}}_{\min }^{𝒞}},$

(6)

$({\epsilon }^{\ast },k^{\ast })=\arg \underset{(\epsilon ,k)\in 𝒞}{\min }\mathrm{score}(\epsilon ,k),$

(7)

with $\lambda =0.5$. Both normalised terms lie in $[1,\infty )$ and per-axis normalisation removes their unit gap, so $\lambda$ controls the calibration-vs.-time weight on $𝒞$. We pick $\lambda <1$ to favour time savings, since F1 within $𝒞$ is already statistically equivalent to the grid top; smaller $\lambda$ would over-weight time, while larger $\lambda$ would push toward Retrain on both axes. The selected cells are stable across $\lambda \in \left[0.3,0.7\right]$; we report $\lambda =0.5$ as the midpoint.

This selects $({10}^{-4},3)$ for $\Delta \alpha$ (2 candidates in $𝒞$; score $1.546$ vs. $1.561$) and $(5\times {10}^{-3},2)$ for $\Delta$Acc (7 candidates; the chosen cell jointly attains ${\mathrm{Time}}_{\min }^{𝒞}$ and ${\mathrm{NLL}}_{\min }^{𝒞}$, score $1.500$); both are applied to all main-results experiments without retuning.

Across the grid, mean validation F1 varies by only $1.1$ pp while mean training time varies by $1.5\times$ to $2.5\times$, so the choice is insensitive to F1 and driven by the (time, NLL) trade-off captured by Eq. 6.

媒体内容 · 前往原文查看

	mean val F1				mean val NLL				mean time (s)
$\epsilon$	$k=2$	$k=3$	$k=5$	$k=7$	$k=2$	$k=3$	$k=5$	$k=7$	$k=2$	$k=3$	$k=5$	$k=7$
$\Delta \alpha$
5e-5	0.858	0.857	0.855	0.855	0.447	0.385	0.370	0.370	290.8	318.0	331.9	322.5
1e-4	0.854	0.865	0.856	0.855	0.492	0.455	0.388	0.370	246.3	284.8	313.7	320.8
1.5e-4	0.855	0.855	0.858	0.855	0.498	0.511	0.435	0.370	214.3	229.4	289.3	320.9
2e-4	0.855	0.855	0.859	0.861	0.498	0.511	0.436	0.416	214.3	229.3	278.9	302.3
$\Delta$Acc
0.005	0.857	0.858	0.855	0.850	0.391	0.400	0.370	0.375	275.3	408.1	646.9	642.3
0.01	0.856	0.861	0.856	0.856	0.448	0.436	0.403	0.388	519.1	528.0	569.5	592.6
0.015	0.855	0.857	0.855	0.857	0.461	0.483	0.426	0.393	473.3	481.4	558.0	578.1
0.02	0.859	0.855	0.855	0.857	0.487	0.482	0.431	0.395	433.3	476.8	541.2	556.1

Appendix G Robustness Ablations

We test HybridAL’s sensitivity to five implementation choices on DistilBERT (all ablations use the $\Delta \alpha$ variant unless noted otherwise): the per-round training schedule (§G.1), the entropy pre-filter subset size $N$ (§G.2), the initial pool size (§G.3), the acquisition batch size (§G.4), and the acquisition sampler (§G.5).

G.1 Early Stopping

Our main experiments use max $10$ epochs per round with early stopping (patience $2$ on val loss); the alternative is a fixed budget of $5$ epochs. Both schedules deliver equivalent F1 on AG News and IMDb (max gap $0.014$, Table 18); early stopping is the default because it lets FineTune converge in $3.4$ epochs vs. Retrain’s $5.5$ (§4.1), the source of HybridAL’s time savings.

媒体内容 · 前往原文查看

Configuration	Method	AG News	IMDb
fixed $5$ ep	FineTune	$0.892$ $\pm$ $0.005$	$0.808$ $\pm$ $0.009$
fixed $5$ ep	Retrain	$0.897$ $\pm$ $0.004$	$0.822$ $\pm$ $0.010$
max $10$ + ES	FineTune	$0.895$ $\pm$ $0.004$	$0.818$ $\pm$ $0.007$
max $10$ + ES	Retrain	$0.892$ $\pm$ $0.007$	$0.821$ $\pm$ $0.012$

G.2 Subset Size Sensitivity

Entropy-based acquisition ranks unlabelled candidates by predictive entropy and selects the top $n$ for labelling each round. Scoring the entire unlabelled pool every round is expensive: on IMDb’s $\phantom{}\sim 40\text{k}$ pool the entropy pass dominates the per-round wall-clock cost for DistilBERT. We instead score a uniformly random subsample of $N$ candidates each round. To select $N$, we run both training strategies (Retrain and FineTune) for $N\in \left\{\mathrm{100,500},1000,2000,5000,10\text{k},20\text{k},\left|\text{pool}\right|\right\}$ on IMDb and SST-2 with three seeds each, measuring final test F1 (macro). Since $N$ is fixed once before all subsequent experiments and is not tuned per dataset, reporting test F1 here does not introduce data leakage.

媒体内容 · 前往原文查看

Figure 12 shows the results. Test F1 is essentially flat for $N\ge 1000$ on both datasets and both strategies: the F1 band over $N\in \left[1000,\left|\text{pool}\right|\right]$ stays within seed noise. At $N=500$ the IMDb Retrain mean drops noticeably, suggesting that this subsample is too small to reliably surface the highest entropy candidates. Increasing $N$ to $\left|\text{pool}\right|$ (the full pool) provides no benefit and on SST-2 even lowers F1 slightly, likely because the top entropy region is dominated by hard, noisy examples that the stochastic sampling at smaller $N$ smooths out. We therefore fix $N=1000$ for all subsequent experiments: the smallest value at which final F1 is statistically indistinguishable from larger $N$ on both datasets, while keeping the per round acquisition cost negligible.

G.3 Initial Pool Size

We vary $\left|{ℒ}_0\right|\in \left\{\mathrm{50,100,200,500}\right\}$ for Retrain and HybridAL (Table 19). HybridAL is within seed noise of Retrain in 11/12 cells; the empirical switch round shifts inversely with pool size (mean $7.8$–$12.0$), confirming that the stabilization signal adapts to data availability.

媒体内容 · 前往原文查看

Method	$\left|{ℒ}_0\right|$	AG News	IMDb	Yahoo
Retrain	50	$0.899$ $\pm$ $0.003$	$0.820$ $\pm$ $0.007$	$0.658$ $\pm$ $0.004$
HybridAL	50	$0.902$ $\pm$ $0.004$	$0.828$ $\pm$ $0.008$	$0.654$ $\pm$ $0.008$
Retrain	100	$0.898$ $\pm$ $0.004$	$0.821$ $\pm$ $0.015$	$0.658$ $\pm$ $0.009$
HybridAL	100	$0.898$ $\pm$ $0.004$	$0.826$ $\pm$ $0.014$	$0.643$ $\pm$ $0.010$
Retrain	200	$0.900$ $\pm$ $0.004$	$0.819$ $\pm$ $0.021$	$0.667$ $\pm$ $0.004$
HybridAL	200	$0.902$ $\pm$ $0.007$	$0.814$ $\pm$ $0.014$	$0.656$ $\pm$ $0.004$∗
Retrain	500	$0.902$ $\pm$ $0.006$	$0.836$ $\pm$ $0.004$	$0.679$ $\pm$ $0.004$
HybridAL	500	$0.903$ $\pm$ $0.003$	$0.829$ $\pm$ $0.011$	$0.667$ $\pm$ $0.014$

G.4 Acquisition Batch Size

We vary $n\in \left\{16,32,64,128\right\}$ on the same three datasets (Table 20). HybridAL is within seed noise of Retrain in 11/12 cells; both methods scale similarly with $n$ ($+0.02$ to $+0.05$ F1 from $n=16$ to $n=128$), so the parity claim is not specific to $n=32$.

媒体内容 · 前往原文查看

Method	$n$	AG News	IMDb	Yahoo
Retrain	16	$0.896$ $\pm$ $0.005$	$0.809$ $\pm$ $0.009$	$0.646$ $\pm$ $0.013$
HybridAL	16	$0.895$ $\pm$ $0.006$	$0.816$ $\pm$ $0.011$	$0.651$ $\pm$ $0.008$
Retrain	32	$0.900$ $\pm$ $0.004$	$0.819$ $\pm$ $0.021$	$0.667$ $\pm$ $0.004$
HybridAL	32	$0.902$ $\pm$ $0.007$	$0.814$ $\pm$ $0.014$	$0.656$ $\pm$ $0.004$∗
Retrain	64	$0.912$ $\pm$ $0.003$	$0.836$ $\pm$ $0.008$	$0.679$ $\pm$ $0.003$
HybridAL	64	$0.911$ $\pm$ $0.003$	$0.834$ $\pm$ $0.005$	$0.668$ $\pm$ $0.009$
Retrain	128	$0.915$ $\pm$ $0.006$	$0.846$ $\pm$ $0.010$	$0.691$ $\pm$ $0.007$
HybridAL	128	$0.916$ $\pm$ $0.005$	$0.852$ $\pm$ $0.004$	$0.687$ $\pm$ $0.010$

G.5 Acquisition Sampler

We compare HybridAL and Retrain across Entropy (the default), Random, and BADGE (Ash et al., 2019) (Table 21). HybridAL is within seed noise of Retrain in 7/9 cells; both exceptions are on Yahoo Answers, where HybridAL is weakest in the main results. Parity holds across uncertainty-based, diversity-based, and random acquisition.

媒体内容 · 前往原文查看

Sampler	Method	AG News	IMDb	Yahoo
Entropy	Retrain	$0.900$ $\pm$ $0.004$	$0.819$ $\pm$ $0.021$	$0.667$ $\pm$ $0.004$
Entropy	HybridAL	$0.902$ $\pm$ $0.007$	$0.814$ $\pm$ $0.014$	$0.656$ $\pm$ $0.004$∗
Random	Retrain	$0.887$ $\pm$ $0.006$	$0.825$ $\pm$ $0.007$	$0.662$ $\pm$ $0.009$
Random	HybridAL	$0.886$ $\pm$ $0.007$	$0.820$ $\pm$ $0.006$	$0.651$ $\pm$ $0.014$
BADGE	Retrain	$0.897$ $\pm$ $0.004$	$0.828$ $\pm$ $0.007$	$0.668$ $\pm$ $0.004$
BADGE	HybridAL	$0.898$ $\pm$ $0.004$	$0.833$ $\pm$ $0.003$	$0.651$ $\pm$ $0.007$∗