ALScope: A Unified Toolkit for Deep Active Learning

Chenkai Wu; Gang Liu; Jueqing Lu; Lan Du; Wray Buntine; Xiaohao Yang; Yuanyuan Qi

REVIEW 3 major objections 5 minor 6 references

ALScope: A Unified Toolkit for Deep Active Learning

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper presents ALScope, a unified deep active learning platform covering 10 datasets and 21 algorithms, and uses it to show that no method dominates across domains or settings, that open-set and imbalanced conditions remain hard, and t

desk verdict ALScope is a genuinely useful DAL testbed spanning CV and NLP with configurable OOD/imbalance settings, but the OOD and imbalance results are undermined by an unspecified test-set protocol that makes those rankings provisional. read the letter →

arxiv 2508.04937 v1 pith:OKJH3AIU submitted 2025-08-06 cs.LG cs.CV

Chenkai Wu , Yuanyuan Qi , Xiaohao Yang , Jueqing Lu , Gang Liu , Wray Buntine , Lan Du This is my paper

classification cs.LGcs.CV

keywords deepactivelearningbenchmarkingopen-setrecognitionclassimbalanceuncertaintysamplingdiversityout-of-distributiondataselection-timeefficiency

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ALScope is a benchmarking platform for deep active learning (DAL), the practice of letting a model choose which unlabeled samples to label next. The paper builds a single codebase that runs 21 DAL algorithms on 10 image and text classification datasets under three conditions: the standard balanced closed set, an open-set setting where 60% of the unlabeled pool comes from unknown classes, and a class-imbalanced setting with a long-tail distribution. Using this platform the authors compare 19 algorithms on seven datasets and report three findings: rankings change substantially across domains and settings, so no algorithm dominates; accuracy drops clearly in open-set and imbalanced conditions, meaning those scenarios are still unsolved; and some of the best-performing methods (BADGE, EntropyCB, NoiseStability) take hours more selection time than the rest. The platform's value is that it makes these comparisons fair and reproducible, and the findings are meant to guide both algorithm choice in practice and the design of future DAL methods.

What carries the argument

The load-bearing object is ALScope itself, a four-module toolkit (control panel, experiment settings, algorithm components, base infrastructure) that constructs each scenario from two knobs: an OOD ratio $\mathrm{OOD\ Ratio} = N_{\mathrm{OOD}}/N_{\mathrm{unlabeled}}$ controlling the fraction of unknown-class samples in the unlabeled pool, and an imbalance ratio $n_i = n_{\max}\,\gamma^{i/(C-1)}$ producing a long-tail class distribution. Evaluation uses final accuracy at the 10th acquisition round averaged over five seeds, an area-under-the-budget-curve metric, and a two-sided paired $t$-test on ten evenly spaced learning-curve points to decide which algorithm significantly beats which.

What would settle it

Run the ALScope OOD experiment on CIFAR-10 with the test set replaced by one containing both known and unknown classes and compute accuracy over the full test set; if the top-ranked methods change, in particular random sampling's 0.570, then the paper's open-set conclusions depend on the unstated choice of test-set composition rather than on the algorithms.

Watch

Extended reading notes

Core claim

The central claim is that a single, modular toolkit can standardize deep active learning evaluation across computer vision and NLP, and that doing so overturns the impression of steady progress given by isolated studies. On ALScope's comparison, classical uncertainty baselines (CONF, Margin, VarRatio) and even random sampling stay competitive with recent specialized methods on standard settings, while hybrid methods like BADGE, ALFA-Mix, and CoreMSE take over on harder datasets such as CIFAR-100 and in imbalanced conditions. Under open-set and imbalanced scenarios the paper finds clear room for improvement: class-balanced core-set variants and some hybrid methods help, but no method consiste

Load-bearing premise

The comparisons are valid only if the test sets are built identically for every algorithm, but the paper never states whether open-set accuracy is computed on known classes only or on known plus unknown classes, nor whether the imbalanced test set follows the same skewed distribution as the training set, so if those details vary the reported accuracies and rankings may not be comparable.

Editorial extensions

If this is right

No universally best DAL algorithm exists: the right choice depends on the domain and on whether the data is closed-set, open-set, or imbalanced, and the choice matters more on harder datasets.
Open-set and class-imbalanced active learning are still open problems: even the best methods underperform their standard-setting results, which is where future research effort should be directed.
Accuracy and selection cost must be traded off at deployment time: BADGE, EntropyCB, and NoiseStability can be one to two orders of magnitude slower than competitors.
Simple baselines (random sampling, CONF, Margin) are strong enough that new DAL methods should be required to beat them under identical test-set conditions.
A common platform should let future algorithms be added and compared under matching conditions, replacing the fragmented codebase comparisons that produced divergent findings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

My inference: the paper's cross-setting comparisons confound query size with setting (for example CIFAR-10 uses a query size of 500 in the standard setting but 100 under OOD), so the ranking shifts across settings may partly reflect budget effects rather than the distribution-shift itself.
My inference: if the open-set accuracy is computed only on known classes, an algorithm that simply avoids labeling unknown-class samples would be unfairly rewarded; a metric covering both known and unknown classes would give a stricter test.
My inference: the unusual result that text OOD accuracy rises as the OOD ratio goes from 0.2 to 0.6 suggests the unknown-class samples may act as a regularizer; a direct test would compare the same algorithms with the OOD samples removed from the unlabeled pool.
My inference: the selection-time figures could be turned into a two-dimensional view of each method (accuracy gain per label versus compute per selection), letting practitioners choose by their own labeling-time-to-compute budget.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

ALScope is a real contribution: a unified deep active learning platform that covers both CV and NLP, integrates 21 algorithms, and adds open-set and imbalanced configurations that prior benchmarks lack. The engineering is solid, the code is public, and the runtime analysis is concrete and actionable—knowing that BADGE, EntropyCB, and NoiseStability can be orders of magnitude slower than baselines is useful. The broad qualitative findings (performance varies across domains, simple baselines like Random are competitive, hybrid methods often do well) are supported by the tables and are consistent with what people in the field already suspect.

The soft spots are real and need to be addressed before the numbers are used as ground truth. Most importantly, the paper never states what test set is used to compute accuracy in the OOD and imbalanced settings. Section 4.3 defines the OOD ratio and imbalance ratio only for the unlabeled pool. In Appendix G, accuracy on text datasets rises monotonically with OOD ratio—SST-5 ALFA-Mix goes 0.399 to 0.473 to 0.629, YELP CoreLog goes 0.437 to 0.569 to 0.695. Under a fixed known-class test set, contaminating the unlabeled pool with 60% OOD should make selection harder, not uniformly improve accuracy across all methods. The paper's explanation that "OOD samples help the model learn more robust representations" is hand-waving, not evidence. Either the test set changes across settings, there is a bug, or there is a surprising effect that needs a controlled demonstration. As written, the OOD and imbalance rankings are not interpretable.

The statistical comparison is also weak: five trials with no error bars, and a pairwise t-test that treats ten correlated points along one learning curve as independent samples. That's not a defensible way to rank methods. Minor point: only 19 of 21 listed algorithms are actually evaluated in the main experiments, and the two missing ones (CCAL, MQ-Net) are exactly the methods designed for open-set settings, which is where the protocol is murkiest.

None of this kills the platform's value. But the empirical claims about non-standard settings need a major revision: specify the test-set construction precisely, report error bars, and either fix the t-test or drop it. The code is public, so the protocol can be verified. I'd send this to peer review—the toolkit deserves referee time—but the accept decision should be conditional on the evaluation protocol being clarified.

Referee Report

3 major / 5 minor

Summary. The paper introduces ALScope, a deep active learning (DAL) benchmark platform covering 10 CV/NLP datasets and 21 DAL algorithms, of which 19 are evaluated. The platform supports standard, open-set (OOD), and class-imbalanced settings through configurable OOD ratio and imbalance ratio. Experiments report final accuracy and AUBC across query sizes, OOD ratios, and imbalance ratios, together with selection-time measurements and pairwise t-test comparison matrices. The paper's main claims are that DAL performance varies strongly across domains and settings, that existing methods show limitations in non-standard scenarios, and that some accurate methods are computationally expensive.

Significance. If the empirical claims hold, ALScope is a useful open-source contribution: it covers more algorithms and more realistic settings than prior platforms (DeepAL+, Baal, Eval-AL, CDALBench), spans both vision and text, and makes code available. The qualitative observations that no single algorithm dominates and that simple baselines such as Random and Margin remain competitive are valuable for practitioners. However, the benchmark's validity currently depends on an unspecified test-set protocol in the OOD and imbalanced settings, and on a statistical comparison that does not use trials as inferential units. These issues are load-bearing for the paper's headline findings and require clarification or additional analysis.

major comments (3)

[4.3] The construction of the OOD and imbalanced test sets is never specified. Equations (1) and (2) define the OOD ratio and imbalance ratio only for the unlabeled pool. The paper does not state whether the accuracy numbers are computed on the original balanced closed-set test set, on a test set rebalanced to the same long-tail distribution, or on a test set augmented with OOD/unknown classes. This makes the OOD and Imb columns of Table 3 and the corresponding Appendix G tables uninterpretable. The pattern in Table 11 — e.g., SST-5 ALFA-Mix accuracy 0.399→0.473→0.629 and YELP CoreLog 0.437→0.569→0.695 as the OOD ratio increases 0.2→0.4→0.6 — is consistent with a changing evaluation distribution rather than with a fixed test set. The explanation in Appendix G.1 that OOD samples 'help the model learn more robust representations' is not supported without specifying what is being measured. The au
[6] Cross-setting performance comparisons are confounded by different query budgets and initial labeled set sizes. In the query-size table in Section 6, CIFAR-10 uses query size 500 for NORMAL but only 100 for OOD and Imb; CIFAR-100 uses 1000 for NORMAL but 500 for OOD and Imb. Appendix B.1 states that the initial training set size is kept equal to the query size. Thus the absolute drops from NORMAL to OOD/Imb in Table 3 reflect, in part, a smaller labeled budget rather than only the effect of distribution shift or class imbalance. The conclusion in Section 7 that DAL algorithms show 'clear limitations' in non-standard scenarios is not supported by these comparisons. Either match query/initial sizes across settings, or explicitly restrict the analysis to within-setting rankings and remove the cross-setting language.
[6.3] The statistical comparison does not use the five trials as the unit of analysis. The t-score formula in Section 6.3 is computed from ten evenly spaced points along a learning curve, with no trial index in the formula; this is a paired test over cycles from a single run, not over independent trials. Learning-curve measurements are correlated and have different means across cycles, so a t-test over cycles does not establish that one algorithm significantly outperforms another. Moreover, no standard deviations over the five trials are reported anywhere in Tables 3–11. Consequently, the win counts in Figure 3 and the derived statements in Appendix F (e.g., ALFA-Mix 'leading with 596 wins') are not statistically grounded. The authors should report per-trial variation and perform inference over trials (or a clearly valid repeated-measures procedure).

minor comments (5)

[3] The statement that implementations are 'primarily based on' official code is vague. Please provide per-algorithm version information and a list of modifications; this matters for a benchmark paper whose goal is fair comparison.
[Table 2] The paper advertises 10 datasets but evaluates only seven. SVHN, MNIST, IMDB, and DBpedia are listed as supported but never used in the reported experiments. Clarify whether results for these datasets exist, or state explicitly that they are supported but not benchmarked.
[G.1] The discussion of the OOD setting for text datasets ('OOD samples helping the model learn more robust representations') is post hoc and unsupported. Either remove the explanation or provide evidence from the protocol.
[C.1] The example open-set command uses --ood-rate 0.2, while the main experiments in Table 3 use OOD ratio 0.6. Align the examples with the reported protocol or clarify that these are illustrative only.
[6.3] The terms 'experiment pairs' and 'Total wins' are not precisely defined. Define Cij in terms of settings, datasets, seeds, and the significance rule.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: ALScope is an empirical benchmark; its findings are measurements, not outputs forced by fitted inputs or self-citations.

full rationale

ALScope does not claim to derive a theoretical result from assumptions. Its central outputs are measured accuracies, AUBCs, runtimes, and pairwise t-test win counts in Tables 3-11. The only equations, OOD Ratio (Eq. 1) and Imbalance Ratio (Eq. 2), define how unlabeled pools are constructed; they do not feed back into the reported accuracy values. The 19 algorithms are implemented primarily from official open-source code (Section 3, Appendix D) and the datasets are standard external corpora, so the benchmark is self-contained against external evidence. Self-citations exist: CoreLog/CoreMSE are from Tan, Du, and Buntine 2023, and the paired t-test methodology in Section 6.3 is adopted from that same paper, whose author list overlaps with the present paper (Du and Buntine). These citations are not load-bearing: the t-test is a standard statistical tool, and the rankings do not reduce to any claim made in the cited paper. Appendix G itself flags that OOD accuracy improvements 'could be explained by the OOD samples helping the model learn more robust representations or the specific nature of the OOD data'; this is an interpretive caveat, not circularity. The unspecified OOD/imbalanced test-set protocol is a validity and comparability concern, not a circular-reasoning defect. Score 2 reflects only the presence of minor, non-load-bearing self-citations; there is no reduction of any prediction to its own input.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

ALScope is an empirical benchmark, so there are no derived physical or mathematical constants. The free parameters listed are experimental protocol choices selected by the authors; they are not fitted to data in a model, but they shape all reported results. The axioms are the unstated assumptions needed for the accuracy tables to support the qualitative conclusions.

free parameters (4)

Query batch size per dataset/setting = CIFAR-10 500/100/100; CIFAR-100 1000/500/500; SST-5 10; YELP-3000 10 (normal/OOD/imbalance)
Chosen by hand based on prior works (Tan et al. 2023, Parvaneh et al. 2022, Ning et al. 2022, Bu and Small 2018) to balance realism and compute. The query size directly affects accuracy and AUBC values and therefore the rankings.
OOD ratio = 0.6 in main Table 3; 0.2, 0.4, 0.6 in appendix
Hand-selected difficulty level for the open-set experiments. It defines the fraction of unknown-class samples in the unlabeled pool (Eq. 1) and directly drives the OOD-setting results.
Imbalance ratio gamma = 0.2 in main Table 3; 0.2, 0.4, 0.6 in appendix
Hand-selected skewness parameter in Eq. 2. It controls the long-tail class distribution and determines the imbalance-setting difficulty.
Initial labeled set size = Equal to query size
A design choice stated in Appendix B.1. It affects the learning curves and AUBC values, and hence the pairwise comparisons.

assumptions (3)

domain assumption The test set is fixed and identical across all algorithms for each dataset and setting
Accuracy comparisons are only meaningful under a common test set. The paper never specifies the test set construction for OOD and imbalanced settings, so this is an unstated assumption.
domain assumption Each algorithm implementation is a faithful port of the original published method
Appendix D states implementations are based on official code. Rankings are invalid if any implementation deviates from the original algorithm or is tuned differently.
domain assumption Final accuracy at the 10th active learning cycle is an adequate summary of DAL performance
The main conclusions in Section 6.1 emphasize final accuracy, while AUBC is only reported in the appendix. The choice of summary metric affects the qualitative findings.

how reviews work

0 comments

Cite this review

Pith. "Pith review of ALScope: A Unified Toolkit for Deep Active Learning." pith.science (2026). https://pith.science/paper/OKJH3AIU

@misc{pith2026250804937,
  author       = {Pith},
  title        = {Pith review of: ALScope: A Unified Toolkit for Deep Active Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OKJH3AIU}},
  note         = {Machine review of arXiv:2508.04937}
}

read the original abstract

Deep Active Learning (DAL) reduces annotation costs by selecting the most informative unlabeled samples during training. As real-world applications become more complex, challenges stemming from distribution shifts (e.g., open-set recognition) and data imbalance have gained increasing attention, prompting the development of numerous DAL algorithms. However, the lack of a unified platform has hindered fair and systematic evaluation under diverse conditions. Therefore, we present a new DAL platform ALScope for classification tasks, integrating 10 datasets from computer vision (CV) and natural language processing (NLP), and 21 representative DAL algorithms, including both classical baselines and recent approaches designed to handle challenges such as distribution shifts and data imbalance. This platform supports flexible configuration of key experimental factors, ranging from algorithm and dataset choices to task-specific factors like out-of-distribution (OOD) sample ratio, and class imbalance ratio, enabling comprehensive and realistic evaluation. We conduct extensive experiments on this platform under various settings. Our findings show that: (1) DAL algorithms' performance varies significantly across domains and task settings; (2) in non-standard scenarios such as imbalanced and open-set settings, DAL algorithms show room for improvement and require further investigation; and (3) some algorithms achieve good performance, but require significantly longer selection time.

Figures

Figures reproduced from arXiv: 2508.04937 by the authors.

**Figure 2.** The average runtime per trial for the 19 AL algo [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

**Figure 4.** A screenshot of an experiment log. MeanSTD Kampffmeyer et al. (Kampffmeyer, Salberg, and Jenssen 2016) proposed this method, which uses the standard deviation across multiple predictions to assess consistency; larger deviations indicate higher uncertainty. BALD Gal et al. (Gal, Islam, and Ghahramani 2017) proposed Bayesian Active Learning by Disagreement (BALD), which employs Bayesian neural networks and measures … view at source ↗

Figures from the paper (5 more)

**Figure 5.** Figure 5: The average runtime per trial for the 19 AL algo [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]

**Figure 6.** Figure 6: The average runtime per trial for the 19 AL algo [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]

**Figure 9.** Figure 9: Pairwise T-Test Matrix for DAL algorithms on text [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]

**Figure 10.** Figure 10: Pairwise T-Test Matrix for DAL algorithms on [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]

**Figure 11.** Figure 11: Pairwise T-Test Matrix for DAL algorithms on [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

6 extracted references · 3 canonical work pages

[3]

Advances in Neural Information Pro- cessing Systems, 35: 31416–31429

Meta-query-net: Resolving purity-informativeness dilemma in open-set active learning. Advances in Neural Information Pro- cessing Systems, 35: 31416–31429. Park, Y .; Choi, W.; Kim, S.; Han, D.-J.; and Moon, J. 2023. Active learning for object detection with evidential deep learning and hi- erarchical uncertainty aggregation. In The Eleventh International...

arXiv 2023
[50]

Uncertainty-based methods like CONF, Margin, and VarRatio also show excellent performance (0.886-0.888). Interestingly, BALD shows relatively weaker performance Dataset CIFAR-10 CIFAR-100 Query Size 100 200 500 200 500 1000 Acc/AUBC Acc/AUBC Acc/AUBC Acc/AUBC Acc/AUBC Acc/AUBC ALFA-Mix 0.473 / 0.353 0.623 / 0.468 0.810 / 0.650 0.204 / 0.135 0.381 / 0.245 ...

work page
[2001]

Figure 4: A screenshot of an experiment log

introduced margin sampling, which measures uncer- tainty by the difference between the top two predicted class probabilities; a smaller margin implies greater uncertainty. Figure 4: A screenshot of an experiment log. MeanSTD Kampffmeyer et al. (Kampffmeyer, Salberg, and Jenssen 2016) proposed this method, which uses the standard deviation across multiple ...

work page 2016
[2022]

Active Learning in Recommendation Systems with Multi-level User Preferences

Class-balanced active learning for image classification. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, 1536–1545. Bu, Y .; and Small, K. 2018. Active learning in recommenda- tion systems with multi-level user preferences. arXiv preprint arXiv:1811.12591. Chen, J.; Ma, B.; Cui, H.; and Xia, Y . 2024. Think Twice Befor...

work page Pith review arXiv 2018
[2024]

Advances in Neural Information Processing Systems, 37: 62875–62911

A cross-domain benchmark for active learning. Advances in Neural Information Processing Systems, 37: 62875–62911. Xie, B.; Li, S.; Guo, Q.; Liu, C.; and Cheng, X. 2023. Annotator: A generic active learning baseline for lidar semantic segmentation. Advances in Neural Information Processing Systems, 36. Yamani, A.; Alyami, A.; Luqman, H.; Ghanem, B.; and Gi...

arXiv 2023
[9927]

IEEE. Ash, J. T.; Zhang, C.; Krishnamurthy, A.; Langford, J.; and Agar- wal, A. 2019. Deep batch active learning by diverse, uncertain gradient lower bounds. arXiv preprint arXiv:1906.03671. Atighehchian, P.; Branchaud-Charron, F.; and Lacoste, A. 2020. Bayesian active learning for production, a systematic study and a reusable library. arXiv preprint arXi...

arXiv 2019

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [3]

Advances in Neural Information Pro- cessing Systems, 35: 31416–31429

Meta-query-net: Resolving purity-informativeness dilemma in open-set active learning. Advances in Neural Information Pro- cessing Systems, 35: 31416–31429. Park, Y .; Choi, W.; Kim, S.; Han, D.-J.; and Moon, J. 2023. Active learning for object detection with evidential deep learning and hi- erarchical uncertainty aggregation. In The Eleventh International...

arXiv 2023

[2] [50]

Uncertainty-based methods like CONF, Margin, and VarRatio also show excellent performance (0.886-0.888). Interestingly, BALD shows relatively weaker performance Dataset CIFAR-10 CIFAR-100 Query Size 100 200 500 200 500 1000 Acc/AUBC Acc/AUBC Acc/AUBC Acc/AUBC Acc/AUBC Acc/AUBC ALFA-Mix 0.473 / 0.353 0.623 / 0.468 0.810 / 0.650 0.204 / 0.135 0.381 / 0.245 ...

work page

[3] [2001]

Figure 4: A screenshot of an experiment log

introduced margin sampling, which measures uncer- tainty by the difference between the top two predicted class probabilities; a smaller margin implies greater uncertainty. Figure 4: A screenshot of an experiment log. MeanSTD Kampffmeyer et al. (Kampffmeyer, Salberg, and Jenssen 2016) proposed this method, which uses the standard deviation across multiple ...

work page 2016

[4] [2022]

Active Learning in Recommendation Systems with Multi-level User Preferences

Class-balanced active learning for image classification. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, 1536–1545. Bu, Y .; and Small, K. 2018. Active learning in recommenda- tion systems with multi-level user preferences. arXiv preprint arXiv:1811.12591. Chen, J.; Ma, B.; Cui, H.; and Xia, Y . 2024. Think Twice Befor...

work page Pith review arXiv 2018

[5] [2024]

Advances in Neural Information Processing Systems, 37: 62875–62911

A cross-domain benchmark for active learning. Advances in Neural Information Processing Systems, 37: 62875–62911. Xie, B.; Li, S.; Guo, Q.; Liu, C.; and Cheng, X. 2023. Annotator: A generic active learning baseline for lidar semantic segmentation. Advances in Neural Information Processing Systems, 36. Yamani, A.; Alyami, A.; Luqman, H.; Ghanem, B.; and Gi...

arXiv 2023

[6] [9927]

IEEE. Ash, J. T.; Zhang, C.; Krishnamurthy, A.; Langford, J.; and Agar- wal, A. 2019. Deep batch active learning by diverse, uncertain gradient lower bounds. arXiv preprint arXiv:1906.03671. Atighehchian, P.; Branchaud-Charron, F.; and Lacoste, A. 2020. Bayesian active learning for production, a systematic study and a reusable library. arXiv preprint arXi...

arXiv 2019