Pith. sign in

REVIEW 3 major objections 6 minor 78 references

Enhancing Cost Efficiency in Active Learning with Candidate Set Query

T0 review · 3 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A new active-learning query design claims to cut labeling cost by up to 54% on standard benchmarks by having annotators choose from a small candidate set selected via conformal prediction, with no loss in accuracy.

desk verdict A genuinely new cost-efficient query design for active learning, but the headline savings are conditional on a log-cost model the user study only partially validates; deserves a serious referee and conditional acceptance. read the letter →

arxiv 2502.06209 v2 pith:VQ4HYJLL submitted 2025-02-10 cs.LG cs.CV

Yeho Gwon , Sehyun Hwang , Hoyoung Kim , Jungseul Ok , Suha Kwak This is my paper · ORCID
classification cs.LGcs.CV MSC 68T05
keywords activelearningcandidatesetqueryconformalpredictionannotationcostcost-awareacquisitionclassificationlabelingdesign
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Active learning usually asks an annotator to pick one class from the full list of $L$ classes, and the paper argues this is needlessly expensive. It proposes candidate set query (CSQ): first show a small set of classes the current model thinks are plausible, along with a "None of the above" option, and only if the model was wrong ask the annotator to search the remaining classes. Because choosing among $k$ options costs roughly $\log_2(k)$ seconds, a small candidate set can cut annotation time sharply even when misses happen. The paper proves CSQ is cheaper in expectation whenever the candidate set is small enough relative to its miss rate, and shows that conformal prediction can build such sets with a user-set coverage guarantee while an optimized error rate and a cost-aware acquisition function keep the savings growing as the model improves. On CIFAR-10, CIFAR-100, and ImageNet64x64 the method reaches the same accuracy as the conventional query at 43%, 54%, and 48% of the labeling cost.

What carries the argument

The load-bearing object is the candidate set query and its cost equation $\Gamma(Y(x),y) = \log_2(k+1)$ when $y \in Y(x)$, and $\Gamma(Y(x),y) = \log_2(k+1) + \log_2(L-k)$ otherwise, together with Theorem 3.1 giving the strict-cheapness condition. Around it sit three mechanisms: conformal prediction with scores $1 - P_\theta(y\mid x)$ and calibration quantile $\hat{Q}(\alpha)$, producing candidate sets $\{y : P_\theta(y\mid x) \ge 1 - \hat{Q}(\alpha)\}$ that carry a guaranteed ground-truth inclusion rate; a per-round optimization $\alpha^* = \arg\min_\alpha \sum_{(x,y)\in D_{\mathrm{cal}}} \Gamma(\hat{Y}_\theta(x,\alpha), y)$ that tunes the miss-rate versus candidate-size trade-off automatically; and the cost-aware acquisition function in Eq. (8), which replaces uniform-cost sample selection with information-per-cost selection. The theorem shows the query is cheaper whenever the candidate set is small compared with $L$ and misses rarely enough.

What would settle it

Run the same user study with candidate sets of size 64, 128, 256, 512, and 1000 on a 1000-class task and fit time per query against $\log_2(k+1)$; if the fitted curve grows faster than logarithmic, or if the cost of the follow-up full-class query is closer to $\log_2 L$ than to $\log_2(L-k)$, the cost savings claimed in Theorem 3.1 evaporate at scale.

Watch

Extended reading notes

Core claim

The paper's central claim is that switching the annotation interface from "choose among all $L$ classes" to "choose among a small candidate set, else look in the rest" reduces expected labeling cost whenever $\log_2(k+1)/\log_2 L < 1 - \alpha$, where $k$ is the candidate set size and $\alpha$ is the probability that the candidate set misses the ground-truth class. With $k=1$, this means CSQ beats the conventional query once top-1 accuracy exceeds 30.1% on 10-class tasks, 15.1% on 100-class tasks, and 10.0% on 1000-class tasks. The paper realizes this in a pipeline: conformal prediction produces per-sample candidate sets with coverage at least $1-\alpha$; the error rate $\alpha$ is re-optimized each round by minimizing the expected labeling cost on a calibration set; and a cost-aware acquisition function $g_{\mathrm{cost}}(x) = (1 + g_{\mathrm{score}}(x))^d \,/\, (\log_2(k+1) + \alpha^* \log_2(L-k))$ selects samples with high information gain per unit cost. The empirical result is that the same model accuracy is reached at roughly half the labeling cost on three image benchmarks, with the cost saving growing as model accuracy rises.

Load-bearing premise

The entire cost accounting rests on the assumption that deciding among $k$ classes costs $\log_2(k)$ (or $\log_2(k+1)$ with the escape option), a trend measured on only 20 images and 40 annotators; if real annotation time grows faster than logarithmically or the follow-up full-class query costs more than assumed, the cost savings shrink or disappear.

Editorial extensions

If this is right

  • The same accuracy as a conventional active learner is reached at about half the labeling cost on CIFAR-10, CIFAR-100, and ImageNet64x64, and the savings grow in later rounds as the model improves and candidate sets shrink.
  • Any existing acquisition score, including entropy, BADGE, ProbCover, and SAAL, can be converted to a cost-aware score through Eq. (8), so the framework can be dropped into current active-learning pipelines without changing the training loss.
  • The benefit scales with class count: the 48% saving on 1000-class ImageNet64x64 and a 65.6 percentage-point cost reduction on a 52-class text task suggest the largest gains appear where conventional queries are most expensive.
  • Because CSQ produces complete labels rather than weak supervision, it trains with standard classification losses and avoids the partial-label machinery needed by one-bit or top-1 query designs.
  • The candidate set sizes decrease automatically as the model improves, so the per-sample labeling cost falls over successive rounds rather than staying fixed.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The logarithmic cost model is validated only at candidate set sizes $k = 4, 8, 16, 32$ on 20 images; real annotation interfaces with scrolling or hierarchical menus may make large-$k$ search super-logarithmic, which would shrink the claimed savings at the 1000-class scale the paper extrapolates to.
  • The per-round $\alpha^*$ is chosen on a calibration set drawn from the actively selected batch; if the acquisition function skews that batch toward outliers, coverage could degrade on the true pool even though the guarantee holds exchangeably on the calibration distribution.
  • The paper itself notes in Section 5 that its acquisition function has no label-complexity guarantee; a theory bounding the cost needed to reach a target accuracy is still open.
  • One natural extension left implicit is that when the candidate set misses, the annotator's full-class answer could be treated as a cheap rejected pseudo-label, possibly avoiding the second query entirely and converting misses into additional training signal.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes candidate set query (CSQ) for active learning in classification. Instead of showing all L class labels, the annotator sees a small candidate set built by conformal prediction; if the true label is absent, a follow-up query over the remaining classes is issued. The paper models annotation cost as log2(k+1) for the first stage plus log2(L-k) on a miss, derives a condition (Theorem 3.1) under which CSQ is cheaper than the conventional query, automatically selects the conformal error rate alpha to minimize expected cost on a calibration set, and introduces a cost-aware acquisition function g_cost that divides an information score by the expected CSQ cost. Experiments on CIFAR-10, CIFAR-100, ImageNet64x64, R52, and Products-10K report reduced labeling cost (43%, 54%, and 48% on the three vision datasets at matched accuracy) compared to conventional query with the same acquisition function and training budget.

Significance. If the log-cost model is accepted, CSQ is a simple and generally applicable query design that preserves full supervision, unlike one-bit or n-ary queries; it combines conformal prediction with active learning in a natural way, and the paper demonstrates compatibility with several acquisition functions. The empirical study is extensive (three vision datasets, text, a 10K-class dataset, ablations, label noise, class imbalance), and the user study is a constructive attempt to ground the cost model. However, the headline cost savings are denominated in the paper's own log-cost units, and the user study validates only part of that model (selecting from k options) at small k, leaving the two-stage CSQ protocol and large-k extrapolation unsupported. The stress-test concern about the cost model is therefore well-founded: the reported 43%, 54%, and 48% reductions are properly described as savings under the paper's cost model rather than as directly measured annotation-time savings.

major comments (3)
  1. [Sec. 3.1, Eq. (1); Appendix A, Table 4] The empirical validation of the cost model is incomplete for the actual CSQ protocol. The user study measures the time to select the correct class from k options (k = 4, 8, 16, 32), but the first stage of CSQ presents k candidates plus a 'None of the above' option, and a miss triggers a follow-up search over L - k classes. The study never measures either of these two components, yet Eq. (1) charges log2(k + 1) and log2(L - k) for them. Since all headline savings (43%, 54%, 48%) are computed in these units, a different but plausible cost for absence verification or for the follow-up search would shrink or erase the reported gains. The authors should either run a user study on the full two-stage CSQ query (including miss cases and larger k), or report the savings under alternative cost models (e.g., linear in k, or log2(k+1) with a different slope) to show that the qualitative conclusion is robust.
  2. [Sec. 3.2, Eq. (6); Sec. 3.3, Eq. (8)] The acquisition function treats alpha* as the per-sample probability that the ground-truth class is absent from the candidate set. Conformal prediction, however, only provides a marginal coverage guarantee (Eq. (6)) over the calibration distribution, not conditional coverage for the samples selected by the acquisition function. Because the acquisition function preferentially selects high-uncertainty examples, the selected batch may have a miss probability larger than alpha*, making the denominator of Eq. (8) an underestimate and potentially biasing sample selection toward expensive queries. The paper should measure the empirical ground-truth inclusion rate on the actually selected batches in each round (as in Fig. 5c) for the full pipeline with alpha*, and, if the rate is below 1 - alpha*, adapt the cost estimate or the candidate set accordingly.
  3. [Sec. 4.1 (relative labeling cost); Eq. (7); Eq. (8)] The evaluation metric (relative labeling cost) is the same cost function that the method optimizes: alpha* is chosen by minimizing Gamma in Eq. (7), the acquisition function g_cost in Eq. (8) divides by that cost, and the reported savings are then computed with the same Gamma. This makes the measured improvement partly self-referential and means that the 43%, 54%, and 48% figures cannot be interpreted as actual annotation-time savings unless the cost model is independently validated. In addition to the user-study extension requested above, the paper should report model-free efficiency indicators - such as average candidate-set size, miss rate, and number of follow-up queries in the main experiments - so that the reader can assess the benefit without accepting the log-cost model.
minor comments (6)
  1. [Appendix A] Appendix A cites 'Hu et al., 2018' for the theoretical cost curves, but the reference list shows Hu et al., 2020 (One-bit supervision) as the source of the log2 L cost model; update the citation.
  2. [Table 1 and Sec. 4.2] Table 1 reports annotation times of 69.4, 91.5, 116.9, and 166.9 seconds, which appear to be totals for 20 images; the caption and text should state whether these are totals or per-query values, since the text below refers to a user study with 40 annotators and the times are more interpretable per query (3.47, 5.20, 6.94, 8.35 s).
  3. [Sec. 3.1] The sentence 'we model the cost of selecting one label from k candidates as log2 k' is followed by Eq. (1) charging log2(k+1) even when the label is present; clarify that the first stage has k+1 choices including 'None of the above'.
  4. [Algorithm 1] Algorithm 1 computes sample-wise cost in step 3 but constructs candidate sets only in step 5; specify that step 3 uses the previous round's quantile and alpha (or restructure the algorithm so cost estimation is defined after candidate-set construction).
  5. [Sec. 4.2] Section 4.2 states that CSQ reduces the labeling cost by 43%, 54%, and 48% on CIFAR-10, CIFAR-100, and ImageNet64x64, but it is not immediately clear which acquisition function these numbers correspond to; identify the specific CSQ variant in the main text (e.g., CSQ+Cost(Ent) at the final round).
  6. [Figure 1 caption] Figure 1 caption says the user study involved 40 participants, while Appendix A describes 44 annotators divided into four groups and then excluding one outlier per group; reconcile the numbers and state the exclusion rule in the figure caption.

Circularity Check

2 steps flagged · score 4.0 of 10

Partial circularity: the guarantee that CSQ is at least as efficient as the conventional query follows by construction from optimizing the same cost model used for evaluation, with the conventional query as the alpha=0 case.

  1. fitted input called prediction [Sec. 3.2, 'Cost-optimized error rate selection', Eq. (7) and following paragraph; with Eq. (1) and Sec. 4.1]
    "Since the optimization in Eq. (7) naturally considers the conventional query as a special case of CSQ at α = 0, CSQ is at least as efficient as, and in general more efficient than, the conventional query."

    Eq. (7) defines α* as the minimizer over α of the sum of Γ(Ŷθ(x,α),y), where Γ is the labeling cost in Eq. (1), and the paper states that Ŷθ(x,0)=Y, i.e., α=0 is exactly the conventional query. Therefore the claim that CSQ is at least as efficient as the conventional query is a tautology: an optimizer over a set containing α=0 cannot do worse than α=0 under the same objective. The subsequent relative-labeling-cost results (Sec. 4.1) are also measured in units of Eq. (1), the very cost model being optimized, so the 'at least as efficient' margin is guaranteed by construction rather than by the user study or conformal coverage.

  2. self definitional [Sec. 3.3, Eq. (8) and the paragraph immediately after it]
    "The denominator is an expected cost derived from our cost model (Eq. (1)), considering two cases: the correct label is included or excluded from the candidate set."

    The proposed acquisition function (Eq. (8)) divides the information score by the expected cost computed from Eq. (1), and the reported savings (relative labeling cost, Sec. 4.1) are computed with the same Eq. (1). The method therefore selects samples that are cheap under the evaluation metric and then reports that those samples are cheap under that same metric. This is an optimization of the evaluation objective rather than an independent measurement of human annotation time; the external user study (Table 4) only calibrates the k-option selection time, not the 'None of the above' branch or the L−k follow-up query used in Eq. (1).

full rationale

The paper is not wholly circular. Conformal prediction is an externally established coverage guarantee (Eq. (6)), the user study is an independent (though small) empirical check, and the specific 43/54/48% cost reductions depend on measured model accuracy and candidate-set sizes, so those magnitudes are not forced by definition. However, two load-bearing components are self-referential. First, Sec. 3.2 optimizes α in Eq. (7) using the same cost function Γ (Eq. (1)) that later defines all reported labeling costs, and it explicitly notes that α=0 is the conventional query; hence the blanket statement that CSQ is at least as efficient as the conventional query is true by construction, not by experiment. Second, the cost-efficient acquisition function (Eq. (8)) uses the expected cost from Eq. (1) as its denominator, while the evaluation also uses Eq. (1), so the measured advantage of Cost(·) partly reflects optimizing the evaluation metric. The calibration/coverage comparison of α* to the 'true optimal α' (Fig. 6b) is likewise an oracle comparison inside the same cost model. The user-study gap (no 'None of the above' option, no follow-up query, k≤32) is a real external-validity risk for the headline percentages, but it is a measurement limitation rather than a circularity. On balance: one by-construction guarantee and one self-referential acquisition/evaluation pairing, with the central accuracy-vs-cost curves still containing independent information, warrant a score of 4.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or mathematical entities. It introduces a new query protocol (CSQ), which is a procedural invention, not a postulated entity. The main assumptions are the cost model and the validity of conformal prediction in the active learning loop. The free parameters are few and mostly validated by ablations, but the cost model itself is the weakest external input.

free parameters (3)
  • d = 0.3
    The hyperparameter controlling the influence of the acquisition score in Eq. (8) is set to 0.3 for all datasets and acquisition functions in the main experiments; the ablation in Figure 11 shows that performance is fairly insensitive to d, so this is a mild free parameter.
  • ncal = 500 (CIFAR), 5K (ImageNet64x64)
    The calibration set size is chosen by the authors; the ablation in Figure 4b shows the method is robust to this choice (less than 2%p variation), so this is a mild free parameter.
  • alpha* = selected per round via Eq. (7)
    The error rate alpha is optimized on the calibration set to minimize the paper's own cost model. This is a fitted value, but it is fitted to a calibration set and the paper shows it tracks the true optimal alpha. It is a legitimate hyperparameter in the method, not a post-hoc fit to the test set.
assumptions (4)
  • domain assumption The labeling cost of choosing among k options is log2(k) (information-theoretic cost model of Hu et al. 2020).
    This is the foundation of the entire cost analysis. The user study provides weak empirical support (4 data points, correlation 0.97), but it is an assumption inherited from Hu et al. (2020) and not independently validated at scale.
  • domain assumption The candidate set query protocol's cost is log2(k+1) plus, in the miss case, log2(L-k), and the annotator behaves according to this two-stage protocol.
    This cost model assumes the annotator first searches the candidate set and then searches the remaining classes if needed, without any additional cost for switching. Real annotator behavior may differ.
  • domain assumption The calibration set is exchangeable with the unlabeled pool and the model is fixed when the conformal quantile is computed.
    Standard conformal prediction assumption. The paper handles the active learning setting by collecting a calibration set from the actively selected data and using the previous round's quantile for the current round's queries.
  • domain assumption The acquisition score gscore(x) is an adequate measure of information gain, and the combination of (1+gscore(x))^d with the inverse cost estimate is a sensible acquisition function.
    The paper relies on the existing AL literature for the information gain measure, and introduces a heuristic combination with cost. The power d=0.3 is an empirical choice.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Enhancing Cost Efficiency in Active Learning with Candidate Set Query." pith.science (2026). https://pith.science/paper/VQ4HYJLL

@misc{pith2026250206209,
  author       = {Pith},
  title        = {Pith review of: Enhancing Cost Efficiency in Active Learning with Candidate Set Query},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VQ4HYJLL}},
  note         = {Machine review of arXiv:2502.06209}
}
read the original abstract

This paper introduces a cost-efficient active learning (AL) framework for classification, featuring a novel query design called candidate set query. Unlike traditional AL queries requiring the oracle to examine all possible classes, our method narrows down the set of candidate classes likely to include the ground-truth class, significantly reducing the search space and labeling cost. Moreover, we leverage conformal prediction to dynamically generate small yet reliable candidate sets, adapting to model enhancement over successive AL rounds. To this end, we introduce an acquisition function designed to prioritize data points that offer high information gain at lower cost. Empirical evaluations on CIFAR-10, CIFAR-100, and ImageNet64x64 demonstrate the effectiveness and scalability of our framework. Notably, it reduces labeling cost by 48% on ImageNet64x64. The project page can be found at https://yehogwon.github.io/csq-al.

Figures

Figures reproduced from arXiv: 2502.06209 by the authors.

Figure 1
Figure 1. Conventional query versus candidate set query. ( [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Accuracy (%) versus relative labeling cost (%) for conventional query ( [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Average size of the candidate set and accuracy (%) of our method with cost-efficient entropy [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (17 more)
Figure 4
Figure 4. Figure 4: (a) Contribution of each component of our method, measured by accuracy (%) versus relative [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Impact of the candidate set design evaluated on CIFAR-100 using the conventional query with all [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Impact of cost-optimized error rate selection as in Eq. (7), evaluated on CIFAR-100 with entropy [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Wall-clock time versus unlabeled pool size on ImageNet64x64. CSQ (dashed) runs in con￾stant time, independent of pool size. Our sampling (Cost(Ent)) adds a negligible overhead, keeping the same linear complexity as plain entropy sampling [PITH_FULL_IMAGE:figures/full_…
Figure 8
Figure 8. Figure 8: Examples of input images and their corresponding candidate sets constructed using our method [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: Questionnaire and examples used in the user study. (a) Each question contains an instruction, [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: Comparison of different sampling methods and their cost-sampling variants on CIFAR-100. Each [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]
Figure 11
Figure 11. Figure 11: Accuracy (%) versus relative labeling cost (%) with varying hyperparameter [PITH_FULL_IMAGE:figures/full_fig_p022_11.png]
Figure 12
Figure 12. Figure 12: Accuracy (%) versus relative labeling cost (%) with varying hyperparameter [PITH_FULL_IMAGE:figures/full_fig_p022_12.png]
Figure 13
Figure 13. Figure 13: (a) Accuracy (%) and (b) Macro-F1 (%) versus relative labeling cost (%) with varying hyperpa [PITH_FULL_IMAGE:figures/full_fig_p023_13.png]
Figure 14
Figure 14. Figure 14: Comparison between conventional query (CQ) and candidate set query (CSQ) with random sam￾pling (Rand), entropy sampling (Ent), and cost-efficient entropy sampling (Cost(Ent)) on text classification task with R52 dataset. We adopt (a) TF-IDF (Manning, 2009) and (b) RoB…
Figure 15
Figure 15. Figure 15: Comparison between conventional query (CQ) and candidate set query (CSQ) with entropy sam￾pling (Ent) and the proposed acquisition function with entropy measure (Cost(Ent) on CIFAR-100 with label noise across AL rounds with varying noise level: (a) Noise rate of 0.05.…
Figure 16
Figure 16. Figure 16: Comparison between conventional query (CQ) and candidate set query (CSQ) with entropy sam￾pling (Ent) and the proposed acquisition function with entropy measure (Cost(Ent) on CIFAR-100-LT, a variant of CIFAR-100 with class imbalance, across AL rounds with varying imba…
Figure 17
Figure 17. Figure 17: Comparison between conventional query combined with entropy sampling ( [PITH_FULL_IMAGE:figures/full_fig_p026_17.png]
Figure 18
Figure 18. Figure 18: Comparison of conventional query (CQ), top-1 query (Top1) (Zhang et al., 2022), and candidate set query (CSQ) combined with entropy sampling on CIFAR-100. Top1 asks the oracle whether the top-1 prediction is the ground-truth class, potentially yielding partial labels.…
Figure 19
Figure 19. Figure 19: Accuracy (%) versus relative labeling cost (%) on CIFAR-10. [PITH_FULL_IMAGE:figures/full_fig_p027_19.png]
Figure 20
Figure 20. Figure 20: Accuracy (%) versus relative labeling cost (%) on CIFAR-100. [PITH_FULL_IMAGE:figures/full_fig_p027_20.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

78 extracted references · 63 canonical work pages

  1. [1]

    Conformal prediction: A gentle introduction

    Anastasios N Angelopoulos, Stephen Bates, et al. Conformal prediction: A gentle introduction. Foundations and Trends in Machine Learning , 16 0 (4): 0 494--591, 2023

    2023

  2. [2]

    Uncertainty sets for image classifiers using conformal prediction

    Anastasios Nikolas Angelopoulos, Stephen Bates, Michael Jordan, and Jitendra Malik. Uncertainty sets for image classifiers using conformal prediction. In Proc. International Conference on Machine Learning (ICML), 2020

    work page 2020

  3. [3]

    Estimating annotation cost for active learning in a multi-annotator environment

    Shilpa Arora, Eric Nyberg, and Carolyn Rose. Estimating annotation cost for active learning in a multi-annotator environment. In Proceedings of the NAACL HLT 2009 Workshop on Active Learning for Natural Language Processing, pp.\ 18--26, 2009

    work page 2009

  4. [4]

    Deep active learning for dialogue generation

    Nabiha Asghar, Pascal Poupart, Xin Jiang, and Hang Li. Deep active learning for dialogue generation. In Proceedings of the 6th Joint Conference on Lexical and Computational Semantics (* SEM ) , 2017

    work page 2017

  5. [5]

    Deep batch active learning by diverse, uncertain gradient lower bounds

    Jordan T Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. In Proc. International Conference on Learning Representations (ICLR), 2020

    work page 2020

  6. [6]

    Products-10k: A large-scale product recognition dataset

    Yalong Bai, Yuxiang Chen, Wei Yu, Linfang Wang, and Wei Zhang. Products-10k: A large-scale product recognition dataset. arXiv preprint arXiv:2008.10545, 2020

    arXiv 2008

  7. [7]

    Active learning with n-ary queries for image recognition

    Aditya R Bhattacharya and Shayok Chakraborty. Active learning with n-ary queries for image recognition. In Proc. IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2019

    work page 2019

  8. [8]

    Seong Jin Cho, Gwangsu Kim, Junghyun Lee, Jinwoo Shin, and Chang D. Yoo. Querying easily flip-flopped samples for deep active learning. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=THUBTfSAS2

    work page 2024

Show all 78 references
  1. [9]

    A downsampled variant of imagenet as an alternative to the cifar datasets

    Patryk Chrabaszcz, Ilya Loshchilov, and Frank Hutter. A downsampled variant of imagenet as an alternative to the cifar datasets. arXiv preprint arXiv:1707.08819, 2017

    2017 arXiv

  2. [10]

    Support-vector networks

    Corinna Cortes. Support-vector networks. Machine Learning, 1995

    1995

  3. [11]

    Learning from partial labels

    Timothee Cour, Ben Sapp, and Ben Taskar. Learning from partial labels. In Journal of Machine Learning Research (JMLR), 2011

    2011

  4. [12]

    Class-balanced loss based on effective number of samples

    Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp.\ 9268--9277, 2019

    2019

  5. [13]

    Two faces of active learning

    Sanjoy Dasgupta. Two faces of active learning. Theoretical computer science, 412 0 (19): 0 1767--1781, 2011

    2011

  6. [14]

    ImageNet: a large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: a large-scale hierarchical image database. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2009

    2009

  7. [15]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at...

    2021

  8. [16]

    Contrastive coding for active learning under class distribution mismatch

    Pan Du, Suyun Zhao, Hui Chen, Shuwen Chai, Hong Chen, and Cuiping Li. Contrastive coding for active learning under class distribution mismatch. In Proc. IEEE/CVF International Conference on Computer Vision (ICCV), pp.\ 8927--8936, 2021

    2021

  9. [17]

    Data determines distributional robustness in contrastive language image pre-training (clip)

    Alex Fang, Gabriel Ilharco, Mitchell Wortsman, Yuhao Wan, Vaishaal Shankar, Achal Dave, and Ludwig Schmidt. Data determines distributional robustness in contrastive language image pre-training (clip). In Proc. International Conference on Machine Learning (ICML), pp.\ 6216--623...

    2022

  10. [18]

    Classification in the presence of label noise: a survey

    Beno \^ t Fr \'e nay and Michel Verleysen. Classification in the presence of label noise: a survey. IEEE transactions on neural networks and learning systems, 25 0 (5): 0 845--869, 2013

    2013

  11. [19]

    a ger, Bertrand Charpentier, Antonio Oroz, and Stephan G \

    Dominik Fuchsgruber, Tom Wollschl \"a ger, Bertrand Charpentier, Antonio Oroz, and Stephan G \"u nnemann. Uncertainty for active learning on graphs. In Forty-first International Conference on Machine Learning (ICML), 2024

    2024

  12. [20]

    How to select which active learning strategy is best suited for your specific problem and budget

    Guy Hacohen and Daphna Weinshall. How to select which active learning strategy is best suited for your specific problem and budget. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems (NeurIPS), volum...

    2023

  13. [21]

    Active learning on a budget: Opposite strategies suit high and low budgets

    Guy Hacohen, Avihu Dekel, and Daphna Weinshall. Active learning on a budget: Opposite strategies suit high and low budgets. In International Conference on Machine Learning (ICML), pp.\ 8175--8195. PMLR, 2022

    2022

  14. [22]

    Theory of disagreement-based active learning

    Steve Hanneke et al. Theory of disagreement-based active learning. Foundations and Trends in Machine Learning , 7 0 (2-3): 0 131--309, 2014

    2014

  15. [23]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016

    2016

  16. [24]

    Towards better uncertainty sampling: Active learning with multiple views for deep convolutional neural network

    Tao He, Xiaoming Jin, Guiguang Ding, Lan Yi, and Chenggang Yan. Towards better uncertainty sampling: Active learning with multiple views for deep convolutional neural network. In IEEE International Conference on Multimedia and Expo (ICME), 2019

    2019

  17. [25]

    A survey on cost types, interaction schemes, and annotator performance models in selection algorithms for active learning in classification

    Marek Herde, Denis Huseljic, Bernhard Sick, and Adrian Calma. A survey on cost types, interaction schemes, and annotator performance models in selection algorithms for active learning in classification. IEEE Access, 9: 0 166970--166989, 2021

    2021

  18. [26]

    Deep learning scaling is predictable, empirically

    Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00409, 2017

    2017 arXiv

  19. [27]

    One-bit supervision for image classification

    Hengtong Hu, Lingxi Xie, Zewei Du, Richang Hong, and Qi Tian. One-bit supervision for image classification. Proc. Neural Information Processing Systems (NeurIPS), 33: 0 501--511, 2020

    2020

  20. [28]

    Squeeze-and-excitation networks

    Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018

    2018

  21. [29]

    Multi-label active learning: query type matters

    Sheng-Jun Huang, Songcan Chen, and Zhi-Hua Zhou. Multi-label active learning: query type matters. In Proc. International Joint Conference on Artificial Intelligence (IJCAI), volume 15, pp.\ 946--952, 2015

    2015

  22. [30]

    Combating label distribution shift for active domain adaptation

    Sehyun Hwang, Sohyun Lee, Sungyeon Kim, Jungseul Ok, and Suha Kwak. Combating label distribution shift for active domain adaptation. In Proc. European Conference on Computer Vision (ECCV), pp.\ 549--566. Springer, 2022

    2022

  23. [31]

    Active learning for semantic segmentation with multi-class label query

    Sehyun Hwang, Sohyun Lee, Hoyoung Kim, Minhyeon Oh, Jungseul Ok, and Suha Kwak. Active learning for semantic segmentation with multi-class label query. Advances in Neural Information Processing Systems (NeurIPS), 36, 2023

    2023

  24. [32]

    Breaking the interactive bottleneck in multi-class classification with active selection and binary feedback

    Ajay J Joshi, Fatih Porikli, and Nikolaos Papanikolopoulos. Breaking the interactive bottleneck in multi-class classification with active selection and binary feedback. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2010

    2010

  25. [33]

    Active learning with complementary sampling for instructing class-biased multi-label text emotion classification

    Xin Kang, Xuefeng Shi, Yunong Wu, and Fuji Ren. Active learning with complementary sampling for instructing class-biased multi-label text emotion classification. IEEE Transactions on Affective Computing, 14 0 (1): 0 523--536, 2020

    2020

  26. [34]

    Active label correction for semantic segmentation with foundation models

    Hoyoung Kim, Sehyun Hwang, Suha Kwak, and Jungseul Ok. Active label correction for semantic segmentation with foundation models. In Proc. International Conference on Machine Learning (ICML), 2024

    2024

  27. [35]

    Saal: sharpness-aware active learning

    Yoon-Yeong Kim, Youngjae Cho, JoonHo Jang, Byeonghu Na, Yeongmin Kim, Kyungwoo Song, Wanmo Kang, and Il-Chul Moon. Saal: sharpness-aware active learning. In Proc. International Conference on Machine Learning (ICML), 2023

    2023

  28. [36]

    Nlnl: Negative learning for noisy labels

    Youngdong Kim, Junho Yim, Juseung Yun, and Junmo Kim. Nlnl: Negative learning for noisy labels. In Proceedings of the IEEE/CVF international conference on computer vision (CVPR), pp.\ 101--110, 2019

    2019

  29. [37]

    Segment anything

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proc. IEEE International Conference on Computer Vision (ICCV), pp.\ 4015--4026, 2023

    2023

  30. [38]

    Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning

    Andreas Kirsch, Joost Van Amersfoort, and Yarin Gal. Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning. Proc. Neural Information Processing Systems (NeurIPS), 32, 2019

    2019

  31. [39]

    Similar: Submodular information measures based active learning in realistic scenarios

    Suraj Kothawade, Nathan Beck, Krishnateja Killamsetty, and Rishabh Iyer. Similar: Submodular information measures based active learning in realistic scenarios. Proc. Neural Information Processing Systems (NeurIPS), 34: 0 18685--18697, 2021

    2021

  32. [40]

    Active learning for cost-sensitive classification

    Akshay Krishnamurthy, Alekh Agarwal, Tzu-Kuo Huang, Hal Daum \'e III, and John Langford. Active learning for cost-sensitive classification. Journal of Machine Learning Research (JMLR), 20 0 (65): 0 1--50, 2019

    2019

  33. [41]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009

    2009

  34. [42]

    David D. Lewis. Reuters-21578 text categorization test collection, 1997. URL http://www.daviddlewis.com/resources/testcollections/reuters21578/

    1997

  35. [43]

    Generative adversarial active learning for unsupervised outlier detection

    Yezheng Liu, Zhe Li, Chong Zhou, Yuanchun Jiang, Jianshan Sun, Meng Wang, and Xiangnan He. Generative adversarial active learning for unsupervised outlier detection. IEEE Transactions on Knowledge and Data Engineering, 32 0 (8): 0 1517--1528, 2019 a

    2019

  36. [44]

    Roberta: A robustly optimized bert pretraining approach

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019 b

    1907 arXiv

  37. [45]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In Proc. International Conference on Learning Representations (ICLR), 2019

    2019

  38. [46]

    An introduction to information retrieval

    Christopher D Manning. An introduction to information retrieval. 2009

    2009

  39. [47]

    Conformal prediction based active learning by linear regression optimization

    Sergio Matiz and Kenneth E Barner. Conformal prediction based active learning by linear regression optimization. Neurocomputing, 388: 0 157--169, 2020

    2020

  40. [48]

    Active learning for open-set annotation

    Kun-Peng Ning, Xun Zhao, Yu Li, and Sheng-Jun Huang. Active learning for open-set annotation. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 41--49, 2022

    2022

  41. [49]

    Gpt-4 technical report

    OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  42. [50]

    Activelink: deep active learning for link prediction in knowledge graphs

    Natalia Ostapuk, Jie Yang, and Philippe Cudr \'e -Mauroux. Activelink: deep active learning for link prediction in knowledge graphs. In The World Wide Web Conference (WWW), 2019

    2019

  43. [51]

    Meta-query-net: Resolving purity-informativeness dilemma in open-set active learning

    Dongmin Park, Yooju Shin, Jihwan Bang, Youngjun Lee, Hwanjun Song, and Jae-Gil Lee. Meta-query-net: Resolving purity-informativeness dilemma in open-set active learning. Proc. Neural Information Processing Systems (NeurIPS), 35: 0 31416--31429, 2022

    2022

  44. [52]

    Active learning from relative queries

    Buyue Qian, Xiang Wang, Fei Wang, Hongfei Li, Jieping Ye, and Ian Davidson. Active learning from relative queries. In Proc. International Joint Conference on Artificial Intelligence (IJCAI), 2013

    2013

  45. [53]

    Abdomenatlas-8k: Annotating 8,000 ct volumes for multi-organ segmentation in three weeks

    Chongyu Qu, Tiezheng Zhang, Hualin Qiao, Yucheng Tang, Alan L Yuille, Zongwei Zhou, et al. Abdomenatlas-8k: Annotating 8,000 ct volumes for multi-organ segmentation in three weeks. Advances in Neural Information Processing Systems (NeurIPS), 36, 2024

    2024

  46. [54]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Proc. International Conference on Machine Learni...

    2021

  47. [55]

    Classification with valid and adaptive coverage

    Yaniv Romano, Matteo Sesia, and Emmanuel Candes. Classification with valid and adaptive coverage. Proc. Neural Information Processing Systems (NeurIPS), 33: 0 3581--3591, 2020

    2020

  48. [56]

    Active learning for convolutional neural networks: A core-set approach

    Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In Proc. International Conference on Learning Representations (ICLR), 2018

    2018

  49. [57]

    Active learning literature survey

    Burr Settles. Active learning literature survey. Computer Sciences Technical Report 1648, University of Wisconsin--Madison, 2009

    2009

  50. [58]

    Active learning with real annotation costs

    Burr Settles, Mark Craven, and Lewis Friedland. Active learning with real annotation costs. In Proceedings of the NIPS workshop on cost-sensitive learning, volume 1, 2008

    2008

  51. [59]

    A tutorial on conformal prediction

    Glenn Shafer and Vladimir Vovk. A tutorial on conformal prediction. Journal of Machine Learning Research, 9 0 (3), 2008

    2008

  52. [60]

    Variational adversarial active learning

    Samarth Sinha, Sayna Ebrahimi, and Trevor Darrell. Variational adversarial active learning. In Proc. IEEE/CVF International Conference on Computer Vision (ICCV), 2019

    2019

  53. [61]

    Efficientnet: Rethinking model scaling for convolutional neural networks

    Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In Proc. International Conference on Machine Learning (ICML), pp.\ 6105--6114. PMLR, 2019

    2019

  54. [62]

    Bayesian generative active deep learning

    Toan Tran, Thanh-Toan Do, Ian Reid, and Gustavo Carneiro. Bayesian generative active deep learning. In International conference on machine learning, pp.\ 6295--6304. PMLR, 2019

    2019

  55. [63]

    Machine-learning applications of algorithmic randomness

    Volodya Vovk, Alexander Gammerman, and Craig Saunders. Machine-learning applications of algorithmic randomness. In Proc. International Conference on Machine Learning (ICML), ICML '99, pp.\ 444–453, San Francisco, CA, USA, 1999. Morgan Kaufmann Publishers Inc. ISBN 1558606122

    1999

  56. [64]

    Who should label what? instance allocation in multiple expert active learning

    Byron C Wallace, Kevin Small, Carla E Brodley, and Thomas A Trikalinos. Who should label what? instance allocation in multiple expert active learning. In Proceedings of the 2011 SIAM international conference on data mining, pp.\ 176--187. SIAM, 2011

    2011

  57. [65]

    Samrs: Scaling-up remote sensing segmentation dataset with segment anything model

    Di Wang, Jing Zhang, Bo Du, Minqiang Xu, Lin Liu, Dacheng Tao, and Liangpei Zhang. Samrs: Scaling-up remote sensing segmentation dataset with segment anything model. Proc. Neural Information Processing Systems (NeurIPS), 36, 2024

    2024

  58. [66]

    Uncertainty-based active learning for reading comprehension

    Jing Wang, Jie Shen, Xiaofei Ma, and Andrew Arnold. Uncertainty-based active learning for reading comprehension. Transactions on Machine Learning Research (TMLR), 2022. ISSN 2835-8856

    2022

  59. [67]

    Incorporating distribution matching into uncertainty for multiple kernel active learning

    Zengmao Wang, Bo Du, Weiping Tu, Lefei Zhang, and Dacheng Tao. Incorporating distribution matching into uncertainty for multiple kernel active learning. IEEE Transactions on Knowledge and Data Engineering (TKDE), 2019

    2019

  60. [68]

    Querying discriminative and representative samples for batch mode active learning

    Zheng Wang and Jieping Ye. Querying discriminative and representative samples for batch mode active learning. ACM Transactions on Knowledge Discovery from Data (TKDD), 2015

    2015

  61. [69]

    Multi-label learning with pairwise relevance ordering

    Ming-Kun Xie and Sheng-Jun Huang. Multi-label learning with pairwise relevance ordering. Proc. Neural Information Processing Systems (NeurIPS), 34: 0 23545--23556, 2021

    2021

  62. [70]

    Not all out-of-distribution data are harmful to open-set active learning

    Yang Yang, Yuxuan Zhang, Xin Song, and Yi Xu. Not all out-of-distribution data are harmful to open-set active learning. Proc. Neural Information Processing Systems (NeurIPS), 36, 2024

    2024

  63. [71]

    Active learning through a covering lens

    Ofer Yehuda, Avihu Dekel, Guy Hacohen, and Daphna Weinshall. Active learning through a covering lens. Proc. Neural Information Processing Systems (NeurIPS), 35: 0 22354--22367, 2022

    2022

  64. [72]

    Cmal: Cost-effective multi-label active learning by querying subexamples

    Guoxian Yu, Xia Chen, Carlotta Domeniconi, Jun Wang, Zhao Li, Zili Zhang, and Xiangliang Zhang. Cmal: Cost-effective multi-label active learning by querying subexamples. IEEE Transactions on Knowledge and Data Engineering (TKDE), 34 0 (5): 0 2091--2105, 2020

    2020

  65. [73]

    Wide residual networks

    Sergey Zagoruyko. Wide residual networks. arXiv preprint arXiv:1605.07146, 2016

    2016 arXiv

  66. [74]

    Scaling vision transformers

    Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 12104--12113, 2022

    2022

  67. [75]

    mixup: Beyond empirical risk minimization

    Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In Proc. International Conference on Learning Representations (ICLR), 2018

    2018

  68. [76]

    Labelbench: A comprehensive framework for benchmarking adaptive label-efficient learning

    Jifan Zhang, Yifang Chen, Gregory Canal, Arnav Mohanty Das, Gantavya Bhatt, Stephen Mussmann, Yinglun Zhu, Jeff Bilmes, Simon Shaolei Du, Kevin Jamieson, and Robert D Nowak. Labelbench: A comprehensive framework for benchmarking adaptive label-efficient learning. Journal of Da...

    2024

  69. [77]

    One-bit active query with contrastive pairs

    Yuhang Zhang, Xiaopeng Zhang, Lingxi Xie, Jie Li, Robert C Qiu, Hengtong Hu, and Qi Tian. One-bit active query with contrastive pairs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 9697--9705, 2022

    2022

  70. [78]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.