Pith. sign in

REVIEW 3 major objections 3 minor 73 references

Revisiting Active Learning under (Human) Label Variation

T0 review · 3 major / 3 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read When labels legitimately vary between annotators, the paper argues, active learning must switch to distributional targets, new acquisition rules, and explicit annotator choice.

desk verdict A solid, honest synthesis of HLV-aware active learning, but its central 'must' claims silently assume a distributional objective and should be conditioned. read the letter →

arxiv 2507.02593 v1 pith:TMTSGFMI submitted 2025-07-03 cs.CL cs.HCcs.LGstat.ML

Cornelia Gruber , Helen Alber , Bernd Bischl , Göran Kauermann , Barbara Plank , Matthias A{ss}enmacher This is my paper
classification cs.CLcs.HCcs.LGstat.ML
keywords activelearninghumanlabelvariationdistributionannotatorselectionannotationdisagreementLLM-as-annotatornoisesubjectiveNLPtasks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Active learning traditionally assumes every instance has one true label, that the oracle always supplies it, and that labeling costs are uniform. The paper argues that once human label variation is taken seriously, labels legitimately differ across annotators because of subjectivity or ambiguity, all three assumptions break, and the whole loop has to be rethought. It proposes a signal–noise decomposition of observed label variation, with human label variation as signal and annotation error as noise, and draws four consequences: distributional training and evaluation measures, rejection of classical informativeness and representativeness sampling, an explicit annotator-selection step, and acceptance that separating variation from noise is itself hard. The payoff is an active learning setup that spends annotation budgets on capturing the spread of human judgments rather than chasing a nonexistent gold label.

What carries the argument

The central device is the decomposition of observed label variation into signal, human label variation, and noise, annotation error, together with a three-tier view of labels: discrete class labels, probability labels representing a belief distribution over classes, and distributional labels representing a full distribution over class probabilities. This decomposition and label typology do the work: they motivate distributional loss functions for training and evaluation, they explain why entropy sampling misfires, and they ground the new annotator-selection step in the idea that different annotators sample different perspectives from a distribution of opinions. The paper's generalized active learning loop inserts an annotator acquisition function between instance selection and labeling.

What would settle it

A concrete test: on a subjective NLP task with a fixed pool and a fixed panel of annotators, run two pipelines with identical budgets, one using the proposed distribution-aware acquisition plus annotator selection and one using classic entropy sampling with a single oracle; if the distribution-aware pipeline does not outperform on a held-out distributional metric, the claim that classical acquisition is unsuitable under human label variation is falsified. A second, more targeted observation would be that remeasuring the same annotators on the same instances produces completely unstable disagreement, indicating that the stable human label variation signal the framework needs is absent.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that human label variation is not an edge case to be averaged away but a structural feature of many annotation tasks, and that active learning as currently formulated is built on assumptions that hide it. Under the traditional assumptions, a single ground truth per instance, an oracle that provides it noiselessly, and equal annotation cost, label variation looks like noise. Once those assumptions are dropped, the paper identifies four consequences: optimization and evaluation must use distributional measures such as KL divergence instead of hard-label losses; classical entropy-based informativeness and representativeness sampling are unsuitable because high entropy may be intrinsic to the task and representativeness overlooks multi-labeling; the loop needs an annotator acquisition function that selects among humans and language models; and distinguishing true label variation from annotation error becomes a nontrivial task in its own right. The paper is explicit that this is a conceptual foundation rather than an implemented pipeline.

Load-bearing premise

Everything rests on being able to split observed disagreement into meaningful human label variation and genuine annotation error in a way that can be operationalized for annotation selection; the paper itself flags that this split risks circularity because disagreement is read as evidence of subjectivity while assumptions about subjectivity determine how much disagreement counts as human label variation.

Editorial extensions

If this is right

  • Classifier training and evaluation on subjective tasks should use distributional losses rather than hard-label cross-entropy, because the target of prediction is a distribution of human judgments.
  • Uncertainty sampling loses its default status: predicted label entropy can reflect task-intrinsic ambiguity, so acquisition functions should instead target instances where model uncertainty and predicted annotator disagreement diverge.
  • Active learning pipelines need an annotator-selection step that decides both the kind of labeler, human or language model, and the specific annotator, balancing perspective diversity rather than assuming one oracle.
  • Labels should be collected with an intended representation in mind, aggregating to soft or distributional labels from the start, and repeated labeling of the same instance becomes a legitimate budget use.
  • Language models become an attractive data source in this setup because they can emit label distributions directly, but their reliability and biases remain an open question.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct extension the authors leave implicit is that evaluation metrics should change alongside training: ranking models by accuracy against a single gold label will mis-rank models that correctly predict the distribution of human opinions, so perspectivist evaluation would make the proposed active learning changes testable.
  • The signal–noise decomposition suggests a concrete allocation rule: annotation budgets should be spent on additional annotators precisely for instances where predicted disagreement exceeds a noise threshold, and on new instances elsewhere, amounting to an adaptive more-labels-or-more-cases policy.
  • If language models can approximate human judgment distributions from few explanations, one can test the framework by replacing human annotators with model-generated distributions in the acquisition step; a divergence between model and human distributions would flag instances where human annotation remains necessary.
  • The circularity between observed variation and assumed subjectivity points to a research program: measure task subjectivity with an external instrument, such as expert meta-annotation, so the noise component of label variation can be estimated independently, making human-label-variation-aware active learning falsifiable rather than interpretive.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper argues that the traditional active learning (AL) loop rests on three assumptions (A1–A3): a single ground-truth label per instance, an oracle that provides labels without noise, and equal labeling cost. It observes that human label variation (HLV), understood as plausible differences in annotations rather than error, violates these assumptions and calls for a different AL design. The paper surveys the (H)LV and AL literatures, proposes a decomposition of observed label variation (LV) into signal (HLV) and noise, and derives four consequences: distributional measures must be used for optimizing and evaluating the classifier (C1); classical informativeness and representativeness sampling are unsuitable (C2); an annotator selection function is needed (C3); and separating true label variation from noise is non-trivial, especially with LLM annotators (C4). It also discusses LLM-as-annotator and closes with an explicit limitations section.

Significance. As a conceptual synthesis, the paper is useful: it makes hidden assumptions in AL explicit, connects the HLV and AL communities, and organizes recent work (e.g., Wang and Plank 2023; van der Meer et al. 2024; Baumler et al. 2023) into a coherent research agenda. The paper is honest about its limits: it does not present an implemented framework or empirical validation, and it acknowledges the circularity in inferring subjectivity from observed label variation. Its value lies in the quality of the conceptual framing rather than in new algorithms or formal results. If the overgeneralized wording of C1 and C2 is properly qualified, this would be a solid reference for researchers working on label variation and active learning.

major comments (3)
  1. [§4.1 and §4.2 (C1 and C2)] The claims that “distributional measures must be used for optimizing and evaluating the classifier” and that “classical informativeness or representativeness sampling are unsuitable” are stated unconditionally, but they are only valid under a distributional learning objective. Under the paper’s own generative model in §2 (Y ~ Multinomial(1, p_i)), if the deployment goal is a hard label under 0/1 loss, the Bayes-optimal prediction is the mode (or a cost-weighted decision), and standard hard-label active learning can be optimal or near-optimal. The discussion of monistic and pluralistic worldviews in §2 is not connected to the force of C1/C2. Please reformulate C1 and C2 as conditional on the goal of reproducing the human judgment distribution (or another explicit pluralistic stance), and provide an argument for why that goal should be assumed in HLV settings.
  2. [§4.2, C2] The argument that representativeness sampling is unsuitable because it ignores the option of labeling instances multiple times is not fully supported. Multiple annotation is a labeling-allocation decision that can be combined with a representativeness criterion defined over label distributions; the paper’s own references (e.g., van der Meer et al., 2024) show that annotator diversity can be incorporated into selection. Similarly, the unsuitability of entropy-based uncertainty sampling depends on whether entropy is used as a proxy for model uncertainty or as the objective itself. Separating the acquisition criterion from the annotation protocol would make the framework more precise and avoid overclaiming.
  3. [§4.4 (C4) and Limitations] The paper correctly identifies the circularity of inferring subjectivity from observed label variation, but this is not merely a limitation: it is a load-bearing condition for the proposed AL loop, because C1–C3 presume that LV can be decomposed into signal and noise in an operationalizable way. The framework would be strengthened by an explicit statement of what evidence (e.g., validated labels, explanation-based checks, or repeated annotations) is needed for such a decomposition, and by listing the decomposition as an open research problem in the main body rather than only in the Limitations paragraph.
minor comments (3)
  1. [§1] The sentence “Equal annotation cost is not, strictly speaking, a critical assumption for AL” is confusing given that A3 is listed alongside A1 and A2; please either remove the hedge or explain how A3 is treated in the rest of the paper.
  2. [§3.1] In the “Treating Hard Cases” paragraph, “Exemplary, Reidsma and Op Den Akker (2008) advocate…” should read “For example, Reidsma and Op Den Akker (2008) advocate…”.
  3. [Figure 2 caption] The caption states that “each cross represents a single label,” but for case (c) the label is itself a distribution (Beta or Dirichlet); please clarify whether the cross marks a summary statistic, a sample, or a particular parametrization of the plotted distribution.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: a conceptual survey whose claims are normative and whose self-citations are background support; the only circularity is one the paper itself acknowledges.

full rationale

This is a survey/conceptual paper, not a derivation. It contains no fitted parameters, no predictive equations, and no first-principles result that reduces to its inputs. C1-C4 are explicitly labeled 'Consequence' but are normative implications of adopting an HLV-aware perspective, not empirical predictions; they are conditional on the paper's stipulative definitions of HLV and label types. Self-citations (Plank 2022; Weber-Genzel et al. 2024; Wang and Plank 2023; Gruber et al. 2024/2025; Hechinger et al. 2024; Koller et al. 2024; Chen et al. 2024) are used either to attribute terminology or to point to prior empirical work, and none is load-bearing as an authority: the HLV-as-signal premise is supported by multiple independent sources (Pavlick and Kwiatkowski 2019; Peterson et al. 2019; Uma et al. 2021; Cabitza et al. 2023). The only circularity in the manuscript is the one the paper itself flags in Section 2: 'The notion of inferring task subjectivity from observed LV introduces a certain circularity: LV is intuitively taken as evidence of subjectivity, while assumptions about subjectivity, in turn, inform how much of the variation is attributed to HLV.' The paper treats this as an open challenge, not as a premise from which it derives its framework, so it does not constitute a hidden circular step. Possible objections to C1/C2 (e.g., that classical AL may be optimal under a hard-label 0/1 objective) are correctness or overgeneralization concerns, not circularity.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

The paper introduces no new parameters, physical entities, or fitted constants. Its central claim rests on two domain assumptions inherited from prior work: the signal-noise decomposition of label variation and the failure of standard AL assumptions under subjectivity. The paper itself notes the circularity risk in the first assumption.

assumptions (2)
  • domain assumption Human label variation is a meaningful signal distinct from annotation noise.
    Adopted from Plank (2022); the paper's entire framework depends on this distinction being valid and useful.
  • domain assumption Traditional active learning assumptions A1-A3 (single ground truth, noise-free oracle, equal annotation cost) fail in subjective NLP tasks.
    Supported by cited empirical works (e.g., Kenyon-Dean et al., 2018; Pavlick and Kwiatkowski, 2019); the paper relies on this to motivate a revised AL loop.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Revisiting Active Learning under (Human) Label Variation." pith.science (2026). https://pith.science/paper/TMTSGFMI

@misc{pith2026250702593,
  author       = {Pith},
  title        = {Pith review of: Revisiting Active Learning under (Human) Label Variation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TMTSGFMI}},
  note         = {Machine review of arXiv:2507.02593}
}
read the original abstract

Access to high-quality labeled data remains a limiting factor in applied supervised learning. While label variation (LV), i.e., differing labels for the same instance, is common, especially in natural language processing, annotation frameworks often still rest on the assumption of a single ground truth. This overlooks human label variation (HLV), the occurrence of plausible differences in annotations, as an informative signal. Similarly, active learning (AL), a popular approach to optimizing the use of limited annotation budgets in training ML models, often relies on at least one of several simplifying assumptions, which rarely hold in practice when acknowledging HLV. In this paper, we examine foundational assumptions about truth and label nature, highlighting the need to decompose observed LV into signal (e.g., HLV) and noise (e.g., annotation error). We survey how the AL and (H)LV communities have addressed -- or neglected -- these distinctions and propose a conceptual framework for incorporating HLV throughout the AL loop, including instance selection, annotator choice, and label representation. We further discuss the integration of large language models (LLM) as annotators. Our work aims to lay a conceptual foundation for HLV-aware active learning, better reflecting the complexities of real-world annotation.

Figures

Figures reproduced from arXiv: 2507.02593 by the authors.

Figure 1
Figure 1. The traditional AL loop with possible adap [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Types of labels visualized. Each label yi is a realization of a random variable Y . Top row: binary classes; bottom row: three classes. (a) Discrete label: yi = 1 with Y ∼ Bin(1, p) (top) and yi = [1, 0, 0] with Y ∼ Multinom(1, p) (bottom), (b) probability label: yi = 0.75 with Y ∼ Beta(α, β) (top) and yi = [0.6, 0.2, 0.2] with Y ∼ Dir(α) (bottom), (c) distributional label: yi = Beta(8, 3.5) (top) and yi = Dir(8, 3,… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

73 extracted references · 45 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Steven Abney. 2007. Semisupervised Learning for Computational Linguistics , 1st edition. Chapman & Hall/CRC

    work page 2007

  4. [4]

    Aggarwal, Xiangnan Kong, Quanquan Gu, Jiawei Han, and S

    Charu C. Aggarwal, Xiangnan Kong, Quanquan Gu, Jiawei Han, and S. Yu Philip. 2014. Active learning: A survey. In Data Classification : Algorithms and Applications , pages 599--634. Chapman and Hall/CRC

    work page 2014

  5. [5]

    Cecilia Ovesdotter Alm. 2011. Subjective natural language problems: Motivations, applications, characterizations, and implications. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 107--112

    work page 2011

  6. [6]

    Lora Aroyo and Chris Welty. 2015. Truth is a lie: Crowd truth and the seven myths of human annotation. AI Magazine, 36(1):15--24

    work page 2015

  7. [7]

    Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal

    Jordan T. Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. 2020. https://openreview.net/forum?id=ryghZJBKPS Deep batch active learning by diverse, uncertain gradient lower bounds . In International Conference on Learning Representations

    work page 2020

  8. [8]

    Parikshit Bansal and Amit Sharma. 2023. https://doi.org/10.48550/arXiv.2306.15766 Large Language Models as Annotators : Enhancing Generalization of NLP Models at Minimal Cost . arXiv preprint. ArXiv:2306.15766 [cs]

    arXiv doi:10.48550/arxiv.2306.15766 2023

Show all 73 references
  1. [9]

    Valerio Basile. 2020. https://doi.org/10.1007/978-3-030-77091-4_26 It’s the End of the Gold Standard as We Know It : Leveraging Non -aggregated Data for Better Evaluation and Explanation of Subjective Tasks . In AIxIA 2020 – Advances in Artificial Intelligence : XIXth Internat...

    2020 doi

  2. [10]

    Valerio Basile, Michael Fell, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, Massimo Poesio, and Alexandra Uma. 2021. https://doi.org/10.18653/v1/2021.bppf-1.3 We need to consider disagreement in evaluation . In Proceedings of the 1st Workshop on Benchmarking: Past...

    2021 doi

  3. [11]

    Connor Baumler, Anna Sotnikova, and Hal Daumé Iii. 2023. https://doi.org/10.18653/v1/2023.findings-acl.658 Which Examples Should be Multiply Annotated ? Active Learning When Annotators May Disagree . In Findings of the Association for Computational Linguistics : ACL 2023 , pag...

    2023 doi

  4. [12]

    Beata Beigman Klebanov and Eyal Beigman. 2009. https://doi.org/10.1162/coli.2009.35.4.35402 From Annotator Agreement to Noise Models . Computational Linguistics, 35(4):495--503

    2009 doi

  5. [13]

    Viktor Bengs, Eyke Hüllermeier, and Willem Waegeman. 2022. https://openreview.net/forum?id=epjxT_ARZW5 Pitfalls of epistemic uncertainty quantification through loss minimisation . In Advances in neural information processing systems

    2022

  6. [14]

    Federico Cabitza, Andrea Campagner, and Valerio Basile. 2023. https://doi.org/10.1609/aaai.v37i6.25840 Toward a Perspectivist Turn in Ground Truthing for Predictive Computing . Proceedings of the AAAI Conference on Artificial Intelligence, 37(6):6860--6868. Number: 6

    2023 doi

  7. [15]

    Shayok Chakraborty. 2020. https://doi.org/10.1609/aaai.v34i04.5738 Asking the Right Questions to the Right Users : Active Learning with Imperfect Oracles . Proceedings of the AAAI Conference on Artificial Intelligence, 34(04):3365--3372. Number: 04

    2020 doi

  8. [16]

    Beiduo Chen, Xinpeng Wang, Siyao Peng, Robert Litschko, Anna Korhonen, and Barbara Plank. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.842 seeing the big through the small : Can LLM s approximate human judgment distributions on NLI from a few explanations? In Findings...

    2024 doi

  9. [17]

    Derek Chen, Zhou Yu, and Samuel R. Bowman. 2022. https://doi.org/10.18653/v1/2022.deeplo-1.17 Clean or annotate: How to spend a limited data collection budget . In Proceedings of the Third Workshop on Deep Learning for Low-Resource Natural Language Processing, pages 152--168, ...

    2022 doi

  10. [18]

    Manning, and Christopher Potts

    Marie-Catherine De Marneffe, Christopher D. Manning, and Christopher Potts. 2012. https://doi.org/10.1162/COLI_a_00097 Did It Happen ? The Pragmatic Complexity of Veridicality Assessment . Computational Linguistics, 38(2):301--333

    2012 doi

  11. [19]

    Carbonell

    Pinar Donmez and Jaime G. Carbonell. 2008. https://doi.org/10.1145/1458082.1458165 Proactive learning: cost-sensitive active learning with multiple imperfect oracles . In Proceedings of the 17th ACM conference on Information and knowledge management , pages 619--628, Napa Vall...

    2008

  12. [20]

    Anca Dumitrache, Lora Aroyo, and Chris Welty. 2019. https://doi.org/10.18653/v1/N19-1224 A crowdsourced frame disambiguation corpus with ambiguity . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Lan...

    2019 doi

  13. [21]

    Ruijiang Gao and Maytal Saar-Tsechansky. 2020. https://doi.org/10.1609/aaai.v34i03.5640 Cost- Accuracy Aware Adaptive Labeling for Active Learning . Proceedings of the AAAI Conference on Artificial Intelligence, 34(03):2569--2576. Number: 03

    2020 doi

  14. [22]

    Hui Wen Goh and Jonas Mueller. 2023. https://doi.org/10.48550/arXiv.2301.11856 ActiveLab : Active Learning with Re - Labeling by Multiple Annotators . arXiv preprint. ArXiv:2301.11856 [cs]

    2023 doi arXiv

  15. [23]

    Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep learning. MIT Press

    2016

  16. [24]

    Cornelia Gruber, Katharina Hechinger, Matthias Assenmacher, G \"o ran Kauermann, and Barbara Plank. 2024. https://aclanthology.org/2024.unimplicit-1.2/ More labels or cases? assessing label variation in natural language inference . In Proceedings of the Third Workshop on Under...

    2024

  17. [25]

    Cornelia Gruber, Patrick Oliver Schenk, Malte Schierholz, Frauke Kreuter, and Göran Kauermann. 2025. https://doi.org/10.48550/arXiv.2305.16703 Sources of Uncertainty in Supervised Machine Learning -- A Statisticians ' View . arXiv preprint. ArXiv:2305.16703 [stat]

    2025 doi arXiv

  18. [26]

    Friedman, and Jerome H

    Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, and Jerome H. Friedman. 2009. The elements of statistical learning: data mining, inference, and prediction, volume 2. Springer

    2009

  19. [27]

    Katharina Hechinger, Christoph Koller, Xiao Xiang Zhu, and Göran Kauermann. 2024 a . https://doi.org/10.48550/arXiv.2311.08874 Human-in-the-loop: Towards Label Embeddings for Measuring Classification Difficulty . arXiv preprint. ArXiv:2311.08874 [cs]

    2024 doi arXiv

  20. [28]

    Katharina Hechinger, Xiao Xiang Zhu, and Göran Kauermann. 2024 b . https://doi.org/10.1093/jrsssc/qlad089 Categorising the world into local climate zones: towards quantifying labelling uncertainty for machine learning models . Journal of the Royal Statistical Society Series C:...

    2024 doi

  21. [29]

    Oana Inel and Lora Aroyo. 2017. https://doi.org/10.1007/978-3-319-58068-5_18 Harnessing Diversity in Crowds and Machines for Better NER Performance . In The Semantic Web , pages 289--304, Cham. Springer International Publishing

    2017 doi

  22. [30]

    Nan-Jiang Jiang, Chenhao Tan, and Marie-Catherine De Marneffe. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.712 Ecologically Valid Explanations for Label Variation in NLI . In Findings of the Association for Computational Linguistics : EMNLP 2023 , pages 10622--10633,...

    2023 doi

  23. [31]

    Kian Kenyon-Dean, Eisha Ahmed, Scott Fujimoto, Jeremy Georges-Filteau, Christopher Glasz, Barleen Kaur, Auguste Lalande, Shruti Bhanderi, Robert Belfer, Nirmal Kanagasabai, Roman Sarrazingendron, Rohit Verma, and Derek Ruths. 2018. https://doi.org/10.18653/v1/N18-1171 Sentimen...

    2018 doi

  24. [32]

    Christoph Koller, Göran Kauermann, and Xiao Xiang Zhu. 2024. https://doi.org/10.1109/TGRS.2023.3336357 Going Beyond One - Hot Encoding in Classification : Can Human Uncertainty Improve Model Performance in Earth Observation ? IEEE Transactions on Geoscience and Remote Sensing,...

    2024

  25. [33]

    Akshay Krishnamurthy, Alekh Agarwal, Tzu-Kuo Huang, Hal Daumé Iii, and John Langford. 2019. http://jmlr.org/papers/v20/17-681.html Active Learning for Cost - Sensitive Classification . Journal of Machine Learning Research, 20(65):1--50

    2019

  26. [34]

    Lewis and Jason Catlett

    David D. Lewis and Jason Catlett. 1994. https://doi.org/10.1016/B978-1-55860-335-6.50026-X Heterogeneous Uncertainty Sampling for Supervised Learning . In William W. Cohen and Haym Hirsh, editors, Machine Learning Proceedings 1994 , pages 148--156. Morgan Kaufmann, San Francisco (CA)

    1994 doi

  27. [35]

    Lewis and William A

    David D. Lewis and William A. Gale. 1994. https://doi.org/10.48550/arXiv.cmp-lg/9407020 A Sequential Algorithm for Training Text Classifiers . arXiv preprint. ArXiv:cmp-lg/9407020

    1994 doi arXiv

  28. [36]

    Christopher Lin, M Mausam, and Daniel Weld. 2016. https://doi.org/10.1609/aaai.v30i1.10315 Re- Active Learning : Active Learning with Relabeling . Proceedings of the AAAI Conference on Artificial Intelligence, 30(1)

    2016 doi

  29. [37]

    Margineantu

    Dragos D. Margineantu. 2005. https://www.ijcai.org/Proceedings/05/Papers/post-0525.pdf Active cost-sensitive learning . In IJCAI , volume 5, pages 1622--1623

    2005

  30. [38]

    Mitchell

    Tom M. Mitchell. 1997. Machine Learning , nachdr. edition. McGraw - Hill series in Computer Science . McGraw-Hill, New York

    1997

  31. [39]

    Jingwei Ni, Yu Fan, Vilém Zouhar, Donya Rooein, Alexander Hoyle, Mrinmaya Sachan, Markus Leippold, Dirk Hovy, and Elliott Ash. 2025. https://doi.org/10.48550/arXiv.2506.19467 Can Large Language Models Capture Human Annotator Disagreements ? arXiv preprint. ArXiv:2506.19467 [cs]

    2025 doi

  32. [40]

    Jennimaria Palomaki, Olivia Rhinehart, and Michael Tseng. 2018. A case for a range of acceptable annotations. In SAD/CrowdBias@ HCOMP, pages 19--31

    2018

  33. [41]

    Passonneau, Vikas Bhardwaj, Ansaf Salleb-Aouissi, and Nancy Ide

    Rebecca J. Passonneau, Vikas Bhardwaj, Ansaf Salleb-Aouissi, and Nancy Ide. 2012. https://doi.org/10.1007/s10579-012-9188-x Multiplicity and word sense: evaluating and learning from multiply labeled word sense annotations . Lang. Resour. Eval., 46(2):219--252

    2012 doi

  34. [42]

    Ellie Pavlick and Tom Kwiatkowski. 2019. https://doi.org/10.1162/tacl_a_00293 Inherent Disagreements in Human Textual Inferences . Transactions of the Association for Computational Linguistics, 7:677--694. Place: Cambridge, MA Publisher: MIT Press

    2019 doi

  35. [43]

    Maja Pavlovic and Massimo Poesio. 2024. https://aclanthology.org/2024.nlperspectives-1.11/ The effectiveness of LLM s as annotators: A comparative overview and empirical analysis of direct representation . In Proceedings of the 3rd Workshop on Perspectivist Approaches to NLP (...

    2024

  36. [44]

    Peterson, Ruairidh M

    Joshua C. Peterson, Ruairidh M. Battleday, Thomas L. Griffiths, and Olga Russakovsky. 2019. Human uncertainty makes classification more robust. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 9616--9625

    2019

  37. [45]

    Barbara Plank. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.731 The problem of human label variation: On ground truth in data, modeling and evaluation . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10671--10682, Abu Dhab...

    2022 doi

  38. [46]

    Barbara Plank, Dirk Hovy, and Anders S gaard. 2014 a . https://doi.org/10.3115/v1/E14-1078 Learning part-of-speech taggers with inter-annotator agreement loss . In Proceedings of the 14th Conference of the E uropean Chapter of the Association for Computational Linguistics , pa...

    2014 doi

  39. [47]

    Barbara Plank, Dirk Hovy, and Anders S gaard. 2014 b . https://doi.org/10.3115/v1/P14-2083 Linguistically debatable or just plain wrong? In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 507--511, Baltimo...

    2014 doi

  40. [48]

    Lukas Rauch, Matthias Aßenmacher, Denis Huseljic, Moritz Wirth, Bernd Bischl, and Bernhard Sick. 2023. https://doi.org/10.1007/978-3-031-43412-9_4 ActiveGLAE : A Benchmark for Deep Active Learning with Transformers . In Machine Learning and Knowledge Discovery in Databases : R...

    2023 doi

  41. [49]

    Dennis Reidsma and Rieks Op Den Akker. 2008. https://doi.org/10.3115/1611628.1611631 Exploiting 'subjective' annotations . In Proceedings of the Workshop on Human Judgements in Computational Linguistics - HumanJudge '08 , pages 8--16, Manchester, United Kingdom. Association fo...

    2008

  42. [50]

    Paul R \"o ttger, Bertie Vidgen, Dirk Hovy, and Janet Pierrehumbert. 2022. https://doi.org/10.18653/v1/2022.naacl-main.13 Two contrasting data annotation paradigms for subjective NLP tasks . In Proceedings of the 2022 Conference of the North American Chapter of the Association...

    2022 doi

  43. [51]

    Bertrand Russell. 1907. https://doi.org/10.1093/aristotelian/7.1.28 II .— On the Nature of Truth . Proceedings of the Aristotelian Society, 7(1):28--49

    1907 doi

  44. [52]

    Yusuf Sale, Viktor Bengs, Michele Caprio, and Eyke Hüllermeier. 2024. Second-order uncertainty quantification: a distance-based approach. In Proceedings of the 41st International Conference on Machine Learning , volume 235 of ICML '24 , pages 43060--43076, Vienna, Austria. JMLR.org

    2024

  45. [53]

    Christoph Schweden, Katharina Hechinger, Göran Kauermann, and Xiao Xiang Zhu. 2025. https://doi.org/10.1109/TGRS.2025.3562233 Can Uncertainty Quantification Benefit From Label Embeddings ? A Case Study on Local Climate Zone Classification . IEEE Transactions on Geoscience and ...

    2025

  46. [54]

    Burr Settles. 2009. Active Learning Literature Survey . University of Wisconsin, Madison, 52

    2009

  47. [55]

    Burr Settles. 2011. From Theories to Queries : Active Learning in Practice . In Active learning and experimental design workshop in conjunction with AISTATS 2010 , pages 1--18

    2011

  48. [56]

    Burr Settles, Mark Craven, and Lewis Friedland. 2008. Active Learning with Real Annotation Costs . In Proceedings of the NIPS Workshop on Cost - Sensitive Learning

    2008

  49. [57]

    H. S. Seung, M. Opper, and H. Sompolinsky. 1992. https://doi.org/10.1145/130385.130417 Query by committee . In Proceedings of the fifth annual workshop on Computational learning theory , COLT '92, pages 287--294, New York, NY, USA. Association for Computing Machinery

    1992

  50. [58]

    Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christopher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, Tim Althoff, and Yejin Choi. 2024. Position: A Roadmap to Pluralistic Alignment . In Proceedings of the 41 st Inte...

    2024

  51. [59]

    Dietrich Trautmann, Johannes Daxenberger, Christian Stab, Hinrich Schütze, and Iryna Gurevych. 2020. https://doi.org/10.1609/aaai.v34i05.6438 Fine- Grained Argument Unit Recognition and Classification . Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):9048...

    2020 doi

  52. [60]

    Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio

    Alexandra N. Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. 2021. https://doi.org/10.1613/jair.1.12752 Learning from Disagreement : A Survey . Journal of Artificial Intelligence Research, 72:1385--1470

    2021 doi

  53. [61]

    Murukannaiah, and Enrico Liscio

    Michiel van der Meer, Neele Falk, Pradeep K. Murukannaiah, and Enrico Liscio. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.1031 Annotator-centric active learning for subjective NLP tasks . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Proc...

    2024 doi

  54. [62]

    Xinpeng Wang and Barbara Plank. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.126 ACTOR : Active learning with annotator-specific classification heads to embrace human label variation . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processi...

    2023 doi

  55. [63]

    Leon Weber-Genzel, Siyao Peng, Marie-Catherine De Marneffe, and Barbara Plank. 2024. https://doi.org/10.18653/v1/2024.acl-long.123 V ari E rr NLI : Separating annotation error from human label variation . In Proceedings of the 62nd Annual Meeting of the Association for Computa...

    2024 doi

  56. [64]

    Lisa Wimmer, Yusuf Sale, Paul Hofman, Bernd Bischl, and Eyke Hüllermeier. 2023. https://proceedings.mlr.press/v216/wimmer23a.html Quantifying aleatoric and epistemic uncertainty in machine learning: Are conditional entropy and mutual information appropriate measures? In Procee...

    2023

  57. [65]

    Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason Weston, and Sainbayar Sukhbaatar. 2024. https://arxiv.org/abs/2407.19594 Meta-rewarding language models: Self-improving alignment with llm-as-a-meta-judge . Preprint, arXiv:2407.19594

    2024 arXiv

  58. [66]

    Kaige Xie, Cheng Chang, Liliang Ren, Lu Chen, and Kai Yu. 2018. https://doi.org/10.18653/v1/W18-5022 Cost-sensitive active learning for dialogue state tracking . In Proceedings of the 19th Annual SIG dial Meeting on Discourse and Dialogue , pages 209--213, Melbourne, Australia...

    2018 doi

  59. [67]

    Chicheng Zhang and Kamalika Chaudhuri. 2015. https://proceedings.neurips.cc/paper_files/paper/2015/file/eba0dc302bcd9a273f8bbb72be3a687b-Paper.pdf Active learning from weak and strong labelers . In Advances in neural information processing systems, volume 28. Curran Associates, Inc

    2015

  60. [68]

    Jing Zhang, Xindong Wu, and Victor S. Shengs. 2015. https://doi.org/10.1109/TCYB.2014.2344674 Active Learning With Imbalanced Multiple Noisy Labeling . IEEE Transactions on Cybernetics, 45(5):1095--1107. Conference Name: IEEE Transactions on Cybernetics

    2015

  61. [69]

    Alexander

    Le Zhang, Ryutaro Tanno, Mou-Cheng Xu, Chen Jin, Joseph Jacob, Olga Ciccarelli, Frederik Barkhof, and Daniel C. Alexander. 2020. Disentangling human error from the ground truth in segmentation of medical images. In Proceedings of the 34th International Conference on Neural Inf...

    2020

  62. [70]

    Ruoyu Zhang, Yanzeng Li, Yongliang Ma, Ming Zhou, and Lei Zou. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.872 LLM a AA : Making large language models as active annotators . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 13088--13103,...

    2023 doi

  63. [71]

    Zhisong Zhang, Emma Strubell, and Eduard Hovy. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.414 A Survey of Active Learning for Natural Language Processing . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages 6166--6190, Abu ...

    2022 doi

  64. [72]

    Liyue Zhao, Gita Sukthankar, and Rahul Sukthankar. 2011. https://doi.org/10.1109/PASSAT/SocialCom.2011.193 Incremental Relabeling for Active Learning with Noisy Crowdsourced Annotations . In 2011 IEEE Third International Conference on Privacy , Security , Risk and Trust and 20...

    2011 doi

  65. [73]

    Gonzalez, and Ion Stoica

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. https://openreview.net/forum?id=uccHPGDlao Judging LLM -as-a-judge with MT -bench and chatbot ...

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.