Pith. sign in

REVIEW 4 major objections 4 minor 2 cited by

Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets

T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Crowdsourced NLU datasets carry a hidden signal: who wrote each example, and models use that signal.

desk verdict Solid empirical study with a useful new measurement apparatus for annotator bias; just don't take the held-out-split interpretation as proof of style over content shift. read the letter →

arxiv 1908.07898 v2 pith:7DI3UHSA submitted 2019-08-21 cs.CL

Mor Geva , Yoav Goldberg , Jonathan Berant This is my paper · ORCID
classification cs.CL
keywords annotatorbiascrowdsourcingnaturallanguageunderstandingdatasetconstructiongeneralizationrecognitionBERTannotationartifacts
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that crowdsourced natural language understanding (NLU) datasets contain a systematic bias tied to the annotator who wrote each example. Across three recent datasets, the authors find that models perform better when given annotator IDs as input features, that models can often recognize the most productive annotators from the text alone, and that in two of the three datasets models generalize poorly to examples written by annotators not seen in training. The central consequence is that reported accuracies may partly reflect how well a model imitates familiar annotators rather than how well it understands language. If correct, dataset creators should track annotator identity during collection and use annotator-disjoint test splits to obtain a truer measure of generalization.

What carries the argument

The load-bearing mechanism is the annotator-disjoint train/test split, in which no annotator appears in both the training and development sets; the gap between such splits and random splits of identical size is taken as a direct measure of annotator bias. Supporting machinery includes adding the annotator ID as an input token to a pre-trained transformer, fine-tuning the model to predict annotator identity as a separate classification task, and an augmentation procedure that moves small fractions of a held-out annotator's examples into training to separate style effects from intrinsic difficulty.

What would settle it

A concrete test is to rebuild one of the datasets so that each underlying item is rewritten by several annotators, then compare accuracy on an annotator-disjoint split against a random split over versions while holding topic and label distribution fixed. If the gap disappears under this matching, the bias is not about annotator identity; if the gap persists even though the same items are written by different people, the paper's conclusion is confirmed.

Watch

Extended reading notes

Core claim

The paper claims that annotator bias—systematic linguistic patterns tied to the worker who produced an example—is present in current NLU datasets and inflates model performance. Evidence comes from three experiments: prepending a textual annotator ID to each training example raises development accuracy on all three datasets (significant for OpenBookQA and MNLI); a model fine-tuned to predict the top-five annotators from an example's text reaches F1 scores as high as 0.91, showing that annotator style is readable; and when training and development sets are split so that annotators are disjoint, development accuracy drops by up to 23 points in OpenBookQA and is often lower in CommonsenseQA, while MNLI shows smaller and mixed effects. An augmentation experiment—moving small fractions of a held-out annotator's examples into training—quickly recovers most of the lost performance in the QA datasets, suggesting the drop reflects annotator style rather than example difficulty.

Load-bearing premise

The interpretation assumes that the performance gap between annotator-disjoint and random splits is caused by annotator identity (writing style) rather than by incidental differences in topic, difficulty, or label distribution between the split groups.

Editorial extensions

If this is right

  • Reported accuracies on crowdsourced NLU benchmarks can overstate true understanding, because models may exploit annotator-specific wording patterns.
  • Dataset creators should test on new annotators during data collection to detect annotator bias early.
  • Holding out entire annotators from the training set would give a more honest estimate of generalization to unseen users.
  • Adding a small number of examples from a held-out annotator to training can quickly restore performance, suggesting a practical remedy when bias is found.
  • Annotator recognition accuracy is a simple diagnostic: if a model can tell who wrote an example, the dataset is likely carrying style signals.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same audit method—annotator-ID benefit, recognition probes, and disjoint splits—could be applied to newer large-scale NLU datasets to check whether they carry annotator style.
  • If annotator bias is widespread, comparing models across papers may be confounded by which annotators happen to land in training versus test; public benchmarks might need to report annotator-holdout scores as a standard metric.
  • A further study could test whether adding annotator IDs encodes topic or label preferences rather than pure lexical style, which would change the choice of mitigation: decorrelating topics from annotators versus aggregating multiple annotators per item.
  • The augmentation result suggests a possible data-collection strategy: periodically recruit a few fresh annotators, measure the performance gap on their examples, and use that gap as an early warning during dataset construction.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This paper investigates whether recent NLU datasets created by small numbers of crowdworkers exhibit annotator bias. Using MNLI, OpenBookQA, and CommonsenseQA with annotator IDs, the authors report three experiments: (1) adding annotator IDs as input features improves development performance (Table 2), with statistical significance for MNLI and OpenBookQA but not CommonsenseQA; (2) fine-tuned BERT can identify the most productive annotators from text, with high F1 for OpenBookQA and CommonsenseQA top annotators (Figure 2); (3) when the development set is composed of examples from annotators held out of training, performance drops relative to a size-matched random split, most dramatically in OpenBookQA multi-annotator splits (up to 23 points, Table 3). An augmentation experiment shows that moving a small fraction of held-out annotator examples into training recovers 10-20 points in OpenBookQA and CommonsenseQA, which the authors interpret as evidence that the drop is due to annotator style rather than example difficulty. The paper concludes that annotator bias should be monitored and that test-set annotators should be disjoint from training-set annotators.

Significance. If the results hold, the paper documents a practically important pitfall in crowdsourced NLU dataset construction: model performance on a random dev set may overestimate performance on examples produced by new annotators. The proposed remedy—disjoint annotators between train and test—is actionable and cheap to implement. The strongest evidence is the OpenBookQA multi-annotator split experiment, where all five held-out annotator sets show large negative performance differences, and the augmentation results showing rapid recovery with only a few examples. The paper also deserves credit for comparing against size-matched random splits, repeating runs to report variance, and using appropriate significance tests (McNemar/bootstrap). However, the interpretation as specifically annotator-style bias is not fully established because the split design also changes topic and label distributions.

major comments (4)
  1. [Section 4, Table 3] The annotator-disjoint splits in Table 3 are not a clean manipulation of annotator identity. In OpenBookQA, each question is generated from a scientific fact, and an annotator who wrote many questions may have concentrated on a subset of facts; the held-out dev set then differs from the training set in topic composition, vocabulary, and possibly label priors, so the up-to-23-point drop could reflect a topic holdout rather than annotator-specific language style. Because the size-matched random split does not control for this content shift, Table 3 does not by itself establish that the failure is attributable to the annotator as opposed to the content distribution. I ask the authors to add a control that matches the topic/fact composition between the annotator-disjoint and random dev sets, or to otherwise measure the amount of topic overlap, and to temper the causal language accordingly.
  2. [Section 4, Figure 3 (Separating annotator bias from annotator difficulty)] The augmentation experiment is designed to separate 'inherent difficulty' from annotator bias, but it does not distinguish annotator style from topic exposure. When examples are moved from the held-out dev set into training, the model gains access to instances from the missing topic/label region, so performance could improve even if the original gap had nothing to do with the annotator's idiolect. The control of keeping the training set size constant does not address this. A stronger control would be to add the same number of examples from the same topics written by other annotators, or to compare augmentation by held-out-annotator examples against augmentation by topic-matched examples from seen annotators.
  3. [Section 4, Table 2] The text states that adding the annotator ID 'improves model performance across all datasets,' but the CommonsenseQA difference is not statistically significant (p = 0.1198). This should be reported as not significant, and the claim in the abstract that models improve with annotator identifiers should be qualified. The other two datasets are significant, so the overall point survives, but the current wording overstates the evidence.
  4. [Section 4, Table 3 and Section 5] The CommonsenseQA multi-annotator results are largely within noise: only one of the five sets shows a clear negative difference (-9.5 ± 8.3), while two are positive. The conclusion that annotator bias is evident in 'two of these datasets' (Section 5) is therefore too strong for CommonsenseQA unless an aggregate test across the five sets is provided. Reporting a combined p-value or effect size would clarify whether CommonsenseQA supports the claim.
minor comments (4)
  1. [Section 3] There are formatting inconsistencies in dataset names: 'OPEN BOOK QA.' has a stray period, and 'O PEN BOOK QA' and 'C OMMONSENSE QA' contain extra spaces. Please fix these throughout.
  2. [Figure 2 caption] The caption notes that OpenBookQA has only four data points because the second annotator is not in the original development set; this limitation should be stated in the main text, since it affects the interpretation of the recognition performance.
  3. [Table 2] The p-values are reported in a needlessly confusing notation (e.g., '1.83e−2' and '11.98e−2'); use conventional decimals such as 0.018 and 0.120.
  4. [Section 4, augmentation experiment] The description does not specify whether evaluation after moving k% of the dev set into training is performed on the remaining dev examples or on the full original dev set; please clarify, and state the number of random seeds used for the repeats.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's claims are empirical measurements against held-out and randomized baselines, not derivations that reduce to their inputs.

full rationale

The paper's load-bearing results are all direct empirical measurements: (1) adding annotator IDs as input features and comparing development accuracy, (2) fine-tuning a classifier to predict top annotator IDs and reporting F1, and (3) comparing annotator-disjoint splits with random splits of identical size, plus a controlled augmentation experiment. None of these results is obtained by fitting a parameter and then renaming it as a prediction, and no quantity that is supposed to be predicted is used to define the experimental inputs. The augmentation experiment's interpretation—that improvement after moving dev examples into training is due to annotator exposure—is an inference that could in principle be confounded by topic or label shift, but a confound is not a circular construction. The only author-related citation is CommonsenseQA (Talmor et al., 2019), a co-authored external benchmark; it is not used as an unverified premise to justify the method or conclusion, and the experiments also use two other datasets. The paper even acknowledges the annotator-difficulty confound and attempts to control it. There is no self-citation chain, uniqueness theorem, or ansatz smuggled in via citation. Therefore the derivation chain is self-contained with respect to circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper does not derive anything formally; its conclusions rest on assumptions about annotator metadata integrity and about what the controlled splits control for. No invented entities are introduced.

free parameters (3)
  • Top-5 annotator threshold for recognition = 6 labels (top 5 plus OTHER)
    The recognition experiment collapses annotators into 6 labels. Results could change with a different threshold, and no sensitivity analysis is reported.
  • Augmentation ratios = 0.1, 0.2, 0.3
    Three small augmentation fractions were chosen by hand to test how much held-out annotator exposure helps.
  • BERT fine-tuning hyperparameters = lr 2e-5, 3 epochs, batch size 10, max length 128
    A standard fixed configuration, not fitted to the annotator-bias hypothesis, but the exact numbers affect all reported accuracies.
assumptions (4)
  • domain assumption Annotator IDs are accurate, complete, and correspond to the true writer of each example.
    All conclusions depend on ID correctness. IDs were supplied by dataset authors for MNLI and CommonsenseQA.
  • domain assumption The random split baselines are matched to annotator splits in size and difficulty except for annotator identity.
    If held-out annotators write about different topics or have different label distributions, performance drops could be domain shift rather than annotator style.
  • domain assumption Adding an annotator ID as a text token provides the model with usable annotator information without fundamentally changing the learning task.
    The comparison with and without ID assumes this is a valid way to expose identity to BERT.
  • domain assumption The augmentation experiment isolates annotator exposure from intrinsic example difficulty.
    The rapid improvement after moving a few held-out annotator examples into training is interpreted as style exposure, not difficulty adjustment.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets." pith.science (2026). https://pith.science/paper/7DI3UHSA

@misc{pith2026190807898,
  author       = {Pith},
  title        = {Pith review of: Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7DI3UHSA}},
  note         = {Machine review of arXiv:1908.07898}
}
read the original abstract

Crowdsourcing has been the prevalent paradigm for creating natural language understanding datasets in recent years. A common crowdsourcing practice is to recruit a small number of high-quality workers, and have them massively generate examples. Having only a few workers generate the majority of examples raises concerns about data diversity, especially when workers freely generate sentences. In this paper, we perform a series of experiments showing these concerns are evident in three recent NLP datasets. We show that model performance improves when training with annotator identifiers as features, and that models are able to recognize the most productive annotators. Moreover, we show that often models do not generalize well to examples from annotators that did not contribute to the training set. Our findings suggest that annotator bias should be monitored during dataset creation, and that test set annotators should be disjoint from training set annotators.

Figures

Figures reproduced from arXiv: 1908.07898 by the authors.

Figure 1
Figure 1. Proportion of examples covered by number [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Annotator recognition F1-scores for the top [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Performance difference between single￾annotator splits and random splits of identical size. The x-axis indicates the fraction of examples taken from the development set to augment the training set. tially. However, if performance on Sdev is low due to annotator bias, then moving examples to Strain would expose the model to the annotator and per￾formance should go up. For every single-annotator data split Strain, Sde… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to a Test Set

    cs.CL 2024-11 conditional novelty 7.0 of 10

    Using per-example in-context demonstrations built from historical same-source human MQM ratings makes an LLM judge dramatically better at fine-grained MT evaluation on WMT'23 and WMT'24.

  2. An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction

    cs.CL 2019-09 accept novelty 7.0 of 10

    The paper releases a 150-intent, 10-domain dialog corpus with dedicated out-of-scope queries and shows current classifiers detect those out-of-scope queries poorly.

Reference graph

Works this paper leans on

30 extracted references · 16 canonical work pages · cited by 2 Pith papers

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Roee Aharoni and Yoav Goldberg. 2018. Split and rephrase: Better evaluation and stronger baselines. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 719--724. Association for Computational Linguistics

    work page 2018

  4. [4]

    Taylor Berg-Kirkpatrick, David Burkett, and Dan Klein. 2012. An empirical investigation of statistical significance in nlp. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 995--1005. Association for Computational Linguistics

    work page 2012

  5. [5]

    Bowman, G

    S. Bowman, G. Angeli, C. Potts, and C. D. Manning. 2015. A large annotated corpus for learning natural language inference. In Empirical Methods in Natural Language Processing (EMNLP)

    2015

  6. [6]

    Chris Callison-Burch, Lyle Ungar, and Ellie Pavlick. 2015. Crowdsourcing for NLP . In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Tutorial Abstracts, pages 2--3

    work page 2015

  7. [7]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

    arXiv 2018

  8. [8]

    Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker’s guide to testing statistical significance in natural language processing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1383--1392

    work page 2018

Show all 30 references
  1. [9]

    Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP : A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proc. of NAACL

    2019

  2. [10]

    Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018. https://www.aclweb.org/anthology/P18-2103 Breaking NLI systems with sentences that require simple lexical inferences . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Sh...

    2018

  3. [11]

    Gururangan, S

    S. Gururangan, S. Swayamdipta, O. Levy, R. Schwartz, S. R. Bowman, and N. A. Smith. 2018. Annotation artifacts in natural language inference data. arXiv preprint arXiv:1803.02324

    2018 arXiv

  4. [12]

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017. Visual Genome : Connecting language and vision using crowdsourced dense image annotations. International Journal of Co...

    2017

  5. [13]

    Omer Levy, Steffen Remus, Chris Biemann, and Ido Dagan. 2015. Do supervised distributional methods really learn lexical inference relations? In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tec...

    2015

  6. [14]

    Quinn McNemar. 1947. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2):153--157

    1947

  7. [15]

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? A new dataset for open book question answering. In In proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP)

    2018

  8. [16]

    Silviu Paun, Bob Carpenter, Jon Chamberlain, Dirk Hovy, Udo Kruschwitz, and Massimo Poesio. 2018. https://doi.org/10.1162/tacl_a_00040 Comparing bayesian models of annotation . Transactions of the Association for Computational Linguistics, 6:571--585

    2018 doi

  9. [17]

    Poliak, J

    A. Poliak, J. Naradowsky, A. Haldar, R. Rudinger, and B. V. Durme. 2018. Hypothesis only baselines in Natural Language Inference . arXiv preprint arXiv:1805.01042

    2018 arXiv

  10. [18]

    Rajpurkar, J

    P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. 2016. SQuAD : 100,000+ questions for machine comprehension of text. In Empirical Methods in Natural Language Processing (EMNLP)

    2016

  11. [19]

    Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for squad. In Association for Computational Linguistics (ACL)

    2018

  12. [20]

    Richardson, C

    M. Richardson, C. J. Burges, and E. Renshaw. 2013. MCTest : A challenge dataset for the open-domain machine comprehension of text. In Empirical Methods in Natural Language Processing (EMNLP), pages 193--203

    2013

  13. [21]

    Marta Sabou, Kalina Bontcheva, Leon Derczynski, and Arno Scharl. 2014. Corpus annotation through crowdsourcing: Towards best practice guidelines. In LREC, pages 859--866

    2014

  14. [22]

    Schwartz, M

    R. Schwartz, M. Sap, Y. Konstas, L. Zilles, Y. Choi, and N. A. Smith. 2017. The effect of different writing tasks on linguistic style: A case study of the ROC story cloze task. In Computational Natural Language Learning (CoNLL)

    2017

  15. [23]

    Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In Thirty-First AAAI Conference on Artificial Intelligence

    2017

  16. [24]

    Alane Suhr, Stephanie Zhou, Iris Zhang, Huajun Bai, and Yoav Artzi. 2018. A corpus for reasoning about natural language grounded in photographs. CoRR, abs/1811.00491

    2018 arXiv

  17. [25]

    Talmor, J

    A. Talmor, J. Herzig, N. Lourie, and J. Berant. 2019. CommonsenseQA : A question answering challenge targeting commonsense knowledge. In North American Association for Computational Linguistics (NAACL)

    2019

  18. [26]

    Trischler, T

    A. Trischler, T. Wang, X. Yuan, J. Harris, A. Sordoni, P. Bachman, and K. Suleman. 2017. NewsQA : A machine comprehension dataset. In Workshop on Representation Learning for NLP

    2017

  19. [27]

    Masatoshi Tsuchiya. 2018. Performance impact caused by hidden bias of training data for recognizing textual entailment. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan

    2018

  20. [28]

    Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. http://aclweb.org/anthology/N18-1101 A broad-coverage challenge corpus for sentence understanding through inference . In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computationa...

    2018

  21. [29]

    Zaidan and Chris Callison-Burch

    Omar F. Zaidan and Chris Callison-Burch. 2011. Crowdsourcing translation: Professional quality from non-professionals. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies

    2011

  22. [30]

    Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. From recognition to cognition: Visual commonsense reasoning. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    2019

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.