Pith. sign in

REVIEW 3 major objections 5 minor 53 references

Aligning NLP Models with Target Population Perspectives using PAIR: Population-Aligned Instance Replication

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read PAIR replication fixes calibration skew from biased annotators

desk verdict PAIR is a clean transfer of post-stratification from survey sampling to annotator composition; the offensive-language simulation is convincing, but the paper overclaims hate speech and never tests heterogeneous bias. read the letter →

arxiv 2501.06826 v3 pith:SWLWFMRM submitted 2025-01-12 stat.ME cs.CL

Stephanie Eckman , Bolei Ma , Christoph Kern , Rob Chew , Barbara Plank , Frauke Kreuter This is my paper · ORCID
classification stat.MEcs.CL
keywords PAIRcalibrationannotatorbiaspost-stratificationdatareplicationhatespeechdetectionoffensivelanguagesimulationstudy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Models trained on crowdsourced annotations can inherit the blind spots of the people who label the data, especially for subjective tasks like offensive language and hate speech detection. This paper argues that poorly calibrated predictions often trace back to an annotator pool that does not mirror the target population, and proposes PAIR, a post-processing fix that requires no additional annotations: replicate annotations from underrepresented annotator groups until the training data matches population proportions. In simulation studies built from 3,000 tweets with two simulated annotator types and bias offsets from 5 to 30 percentage points, models trained on non-representative pools show calibration error (Absolute Calibration Bias) that grows with the bias, while accuracy stays roughly flat. Applying PAIR brings calibration close to models trained on a representative pool, and the improvement is clearest on ambiguous tweets, where calibration matters most.

What carries the argument

The central object is the PAIR algorithm, a post-stratification weighting scheme that constructs a pseudo-population from the training data. Each annotator stratum $s$ gets weight $w_{s,i} = P_s / S_s$, where $P_s$ is the population share and $S_s$ the sample share; weights are normalized by a constant $K$ and each annotation is deterministically replicated $\mathrm{round}(w_i^{\mathrm{normalized}}) - 1$ times. The algorithm assumes access to annotator-level characteristics, population benchmark shares, and a simulated two-type annotator model (Eqs. 4--5) with bias $\beta$ shifting the probability of a positive label. The evaluation load-bearing piece is the Absolute Calibration Bias (ACB), which compares model predicted probabilities to the annotator-agreement proportion $p_i$, avoiding binning and giving a direct L1 view of miscalibration.

What would settle it

Train pairs of models on the same texts and same annotator pool, but with weights computed from two different sets of characteristics; if calibration does not improve when the weighted characteristics are the ones that actually drive disagreement, the replication step is not doing the work. A concrete version: take an existing multi-annotator dataset with rich demographic metadata, sub-sample the annotators to create known imbalance, apply PAIR, and measure test-set ACB on held-out items whose disagreement was driven by annotator identity rather than by the single additive shift.

Watch

Extended reading notes

Core claim

On the paper's own terms, PAIR establishes that a survey-style post-stratification adjustment applied to annotations, rather than to people, transfers the benefits of representative training data to models trained on skewed annotator pools. For offensive language, models trained on the PAIR-adjusted dataset track the representative dataset's Absolute Calibration Bias across every bias level, while both unbalanced datasets show higher ACB that rises with $\beta$. For hate speech, the full-set results are noisier because the class is rare, but restricting to difficult tweets with $0.4 \leq p_i \leq 0.6$ reveals the same pattern: adjusted models perform similarly to representative models and better than non-representative ones on both calibration and F1. The paper also reports that binary accuracy metrics obscure annotator-pool effects that calibration metrics expose, and it closes with a four-step recipe: identify annotator characteristics that matter, collect them alongside annotations, obtain population benchmarks, and use the resulting weights in training.

Load-bearing premise

The whole method rests on the assumption that annotator differences can be captured by a single shift in the probability of a positive label that is the same for every item; if the characteristics that drive disagreement vary from item to item or interact with context, the replication weights calculated from this simple model may not transfer.

Editorial extensions

If this is right

  • Models trained on non-representative annotator pools carry a calibration penalty that grows with annotator bias; accuracy metrics alone will miss this penalty.
  • Applying PAIR to skewed training data yields calibration close to a representative dataset's without collecting new annotations, so annotation budgets can be spent on breadth rather than re-collection.
  • For rare classes such as hate speech, PAIR's benefit is clearest on ambiguous instances, which suggests practitioners should evaluate calibration on difficult subsets rather than only on the full test set.
  • The four-step recipe (identify influential annotator characteristics, collect them, obtain population benchmarks, weight and replicate) gives an actionable pipeline for reducing annotator-pool bias in subjective NLP tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The uniform additive-shift bias model is the hinge: if real annotator disagreement is concentrated on ambiguous items and heterogeneous across annotators, fixed replication weights may under- or over-correct; a natural extension is to estimate $\beta$ per tweet-difficulty strata and use strata-specific weights.
  • PAIR is a data-space analogue of importance weighting in the loss; one testable extension is comparing replication to soft weights during training, which would avoid inflating dataset size while achieving the same calibration target.
  • The same pseudo-population logic could be applied to multi-class and multi-label tasks, where bias might shift different labels in different directions, and to align with intersectional demographic strata.
  • Because PAIR corrects calibration rather than accuracy, it is most consequential for downstream systems that threshold confidence scores, such as content moderation triage systems.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes PAIR (Population-Aligned Instance Replication), a post-processing method that reweights training data from non-representative annotator pools to match target population proportions by replicating annotations from underrepresented groups. The method is tested in a simulation study on offensive language and hate speech detection using 3,000 tweets with simulated A/B annotator types and a bias parameter beta. The authors fine-tune RoBERTa on representative, two non-representative, and PAIR-adjusted datasets and evaluate calibration (ACB) and accuracy (F1). They find that non-representative pools degrade calibration while leaving accuracy largely unchanged; PAIR-adjusted models track representative models for offensive language on the full set and for both tasks on a difficult-tweet subset, while hate speech results on the full set are inconclusive. The paper includes recommendations for collecting annotator characteristics and weighting training data.

Significance. If the central claim holds, PAIR offers a cheap and interpretable method for mitigating annotator-pool bias without additional data collection, bridging survey statistics and NLP. The paper is honest about limitations, releases code, and provides a reproducible simulation framework. The main value is in demonstrating that calibration, not just accuracy, is sensitive to annotator composition and that post-stratification can help in stylized settings. However, the strength of the evidence is limited by the homogeneity of the simulated bias model, the use of the original annotation proportion as the evaluation target, and the lack of repeated simulation draws.

major comments (3)
  1. [Sec. 4.1, Eqs. (4)-(6)] The evaluation target in ACB (Eq. 6) is the original annotation proportion pi, but the simulated population proportion for tweet i is 0.5*pA_i + 0.5*pB_i, which equals pi only when neither pA_i nor pB_i is truncated at 0 or 1. For boundary tweets, especially in the rare hate-speech class, the population proportion differs from pi by up to beta/2. Thus the reported calibration of the PAIR-adjusted model is measured against pi, not against the target population distribution that PAIR is designed to match. Please report ACB against the true population proportion (0.5*pA_i + 0.5*pB_i) as well as against pi, or restrict the analysis to non-boundary tweets; otherwise the central claim that PAIR corrects calibration problems is not directly supported by the metric used.
  2. [Sec. 4.1 and Sec. 5.3] The simulation assumes a single additive bias beta on the probability scale that is identical across all tweets and identical for all annotators within each type. The sensitivity analysis in Sec. 5.3 filters to difficult tweets (0.4 <= pi <= 0.6) but retains the same homogeneous beta; it does not test heterogeneous or content-dependent bias. If beta varies per tweet or by content, the post-stratification weights in Eqs. (1)-(3), which depend only on marginal group shares, do not guarantee that the weighted training distribution matches the population's conditional label distribution. Please add a simulation with tweet-level heterogeneous bias (e.g., beta_i drawn from a distribution or increasing with ambiguity) or state precisely the conditions under which marginal reweighting suffices; otherwise the transportability of PAIR to real annotator disagreement remains unsupported.
  3. [Sec. 4.1 and Sec. 7] For each beta value, only one realization of the Bernoulli annotation draws is used to construct each dataset, and the five random seeds average over model training only, not over annotation sampling. Figures 5, 7, 11, and 13 therefore show training-seed variability but not sampling variability in the annotation generation process. The Limitations section acknowledges this, but the paper does not quantify how much of the observed ACB differences between datasets could be due to a single draw. Please repeat the annotation simulation many times (e.g., 20-50 replicates) and report intervals over these replicates, or at least bootstrap the annotation draw; without this, the claim that PAIR improves calibration over non-representative datasets for OL cannot be distinguished from noise for small differences.
minor comments (5)
  1. [Figures 5, 7, 11, 13] The legend in several figures uses 'Balanced' while the text and Table 1 use 'Representative'; please use consistent dataset names throughout.
  2. [Sec. 5.2] The sentence 'All models show lower average ACB values than the representative model across a wide range of the bias offset (0.10 - 0.20)' is confusing: if the adjusted model roughly tracks the representative model, it cannot be that all models have lower ACB than the representative model unless 'representative' is a typo for 'non-representative'. Please clarify the intended comparison.
  3. [Sec. 4.2] The choice of normalization constant K = 4/3 is explained arithmetically but it might be worth stating explicitly that any positive K leaves relative weights unchanged and that K was chosen to make the smaller weight equal to 1 for interpretability.
  4. [Eq. (1)] The notation ws,i defines a stratum-level weight but the subscript i suggests unit-level variation; since the weight is constant within stratum s, consider writing w_s to avoid confusion.
  5. [Abstract] The abstract states that PAIR 'corrects these calibration problems' without noting that the hate-speech full-data results in Sec. 5.2 do not show improvement; please qualify the claim to match the presented evidence.

Circularity Check

2 steps flagged · score 6.0 of 10

PAIR's simulation success is partly by construction: the ACB target pi is the mean of the two simulated group probabilities, and PAIR reweights to the known 50/50 population mix.

  1. fitted input called prediction [Section 4.1–4.3, Equations (4), (5), (6)]
    "pA_i,OL = max(pi,OL − β, 0) (4) pB_i,OL = min(pi,OL + β, 1) (5) ... ACBOL = 1/n Σ |predsi,OL − pi,OL| (6)"

    Under the simulation, the population is exactly 50% A and 50% B, and the two group probabilities are symmetric around pi (away from boundary truncation). PAIR's weights replicate B annotations so that the adjusted dataset has 6 A and 6 B annotations per tweet, making the expected annotation proportion (pA+pB)/2 = pi by construction. The ACB metric then measures the model against this same pi. The central result, that PAIR-trained models have ACB close to Representative and lower than unbalanced datasets, is therefore baked into the data-generating process: reweighting to the known 50/50 mix recovers the target pi in expectation before any model is trained. The simulation cannot fail under its own assumptions, so the 'prediction' that PAIR corrects calibration is forced by construction.

  2. self definitional [Section 4.2, Applying PAIR Algorithm]
    "The population proportions, by construction, are PA = 0.5, PB = 0.5. Applying (1), we get wA,i = 0.75, wB,i = 1.5. ... These weights give us nA,i = 0, nB,i = 1, which leads us to replicate all B annotations in the Non-representative 1 Dataset (see Table 1)."

    The Adjusted dataset is defined to be the Non-representative 1 dataset plus one replicated copy of every B annotation, yielding exactly the same 6A/6B per-tweet composition as the Representative dataset. Thus 'Adjusted' and 'Representative' are two versions of the same post-stratified multinomial distribution by construction; comparing them measures the effect of reweighting a known target composition against an independent draw from that same composition, not an independent validation of the weighting scheme. The improvement over the unbalanced pools follows from the definition of the adjusted set rather than from a falsifiable empirical test.

full rationale

The paper is transparent about its simulation design and states in Section 7 that it makes strong assumptions: only two annotator types, homogeneous within type, and additive bias on the probability scale. However, transparency does not remove the by-construction character of the headline result. In the generative model, pi is the midpoint of the two group probabilities, and PAIR's replication exactly restores the 50/50 group mix, so the expected annotation proportion in the adjusted training data equals pi, which is also the ACB ground truth. The demonstrated calibration improvement of Adjusted over Non-representative datasets is therefore an algebraic consequence of the simulation setup rather than an empirical discovery about real annotator pools. The sensitivity analysis in Section 5.3 filters to difficult tweets but keeps the same homogeneous beta, so it does not test the heterogeneous-bias scenario that the paper itself flags as the likely real-world situation. The method PAIR itself is standard post-stratification and the paper credits survey-sampling literature for it; the self-citations to Eckman et al. (2024) for weighting theory and to Kern et al. (2023) for the base dataset are not load-bearing circularity. The central empirical claim, however, reduces by construction to the identity (pA+pB)/2 = pi under the paper's own equations, which warrants a score of 6 rather than a clean bill.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new physical or conceptual entities. Its free parameter is the simulation bias beta, and its main assumptions are the stylized two-type annotator model and the use of pi as the reference probability. The post-stratification axioms are standard from survey statistics.

free parameters (1)
  • beta (bias magnitude) = 0.05 to 0.30 in steps of 0.05
    Simulation parameter controlling the probability shift between annotator types A and B. All results are conditional on this range, which the authors state covers 'most reasonable situations' (Section 4.1).
assumptions (3)
  • standard math Post-stratification weights (Eq. 1: ws,i = Ps / Ss) correctly align a sample to a population when stratum shares are known.
    The paper relies on the standard survey-sampling theory of post-stratification (Bethlehem et al. 2011, Valliant et al. 2013) as the foundation for PAIR.
  • domain assumption Annotator behavior is modeled by Bernoulli draws with probabilities pA and pB that are deterministic shifts of the original pi (Eqs. 4-5).
    This is the central simulation assumption about how annotator types differ. The paper acknowledges it is stylized and that real-world biases may be more nuanced (Section 7).
  • domain assumption The original 12-annotator proportion pi is a valid reference for the 'true' population probability in the simulation and as the evaluation target.
    The simulation defines the two annotator-type probabilities as shifts of pi and the ACB metric compares model predictions to pi. This makes pi a load-bearing anchor for both data generation and evaluation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Aligning NLP Models with Target Population Perspectives using PAIR: Population-Aligned Instance Replication." pith.science (2026). https://pith.science/paper/SWLWFMRM

@misc{pith2026250106826,
  author       = {Pith},
  title        = {Pith review of: Aligning NLP Models with Target Population Perspectives using PAIR: Population-Aligned Instance Replication},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SWLWFMRM}},
  note         = {Machine review of arXiv:2501.06826}
}
read the original abstract

Models trained on crowdsourced annotations may not reflect population views, if those who work as annotators do not represent the broader population. In this paper, we propose PAIR: Population-Aligned Instance Replication, a post-processing method that adjusts training data to better reflect target population characteristics without collecting additional annotations. Using simulation studies on offensive language and hate speech detection with varying annotator compositions, we show that non-representative pools degrade model calibration while leaving accuracy largely unchanged. PAIR corrects these calibration problems by replicating annotations from underrepresented annotator groups to match population proportions. We conclude with recommendations for improving the representativity of training data and model performance.

Figures

Figures reproduced from arXiv: 2501.06826 by the authors.

Figure 1
Figure 1. Top: Adjusting survey data to match popula￾tion produces high quality results. Bottom: Can a similar adjustment in data annotations also improve model performance? et al., 2024). The influence of annotator charac￾teristics underscores that language understanding is not a single objective truth but a constellation of equally valid interpretations anchored in dif￾ferent lived experiences. When this perspectivist inter… view at source ↗
Figure 2
Figure 2. Distribution of pi,OL and pi,HS in original data The population contains two types of people (50% each). Type A people are less likely to say a tweet contains OL. Type B people are more likely: p A i,OL = max(pi,OL − β, 0) (4) p B i,OL = min(pi,OL + β, 1) (5) Here β captures the magnitude of the bias. We vary β from [0.05, 0.3] by 0.05, corresponding to an increase or decrease in the probability to judge a tweet as … view at source ↗
Figure 3
Figure 3. Percentage of instances annotated as OL, by [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Percentage of instances annotated as HS, by [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: ACB scores for OL Models, by dataset and [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: F1 scores for OL Models, by dataset and bias [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 7
Figure 7. Figure 7: ACB scores for HS Models, by dataset and [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: F1 scores for HS Models, by dataset and bias [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]
Figure 11
Figure 11. Figure 11: ACB scores for OL Models, on difficult tweets (0.4 ≤ pi,OL ≤ 0.6), by dataset and bias (β) 0.05 0.10 0.15 0.20 0.25 0.30 Bias 0.25 0.30 0.35 0.40 0.45 0.50 F1 OL (0.4 p_i 0.6), Accuracy Dataset Representative Non-representative 1 Non-representative 2 Adjusted [PITH_F…
Figure 14
Figure 14. Figure 14: F1 scores for HS Models, on difficult tweets [PITH_FULL_IMAGE:figures/full_fig_p007_14.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

53 extracted references · 18 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Lisa P Argyle, Ethan C Busby, Nancy Fulda, Joshua R Gubler, Christopher Rytting, and David Wingate. 2023. https://doi.org/10.1017/pan.2023.2 Out of one, many: Using language models to simulate human samples . Political Analysis, 31(3):339--355

    doi:10.1017/pan.2023.2 2023

  4. [4]

    Bender and Batya Friedman

    Emily M. Bender and Batya Friedman. 2018. https://doi.org/10.1162/tacl_a_00041 Data statements for natural language processing: Toward mitigating system bias and enabling better science . Transactions of the Association for Computational Linguistics, 6:587--604

    doi:10.1162/tacl_a_00041 2018

  5. [5]

    Berinsky, Gregory A

    Adam J. Berinsky, Gregory A. Huber, and Gabriel S. Lenz. 2012. https://doi.org/10.1093/pan/mpr057 Evaluating online labor markets for experimental research: Amazon.com’s mechanical turk . Political Analysis, 20(3):351–368

    work page doi:10.1093/pan/mpr057 2012

  6. [6]

    Jelke Bethlehem, Fannie Cobben, and Barry Schouten. 2011. https://doi.org/10.1002/9780470891056 Handbook of Nonresponse in Household Surveys . Wiley

    work page doi:10.1002/9780470891056 2011

  7. [7]

    Altman, Patrick Royston, and Roger L

    Andrea Burton, Douglas G. Altman, Patrick Royston, and Roger L. Holder. 2006. https://doi.org/10.1002/sim.2673 The design of simulation studies in medical statistics . Statistics in Medicine, 25(24):4279–4292

    doi:10.1002/sim.2673 2006

  8. [8]

    Toon Calders, Faisal Kamiran, and Mykola Pechenizkiy. 2009. https://doi.org/10.1109/ICDMW.2009.83 Building classifiers with independency constraints . In ICDMW '09: Proceedings of the 2009 IEEE International Conference on Data Mining Workshops, pages 13--18

    doi:10.1109/icdmw.2009.83 2009

Show all 53 references
  1. [9]

    Chandler and Gabriele Paolacci

    Jesse J. Chandler and Gabriele Paolacci. 2017. https://doi.org/10.1177/1948550617698203 Lie for a dime: When most prescreening responses are honest but most study participants are impostors . Social Psychological and Personality Science, 8(5):500–508

    2017 doi

  2. [10]

    Stephanie Eckman, Barbara Plank, and Frauke Kreuter. 2024. https://proceedings.mlr.press/v235/eckman24a.html Position: Insights from survey methodology can improve training data . In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceeding...

    2024

  3. [11]

    Marco Favier, Toon Calders, Sam Pinxteren, and Jonathan Meyer. 2023. https://doi.org/10.1007/s10994-023-06401-1 How to be fair? a study of label and selection bias . Machine Learning, 112(12):5081--5104

    2023 doi

  4. [12]

    Eve Fleisig, Rediet Abebe, and Dan Klein. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.415 When the majority is wrong: Modeling annotator disagreement for subjective tasks . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6...

    2023 doi

  5. [13]

    Eve Fleisig, Su Lin Blodgett, Dan Klein, and Zeerak Talat. 2024. https://doi.org/10.18653/v1/2024.naacl-long.126 The perspectivist paradigm shift: Assumptions and challenges of capturing human labels . In Proceedings of the 2024 Conference of the North American Chapter of the ...

    2024 doi

  6. [14]

    Tommaso Giorgi, Lorenzo Cima, Tiziano Fagni, Marco Avvenuti, and Stefano Cresci. 2025. https://doi.org/10.1609/icwsm.v19i1.35837 Human and llm biases in hate speech annotations: A socio-demographic analysis of annotators and targets . Proceedings of the International AAAI Conf...

    2025 doi

  7. [15]

    Ursula Hebert-Johnson, Michael Kim, Omer Reingold, and Guy Rothblum. 2018. https://proceedings.mlr.press/v80/hebert-johnson18a.html Multicalibration: Calibration for the ( C omputationally-identifiable) masses . In Proceedings of the 35th International Conference on Machine Le...

    2018

  8. [16]

    Olivia Huang, Eve Fleisig, and Dan Klein. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.64 Incorporating worker perspectives into MT urk annotation practices for NLP . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1010--10...

    2023 doi

  9. [17]

    Eyke H \"u llermeier and Willem Waegeman. 2021. https://doi.org/10.1007/s10994-021-05946-3 Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods . Machine learning, 110(3):457--506

    2021 doi

  10. [18]

    Faisal Kamiran and Toon Calders. 2012. https://doi.org/10.1007/s10115-011-0463-8 Data preprocessing techniques for classification without discrimination . Knowledge and Information Systems, 33(1):1--33

    2012 doi

  11. [19]

    Alan F. Karr. 2016. https://doi.org/10.1146/annurev-statistics-041715-033438 Data sharing and access . Annual Review of Statistics and Its Application, 3(Volume 3, 2016):113--132

    2016 doi

  12. [20]

    Lipton, and Alex John London

    Divyansh Kaushik, Zachary C. Lipton, and Alex John London. 2024. https://doi.org/10.1145/3641858 Resolving the human-subjects status of ml's crowdworkers . Commun. ACM, 67(5):52–59

    2024 doi

  13. [21]

    Christoph Kern, Stephanie Eckman, Jacob Beck, Rob Chew, Bolei Ma, and Frauke Kreuter. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.992 Annotation sensitivity: Training data collection methods affect model performance . In Findings of the Association for Computational ...

    2023 doi

  14. [22]

    Hannah Rose Kirk, Alexander Whitefield, Paul R\" o ttger, Andrew Bean, Katerina Margatina, Juan Ciro, Rafael Mosquera, Max Bartolo, Adina Williams, He He, Bertie Vidgen, and Scott A. Hale. 2024. https://proceedings.neurips.cc/paper_files/paper/2024/file/be2e1b68b44f2419e19f6c3...

    2024

  15. [23]

    Elisa Leonardelli, Gavin Abercrombie, Dina Almanea, Valerio Basile, Tommaso Fornaciari, Barbara Plank, Verena Rieser, Alexandra Uma, and Massimo Poesio. 2023. https://doi.org/10.18653/v1/2023.semeval-1.314 S em E val-2023 task 11: Learning with disagreements ( L e W i D i) . I...

    2023 doi

  16. [24]

    Ling and Chenghui Li

    Charles X. Ling and Chenghui Li. 1998. https://cdn.aaai.org/KDD/1998/KDD98-011.pdf Data mining for direct marketing: problems and solutions . In Proceedings of the Fourth International Conference on Knowledge Discovery and Data Mining, KDD'98, page 73–79. AAAI Press

    1998

  17. [25]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. https://arxiv.org/abs/1907.11692 Roberta: A robustly optimized bert pretraining approach . Preprint, arXiv:1907.11692

    2019 arXiv

  18. [26]

    London Lowmanstone, Ruyuan Wan, Risako Owan, Jaehyung Kim, and Dongyeop Kang. 2023. https://ceur-ws.org/Vol-3494/paper10.pdf Annotation imputation to individualize predictions: Initial studies on distribution dynamics and model predictions . In NLPerspectives@ECAI

    2023

  19. [27]

    Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. 2021. https://doi.org/10.1145/3457607 A Survey on Bias and Fairness in Machine Learning . ACM Computing Surveys, 54(6):1--36

    2021 doi

  20. [28]

    Negar Mokhberian, Myrl Marmarelis, Frederic Hopp, Valerio Basile, Fred Morstatter, and Kristina Lerman. 2024. https://doi.org/10.18653/v1/2024.naacl-long.407 Capturing perspectives of crowdsourced annotators in subjective learning tasks . In Proceedings of the 2024 Conference ...

    2024 doi

  21. [29]

    Morris, Ian R

    Tim P. Morris, Ian R. White, and Michael J. Crowther. 2019. https://doi.org/10.1002/sim.8086 Using simulation studies to evaluate statistical methods . Statistics in Medicine, 38(11):2074–2102

    2019 doi

  22. [30]

    Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. 2015. https://doi.org/10.1609/aaai.v29i1.9602 Obtaining well calibrated probabilities using bayesian binning . Proceedings of the AAAI Conference on Artificial Intelligence, 29(1)

    2015 doi

  23. [31]

    Matthias Orlikowski, Jiaxin Pei, Paul R \"o ttger, Philipp Cimiano, David Jurgens, and Dirk Hovy. 2025. https://aclanthology.org/2025.acl-long.104/ Beyond demographics: Fine-tuning large language models to predict individuals' subjective text perceptions . In Proceedings of th...

    2025

  24. [32]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and...

    2022 arXiv

  25. [33]

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu ...

    2019

  26. [34]

    Jiaxin Pei and David Jurgens. 2023. https://doi.org/10.18653/v1/2023.law-1.25 When do annotator demographics matter? measuring the influence of annotator demographics with the POPQUORN dataset . In Proceedings of the 17th Linguistic Annotation Workshop (LAW-XVII), pages 252--2...

    2023 doi

  27. [35]

    Barbara Plank. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.731 The ``problem'' of human label variation: On ground truth in data, modeling and evaluation . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10671--10682, Abu ...

    2022 doi

  28. [36]

    Vinodkumar Prabhakaran, Aida Mostafazadeh Davani, and Mark Diaz. 2021. https://doi.org/10.18653/v1/2021.law-1.14 On releasing annotator-level labels and information in datasets . In Proceedings of the Joint 15th Linguistic Annotation Workshop (LAW) and 3rd Designing Meaning Re...

    2021 doi

  29. [37]

    Andreas Quatember. 2015. https://doi.org/10.1007/978-3-319-11785-0 Pseudo-Populations: A Basic Concept in Statistical Surveys . Springer

    2015 doi

  30. [38]

    Rebecca Roelofs, Nicholas Cain, Jonathon Shlens, and Michael C. Mozer. 2022. https://proceedings.mlr.press/v151/roelofs22a.html Mitigating bias in calibration error estimation . In Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, volu...

    2022

  31. [39]

    Esther Rolf, Theodora T Worledge, Benjamin Recht, and Michael Jordan. 2021. https://proceedings.mlr.press/v139/rolf21a.html Representation matters: Assessing the importance of subgroup allocations in training data . In Proceedings of the 38th International Conference on Machin...

    2021

  32. [40]

    Sebastin Santy, Jenny Liang, Ronan Le Bras, Katharina Reinecke, and Maarten Sap. 2023. https://doi.org/10.18653/v1/2023.acl-long.505 NLP ositionality: Characterizing design biases of datasets and models . In Proceedings of the 61st Annual Meeting of the Association for Computa...

    2023 doi

  33. [41]

    Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A. Smith. 2022. https://doi.org/10.18653/v1/2022.naacl-main.431 Annotators with attitudes: How annotator beliefs and identities bias toxic language detection . In Proceedings of the 2022 Conference...

    2022 doi

  34. [42]

    Andrew Smart, Ding Wang, Ellis Monk, Mark Díaz, Atoosa Kasirzadeh, Erin Van Liemt, and Sonja Schmer-Galunder. 2024. https://arxiv.org/abs/2402.06811 Discipline and label: A weird genealogy and social theory of data annotation . Preprint, arXiv:2402.06811

    2024 arXiv

  35. [43]

    Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell L Gordon, Niloofar Mireshghallah, Christopher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, Tim Althoff, and Yejin Choi. 2024. https://proceedings.mlr.press/v235/sorensen24a.html Position: A roadmap to pl...

    2024

  36. [44]

    Huaman Sun, Jiaxin Pei, Minje Choi, and David Jurgens. 2025. https://aclanthology.org/2025.naacl-short.71/ Sociodemographic prompting is not yet an effective approach for simulating subjective judgments with LLM s . In Proceedings of the 2025 Conference of the Nations of the A...

    2025

  37. [45]

    Amos Tversky and Daniel Kahneman. 1974. https://doi.org/10.1126/science.185.4157.1124 Judgment under uncertainty: Heuristics and biases: Biases in judgments reveal some heuristics of thinking under uncertainty. science, 185(4157):1124--1131

    1974

  38. [46]

    Alexandra N Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. 2021. https://doi.org/10.1613/jair.1.12752 Learning from disagreement: A survey . Journal of Artificial Intelligence Research, 72:1385--1470

    2021 doi

  39. [47]

    Richard Valliant. 2019. https://doi.org/10.1093/jssam/smz003 Comparing alternatives for estimation from nonprobability samples . Journal of Survey Statistics and Methodology, 8(2):231–263

    2019 doi

  40. [48]

    Richard Valliant, Jill A Dever, and Frauke Kreuter. 2013. https://doi.org/10.1007/978-3-319-93632-1 Practical tools for designing and weighting survey samples , volume 1. Springer

    2013 doi

  41. [49]

    McLernon, Maarten van Smeden, Laure Wynants, Ewout W

    Ben Van Calster, David J. McLernon, Maarten van Smeden, Laure Wynants, Ewout W. Steyerberg, Patrick Bossuyt, Gary S. Collins, Petra Macaskill, David J. McLernon, Karel G. M. Moons, Ewout W. Steyerberg, Ben Van Calster, Maarten van Smeden, and Andrew J. Vickers. 2019. https://d...

    2019 doi

  42. [50]

    Nikolas Vitsakis, Amit Parekh, and Ioannis Konstas. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.696 Voices in a crowd: Searching for clusters of unique perspectives . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 12517--...

    2024 doi

  43. [51]

    Ruyuan Wan, Jaehyung Kim, and Dongyeop Kang. 2023. https://doi.org/10.1609/aaai.v37i12.26698 Everyone’s voice matters: Quantifying annotation disagreement using demographic information . Proceedings of the AAAI Conference on Artificial Intelligence, 37(12):14523--14530

    2023 doi

  44. [52]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

    2020 doi

  45. [53]

    Zhisheng Zhong, Jiequan Cui, Shu Liu, and Jiaya Jia. 2021. https://doi.org/10.1109/CVPR46437.2021.01622 Improving calibration for long-tailed recognition . In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16484--16493

    2021

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.