Pith. sign in

REVIEW 4 major objections 5 minor 34 references

Showing a language model a handful of an annotator's past labels lets it predict that annotator's future labels, and pooling those predictions produces competitive soft labels.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

In-context learning with up to 10 demonstrations per annotator can mimic individual annotator behavior well enough to produce competitive soft labels on four disagreement-rich datasets.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection Solid shared-task paper with an underdetermined ICL claim—missing the obvious no-personalization control—but an honest, useful look at LDL for perspectivist NLP. the 4 major comments →

arxiv 2509.09524 v1 pith:ZFCIYYGS submitted 2025-09-11 cs.CL cs.LG

DeMeVa at LeWiDi-2025: Modeling Perspectives with In-Context Learning and Label Distribution Learning

Daniil Ignatev , Nan Li , Hugh Mee Wong , Anh Dang , Shane Kaszefski Yaschuk This is my paper
classification cs.CL cs.LG
keywords in-context learningperspectivist annotationlabel distribution learningsoft labelsannotator disagreementdemonstration samplingLikert scalesshared task
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Human annotators often disagree on subjective language tasks, and traditional majority-vote aggregation throws away the signal in that disagreement. This paper tests two ways to preserve it: predicting each annotator's judgment separately, and predicting the full distribution of labels. The authors show that in-context learning—prompting a large language model with ten of an annotator's earlier labels—can track that annotator's personal labeling style, and that pooling these per-annotator predictions yields soft labels that perform competitively. They also argue that label distribution learning techniques, such as cumulative-distance losses and clustering, transfer well to perspectivist NLP, with a direct link between a proposed loss and the evaluation metric.

Core claim

The central discovery is that a pretrained language model, with no task-specific fine-tuning, can act as a proxy for a specific annotator if it is shown a small sample of that annotator's past labels. For each test item, the system retrieves ten demonstrations from the annotator's training annotations—either by semantic similarity with redundancy penalty or by stratified sampling over the annotator's label distribution—and prompts the model to 'guess my response.' The predicted labels are then averaged across annotators to form a soft label distribution. The paper reports that this two-step procedure beats simple baselines on all four datasets and matches or exceeds fine-tuned approaches on

What carries the argument

The key mechanism is demonstration selection inside an in-context-learning prompt. For each annotator and each test input, the pipeline builds a prompt from ten of that annotator's past annotations using one of two sampling strategies: similarity-based retrieval with maximal marginal relevance, or stratified label-based sampling that preserves the annotator's empirical label proportions. The prompt asks the model to infer the annotator's response; these per-annotator predictions are then aggregated by averaging into a soft label. The fine-tuning strand relies on the identity that the cumulative absolute distance (CAD) between two discrete distributions equals the one-dimensional Wasserstein

Load-bearing premise

The pipeline assumes that ten past labels from an annotator are enough for a language model to learn and generalize that annotator's decision heuristics to new items; the paper's own qualitative analysis shows models often fall back to unanimous, common-sense predictions on contentious items, so this assumption can fail.

What would settle it

On a held-out set of items where annotators disagree, take an annotator whose labels are idiosyncratic (e.g., predominantly using a middle scale value), give the model ten of their labels, and check whether its predictions on the remaining items beat a majority-vote baseline for that annotator; if it does not for several such annotators, the claim that ICL captures individual perspectives fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Per-annotator modeling becomes feasible without training a separate model: any sampler that can find past examples for an annotator can produce that annotator's likely labels on new items.
  • Label-stratified sampling improves calibration of numeric predictions (e.g., Likert-scale values) more than similarity-based sampling, and the benefit grows when annotator explanations are included in the prompt.
  • Including explanations in prompts improves the derived soft labels (Task A) more than the per-annotator error rate (Task B), suggesting the explanations mainly calibrate the distribution shape.
  • Using cumulative absolute distance as a loss function for Likert-scale prediction is principled because it equals the 1D Wasserstein distance used in evaluation, and it yields scores competitive with in-context learning on the sarcasm dataset.
  • Clustering items into groups with similar label distributions and training a multitask model to predict both the soft label and the cluster can outperform baselines, but underperforms the ICL pipeline, pointing to headroom for LDL methods.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to reduce annotation cost: have each annotator label only a small personal training set and use ICL to impute their labels for the rest of a corpus, effectively multiplying each annotator's coverage.
  • The observed failure mode—models defaulting to unanimous, common-sense labels on items where annotators disagree—suggests alignment-induced priors may suppress pluralism; explicitly prompting with disagreement examples or tuning decoding temperature could be a direct test.
  • Because CAD equals 1D Wasserstein, any NLP task with ordinal labels evaluated by distribution distance can adopt CAD as a loss; this bridges the train/eval gap for Likert-scale soft-label prediction beyond the two datasets studied.
  • A concrete next experiment would vary k (the number of demonstrations) to map how quickly per-annotator accuracy saturates, and to test whether annotators with skewed label distributions need more (or fewer) examples than k=10.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper describes the DeMeVa team's submission to the LeWiDi 2025 shared task. For Task B (perspectivist prediction) and Task A (soft-label prediction), the authors propose a two-step ICL pipeline: first, GPT-4o, Claude Haiku 3.5, or Llama 3.1 70B are prompted with k=10 demonstrations sampled from an individual annotator's past labels; second, the predicted labels are aggregated into soft labels. Two sampling strategies are compared (similarity-based MMR and stratified label-based), and the effect of including annotator explanations is tested on Par and VariErrNLI. The authors also fine-tune RoBERTa using ordinal label distribution losses (cumulative Jensen-Shannon divergence, cumulative absolute distance) and a clustering-based multitask approach, reporting Wasserstein distances. The paper claims that ICL can effectively predict annotator-specific annotations and that LDL methods are promising; the team ranked 2nd on the shared task leaderboard.

Significance. If the central ICL claim were fully established, the paper would be a useful contribution to perspectivist NLP: it would show that a small number of annotator-specific demonstrations can steer LLMs toward individual labeling behavior, and that aggregating such predictions yields competitive soft labels. The paper also helpfully connects ordinal LDL losses to the 1D Wasserstein distance, and it is transparent about negative results and failure modes. The shared-task leaderboard rank provides external validation of the pipeline's overall utility. However, the central claim is underdetermined by the reported experiments: the design lacks the crucial control of conditioning on non-annotator-specific demonstrations, and the ICL results have no error bars or significance tests. The qualitative analysis in Section 3.4 partly undermines rather than supports the claim, and the authors themselves concede in Section 5 that the ICL comparison is 'not yielding fully conclusive results.'

major comments (4)
  1. [Section 3.3, Table 2] The central contribution 'ICL can effectively predict annotator-specific annotations' is not supported because the experiments never include a zero-shot condition or a condition in which demonstrations are taken from other annotators or pooled across annotators. The prompt in Fig. 2 instructs the model to learn 'my response behavior' from 'my previous responses,' but without a non-personalized control the observed gains over the majority baseline could be due to GPT-4o's general language understanding or to the label-range cue in the prompt, rather than to perspectivist learning. Section 3.4's observation that models default to unanimous/common-sense labels on ambiguous items is exactly what a non-personalized model would do. Add at least a zero-shot baseline and a condition with demonstrations sampled from annotators other than the target annotator; report these alongside Table 2.
  2. [Section 3.3, Table 2] All ICL results appear to come from a single run at temperature 0.0. The differences between sampling strategies are often tiny (e.g., GPT-4o+sim vs GPT-4o+strat on MP Task A: 0.466 vs 0.469; on CSC Task A: 0.84 vs 0.792), and there are no standard deviations, confidence intervals, or significance tests. The paper draws qualitative conclusions (e.g., 'label-based sampling often results in improvements on Task A') from differences that may be within run-to-run noise. Report repeated runs with variance estimates, and where possible perform paired significance tests across items.
  3. [Section 3.4] The claim that 'models are generally successful in mimicking specific annotators' labeling strategies' rests on a single anecdotal example (Ann3 in Par) and a qualitative description of VariErrNLI. This is not sufficient to support the central claim. Provide quantitative per-annotator metrics: for each annotator, compare predicted labels against that annotator's held-out labels, and compare against a zero-shot or pooled-demonstration baseline. Also quantify the frequency of the 'common-sense default' failure mode described for MP-dev-1597. The paper's own statement that 'complete pluralistic alignment ... may be impossible' is compatible with the alternative explanation that the model is not learning annotator-specific behavior at all.
  4. [Section 4.1, Table 3] The fine-tuning claim that CAD/CAD+MAE 'can yield scores that are competitive with in-context learning' (0.797 vs 0.792 on CSC) is based on means without significance tests. The stated standard deviations (±0.01) suggest the difference is within noise. Since the paper's second contribution is a call for greater use of LDL methods, this comparison needs at least a variance estimate and, ideally, a statistical test or a statement that the difference is not significant.
minor comments (5)
  1. [Section 3.2] The definition 'L = max{|Y_a|, k}' is confusing when k is fixed at 10 and |Y_a| is usually smaller than 10; in that case the condition |D_a| ≤ L is simply |D_a| ≤ 10. Clarify the intended role of L, especially if the goal is to decide when stratification is feasible.
  2. [Section 3.4] The phrase 'possibly induced by RLHF' is speculative. The common-sense defaulting behavior could have many causes; consider rewording as a hypothesis rather than an explanation.
  3. [Section 2.1, Table 1] The Par dataset is described as 'as yet unpublished'; if a preprint or companion description exists, adding a citation would help readers locate the data.
  4. [Section 3.3] The explanation for MP results says 'the error rate is not significantly influenced' but no significance test is reported; rephrase to 'not substantially different' or add the test.
  5. [Section 4.2] The choice of k-means with 'a maximum of 5 clusters' and the tuning of α are not justified beyond a brief mention. A sentence on how these were selected (e.g., grid search on dev) would improve reproducibility.

Circularity Check

0 steps flagged

No significant circularity: the ICL predictions and LDL training are independent of the reported evaluation targets.

full rationale

The paper's central claims are empirical results from prompting LLMs with held-out training demonstrations and from fine-tuning RoBERTa with defined loss functions, both evaluated on separate test sets. No step reduces to its own input by construction: the ICL predictions are produced by the LLM given k=10 annotator-specific demonstrations and are not fitted to the test labels; the soft labels are aggregates of these independent predictions; and the LDL losses (CJS, CAD) are defined from first principles and used as training objectives, with the observation that CAD coincides with the 1D Wasserstein metric being a mathematical equivalence rather than a circular argument. The paper contains no load-bearing self-citations, and the acknowledged limitation that models often default to common-sense labels on ambiguous items is an empirical weakness (and the absence of a zero-shot control is a missing baseline), not circular reasoning. The derivation chain is therefore self-contained with respect to the evaluation.

Axiom & Free-Parameter Ledger

5 free parameters · 3 axioms · 0 invented entities

The paper introduces no new theoretical entities. Its central claims rest on empirical assumptions about LLM behavior and the reliability of sparse annotation distributions, plus several hand-set hyperparameters (k, lambda, cluster count, alpha). These are not fitted to derive a result but are chosen by hand and affect the reported performance.

free parameters (5)
  • k (number of demonstrations) = 10
    Hand-set for all ICL experiments; number of examples per annotator in the prompt.
  • lambda (MMR diversity weight) = 0.7
    Hand-set in the maximal marginal relevance formula to balance relevance and redundancy.
  • number of clusters (k-means on Par) = 3 (best)
    Chosen experimentally with a maximum of 5; affects the clustering-based LDL approach.
  • alpha (cluster loss weight) = tunable
    Weights the cluster classification loss in the total training loss; exact value not reported.
  • temperature = 0.0
    Set to enforce greedy decoding and minimal randomness in ICL outputs.
axioms (3)
  • domain assumption An LLM can infer an individual annotator's labeling policy from a small number of in-context examples.
    Load-bearing premise of the ICL pipeline; the paper motivates it with prior work and its own results, but does not prove it.
  • domain assumption Cosine similarity of sentence embeddings captures the kind of similarity relevant to annotation behavior.
    Used in similarity-based sampling; prior work suggests annotation heuristics are not purely topical, so this is an approximation.
  • domain assumption The empirical label distribution from four or five annotators is a reliable target for soft label evaluation.
    The paper itself discusses that sparse annotations are a 'noisy, undersampled proxy' (Section 4.3), yet uses them as ground truth.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of DeMeVa at LeWiDi-2025: Modeling Perspectives with In-Context Learning and Label Distribution Learning." pith.science (2026). https://pith.science/paper/ZFCIYYGS

@misc{pith2026250909524,
  author       = {Pith},
  title        = {Pith review of: DeMeVa at LeWiDi-2025: Modeling Perspectives with In-Context Learning and Label Distribution Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZFCIYYGS}},
  note         = {Machine review of arXiv:2509.09524}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

This system paper presents the DeMeVa team's approaches to the third edition of the Learning with Disagreements shared task (LeWiDi 2025; Leonardelli et al., 2025). We explore two directions: in-context learning (ICL) with large language models, where we compare example sampling strategies; and label distribution learning (LDL) methods with RoBERTa (Liu et al., 2019b), where we evaluate several fine-tuning methods. Our contributions are twofold: (1) we show that ICL can effectively predict annotator-specific annotations (perspectivist annotations), and that aggregating these predictions into soft labels yields competitive performance; and (2) we argue that LDL methods are promising for soft label predictions and merit further exploration by the perspectivist community.

Figures

Figures reproduced from arXiv: 2509.09524 by Anh Dang, Daniil Ignatev, Hugh Mee Wong, Nan Li, Shane Kaszefski Yaschuk.

Figure 1
Figure 1. Figure 1: Our two-step pipeline to solve both tasks, [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Our ICL prompt template. The template sup [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: A sample from the MP development set. The majority of the annotators marked this example as ironic. the three other datasets, as annotators often demon￾strate vastly different annotation behaviors. These examples possibly show that complete pluralistic alignment of language models may be impossible to fully achieve (at least in the linguistic domain), as the model adhering to common sense in all ex￾amples … view at source ↗
Figure 4
Figure 4. Figure 4: ICL prompt for entry CSC-test-2143 and Annotator Ann743 (excerpt). The in-context examples are selected from Ann743’s annotations in the train set, following the stratified label-based sampling method. 11 [PITH_FULL_IMAGE:figures/full_fig_p011_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

34 extracted references · 1 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    OpenAI Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haim ing Bao, Mo Bavarian, Jeff Belgum, Irwan Bello, and 260 others. 2023. https://api.semanticscholar.org/CorpusID:25...

    2023

  4. [4]

    Anthropic. 2024. https://anthropic.com/claude-3-model-card The claude 3 model family: Opus, sonnet, haiku

    2024

  5. [5]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

    2020

  6. [6]

    Federico Cabitza, Andrea Campagner, and Valerio Basile. 2023. https://doi.org/10.1609/aaai.v37i6.25840 Toward a perspectivist turn in ground truthing for predictive computing . Proceedings of the AAAI Conference on Artificial Intelligence, 37(6):6860--6868

    doi:10.1609/aaai.v37i6.25840 2023

  7. [7]

    Silvia Casola, Simona Frenda, Soda Marem Lo, Erhan Sezerer, Antonio Uva, Valerio Basile, Cristina Bosco, Alessandro Pedrani, Chiara Rubagotti, Viviana Patti, and Davide Bernardi. 2024. https://doi.org/10.18653/v1/2024.acl-long.849 M ulti PIC o: Multilingual perspectivist irony corpus . In Proceedings of the 62nd Annual Meeting of the Association for Compu...

    doi:10.18653/v1/2024.acl-long.849 2024

  8. [8]

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, and 1 others. 2024. A survey on in-context learning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1107--1128

    2024

  9. [9]

    Sujan Dutta, Deepak Pandita, Tharindu Cyril Weerasooriya, Marcos Zampieri, Christopher M Homan, and Ashiqur R KhudaBukhsh. 2025. Annotator reliability through in-context learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 14230--14237

    2025

  10. [10]

    Xin Geng and Rongzi Ji. 2013. https://doi.org/10.1109/ICDMW.2013.19 Label distribution learning . In 2013 IEEE 13th International Conference on Data Mining Workshops, pages 377--383

    work page doi:10.1109/icdmw.2013.19 2013

  11. [11]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    Pith/arXiv arXiv 2024

  12. [12]

    Tiancheng Hu and Nigel Collier. 2024. https://doi.org/10.18653/v1/2024.acl-long.554 Quantifying the persona effect in LLM simulations . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10289--10307, Bangkok, Thailand. Association for Computational Linguistics

    doi:10.18653/v1/2024.acl-long.554 2024

  13. [13]

    Hyewon Jang and Diego Frassinelli. 2024. https://doi.org/10.18653/v1/2024.naacl-long.238 Generalizable sarcasm detection is just around the corner, of course! In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 4238--4249, Mexico Ci...

    doi:10.18653/v1/2024.naacl-long.238 2024

  14. [14]

    Nan-Jiang Jiang and Marie-Catherine de Marneffe. 2022. https://doi.org/10.1162/tacl_a_00523 Investigating reasons for disagreement in natural language inference . Transactions of the Association for Computational Linguistics, 10:1357--1374

    doi:10.1162/tacl_a_00523 2022

  15. [15]

    Nan-Jiang Jiang, Chenhao Tan, and Marie-Catherine de Marneffe. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.712 Ecologically valid explanations for label variation in NLI . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10622--10633, Singapore. Association for Computational Linguistics

    doi:10.18653/v1/2023.findings-emnlp.712 2023

  16. [16]

    Gauri Kambhatla, Sanjana Gautam, Angela Zhang, Alex Liu, Ravi Srinivasan, Junyi Jessy Li, and Matthew Lease. 2025. Beyond sociodemographic prompting: Using supervision to align llms with human response distributions. arXiv preprint arXiv:2507.00439

    Pith/arXiv arXiv 2025

  17. [17]

    Marvin Kaster, Wei Zhao, and Steffen Eger. 2021. Global explainability of bert-based evaluation metrics by disentangling along linguistic factors. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8912--8925

    2021

  18. [18]

    Elisa Leonardelli, Silvia Casola, Siyao Peng, Giulia Rizzi, Valerio Basile, Elisabetta Fersini, Diego Frassinelli, Hyewon Jang, Maja Pavlovic, Barbara Plank, and Massimo Poesio. 2025. Lewidi-2025 at nlperspectives: third edition of the learning with disagreements shared task. In Proceedings of the 4th Workshop on Perspectivist Approaches to NLP (NLPerspec...

    2025

  19. [19]

    Tong Liu, Akash Venkatachalam, Pratik Sanjay Bongale, and Christopher Homan. 2019 a . https://doi.org/10.1145/3308560.3317082 Learning to predict population-level label distributions . In Companion Proceedings of The 2019 World Wide Web Conference, WWW '19, page 1111–1120, New York, NY, USA. Association for Computing Machinery

    arXiv 2019

  20. [20]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019 b . Roberta: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692

    Pith/arXiv arXiv 2019

  21. [21]

    Nicole Meister, Carlos Guestrin, and Tatsunori B Hashimoto. 2025. Benchmarking distributional alignment of large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 24--49

    2025

  22. [22]

    Jingwei Ni, Yu Fan, Vilém Zouhar, Donya Rooein, Alexander Hoyle, Mrinmaya Sachan, Markus Leippold, Dirk Hovy, and Elliott Ash. 2025. https://arxiv.org/abs/2506.19467 Can reasoning help large language models capture human annotator disagreement? Preprint, arXiv:2506.19467

    arXiv 2025

  23. [23]

    Yixin Nie, Xiang Zhou, and Mohit Bansal. 2020. What can we learn from collective human opinions on natural language inference data? arXiv preprint arXiv:2010.03532

    Pith/arXiv arXiv 2020

  24. [24]

    Pedregosa, G

    F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in P ython. Journal of Machine Learning Research, 12:2825--2830

    2011

  25. [25]

    Barbara Plank. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.731 The ``problem'' of human label variation: On ground truth in data, modeling and evaluation . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10671--10682, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics

    doi:10.18653/v1/2022.emnlp-main.731 2022

  26. [26]

    Filip Radlinski, Krisztian Balog, Fernando Diaz, Lucas Dixon, and Ben Wedin. 2022. https://doi.org/10.1145/3477495.3531873 On natural language user profiles for transparent and scrutable recommendation . In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '22, page 2863–2874, New York, ...

    arXiv 2022

  27. [27]

    Rahmani, Xi Wang, Xiao Fu, and Aldo Lipani

    Jerome Ramos, Hossein A. Rahmani, Xi Wang, Xiao Fu, and Aldo Lipani. 2024. https://doi.org/10.18653/v1/2024.acl-long.753 Transparent and scrutable recommendations using natural language user profiles . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13971--13984, Bangkok, Thailand. ...

    doi:10.18653/v1/2024.acl-long.753 2024

  28. [28]

    Nils Reimers and Iryna Gurevych. 2020. https://arxiv.org/abs/2004.09813 Making monolingual sentence embeddings multilingual using knowledge distillation . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics

    Pith/arXiv arXiv 2020

  29. [29]

    Taylor Sorensen, Pushkar Mishra, Roma Patel, Michael Henry Tessler, Michiel Bakker, Georgina Evans, Iason Gabriel, Noah Goodman, and Verena Rieser. 2025. Value profiles for encoding human variation. arXiv preprint arXiv:2503.15484

    arXiv 2025

  30. [30]

    Huaman Sun, Jiaxin Pei, Minje Choi, and David Jurgens. 2025. https://doi.org/10.18653/v1/2025.naacl-short.71 Sociodemographic prompting is not yet an effective approach for simulating subjective judgments with LLM s . In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Langua...

    doi:10.18653/v1/2025.naacl-short.71 2025

  31. [31]

    Leon Weber-Genzel, Siyao Peng, Marie-Catherine De Marneffe, and Barbara Plank. 2024. https://doi.org/10.18653/v1/2024.acl-long.123 V ari E rr NLI : Separating annotation error from human label variation . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2256--2269, Bangkok, Thailand....

    doi:10.18653/v1/2024.acl-long.123 2024

  32. [32]

    Changsong Wen, Xin Zhang, Xingxu Yao, and Jufeng Yang. 2023. https://doi.org/10.1109/ICCV51070.2023.02146 Ordinal label distribution learning . In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 23424--23434

    arXiv 2023

  33. [33]

    Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. https://doi.org/10.18653/v1/N18-1101 A broad-coverage challenge corpus for sentence understanding through inference . In Proceedings of the 2018 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) , pages 1112...

    doi:10.18653/v1/n18-1101 2018

  34. [34]

    Xi Ye and Greg Durrett. 2022. The unreliability of explanations in few-shot prompting for textual reasoning. Advances in neural information processing systems, 35:30378--30392

    2022

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.