Pith. sign in

REVIEW 3 major objections 90 references

PERSONAJUDGE: Simulating Individual Human Preference Judgments with Evaluator-Specific Demonstration Data

T0 review · 3 major / 0 minor · reviewed 2026-07-11 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Evaluator-specific reasoning traces let LLMs simulate individual preference judges better than consensus-only baselines.

desk verdict Careful factorial study showing that evaluator-specific ICL (especially retrospective reasoning) can beat a same-model base judge by up to ~10 points, with honest negative telemetry results and systematic predictors of who is hard to simulate. read the letter →

arxiv 2607.05742 v1 pith:SFAGBFGO submitted 2026-07-07 cs.HC

Zeyu He , Xuan Qi , Subramanian Chidambaram , Zhichao Xu , Vinayak Arannil , Lydia Chilton , Alex C. Williams This is my paper · ORCID
classification cs.HC
keywords LLM-as-judgeindividualpreferencesimulationin-contextlearningretrospectivereasoninginterfacetelemetryevaluatorvariationpersonalizedevaluationHelpfulandHarmless
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Most LLM-as-judge pipelines learn crowd consensus and erase how different people actually decide. PERSONAJUDGE instead feeds an LLM each evaluator's own past labels plus optional process data—retrospective reasoning and interface telemetry—and asks the model to simulate that person's next three-way judgment (prefer A, neutral, prefer B). Across 32 trained annotators and 4,200 judgments on helpfulness and harmlessness tasks, the best configuration improves accuracy by as much as 9.9 points over the same model with no personal demonstrations. Retrospective reasoning supplies the largest lift; raw interface telemetry often hurts. Simulation difficulty is systematic: people who use Neutral a lot or diverge from consensus are harder to mimic, and the Neutral-usage tendency itself is stable across tasks even when simulatability is not. The work shows that individual-aware evaluation is feasible, but only when the right kind of process data is collected.

What carries the argument

PERSONAJUDGE: a two-round in-context learning cascade that first predicts whether the target evaluator will express any preference, then (if needed) predicts its direction, using demonstrations that can include labels, interface telemetry, and retrospective reasoning traces.

What would settle it

A delayed re-test of the same evaluators on held-out items, or a side-by-side comparison of concurrent versus retrospective verbal reports, that shows the reasoning traces fail to improve simulation once rationalization is controlled for.

Watch

Extended reading notes

Core claim

Conditioning an LLM on an evaluator's own multi-facet demonstrations—especially categorical judgments paired with retrospective reasoning—raises three-class simulation accuracy over a zero-shot Base Judge by up to 9.9 percentage points, and the gain is personalization rather than generic demonstration benefit.

Load-bearing premise

The post-hoc think-alouds collected after replaying each judgment are assumed to be faithful enough accounts of the original decision criteria rather than after-the-fact rationalizations.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 0 minor

Summary. The paper proposes PERSONAJUDGE, an ICL framework that simulates an individual evaluator’s three-class preference judgment (Prefer A / Neutral / Prefer B) by conditioning an LLM on that evaluator’s prior categorical labels plus optional interface telemetry and retrospective reasoning traces. Using a 4×4×4 factorial design over 32 trained annotators and 4,200 HH-style judgments (helpfulness and harmlessness), the authors report that evaluator-specific demonstrations improve three-class accuracy over the same model’s zero-shot Base Judge by up to 9.9 pp (Claude-3.5-Sonnet, 8-shot J+RR on Harmlessness), that retrospective reasoning is the most useful complementary signal while event-level telemetry often hurts, and that simulation difficulty is systematic—predicted by neutral usage and divergence from consensus—with neutral usage a stable cross-task trait (r=0.728). Controls include a cross-evaluator demonstration control, oracle/majority baselines, and a deviation-item analysis showing modest but genuine individual capture.

Significance. If the results hold, the work supplies a concrete, carefully controlled methodology for moving LLM-as-Judge pipelines from consensus simulation toward individual-aware evaluation. The multi-facet data collection protocol, two-round cascade, factorial design, cross-evaluator personalization control, and deviation-item analysis are reusable contributions for the field. The honest reporting of modest absolute accuracy (near per-evaluator majority-class), the negative telemetry effect, and the cost–benefit asymmetry between reasoning and telemetry are themselves useful methodological findings for scaling personalized assessment and for reward modeling under heterogeneous preferences. Strengths include transparent baselines, non-parametric significance testing with multiple-comparison correction, and explicit limitations on post-hoc reasoning fidelity.

major comments (3)
  1. §5.1.1–5.1.2 and Table 2: the headline “up to 9.9 pp” gain is configuration-specific (Claude-3.5, 8-shot J+RR on Harmlessness). After FDR correction over the 64 conditions (Appendix H.4), only 3 Harmlessness and 0 Helpfulness configurations remain significant; the recommended configuration is significant only as a planned comparison. The abstract and main claims should state more clearly that average gains are small (+1.4 / +2.8 pp) and that most of the 64 cells do not survive family-wise correction, so that readers do not over-generalize the peak number.
  2. §5.1.3 and Appendix I: PERSONAJUDGE does not significantly exceed the per-evaluator majority-class baseline (∆ = −0.019, p=0.95 Harmlessness; +0.042, p=0.14 Helpfulness). The deviation-item analysis shows genuine individual capture (accuracy ~0.36 on items where consensus predictors score 0 by construction), but the absolute individual signal remains modest. The paper’s framing of “individual evaluator simulation” should more explicitly position the method as a complement to, rather than a replacement for, simple per-person predictors, and discuss what additional signal would be needed to clear that bar.
  3. Limitations and §3.2.2 / Stage-2 protocol: the largest gains rest on retrospective think-alouds cued by interaction replay. The authors correctly note possible rationalization (Ericsson & Simon), but provide no quantitative check (e.g., inter-rater agreement on criteria extracted from traces, or correlation of trace content with Stage-1 dwell/revisit patterns). A short validation or sensitivity analysis would strengthen the claim that J+RR gains reflect decision criteria rather than post-hoc narrative.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: purely empirical held-out accuracy comparisons with no derivation that reduces to its inputs by construction.

full rationale

PERSONAJUDGE is an empirical simulation study. Its load-bearing claims are three-class accuracy of ICL simulations against each evaluator's own held-out validation labels (disjoint 40-item demo pool / 60-item validation split), compared to a zero-shot Base Judge, a cross-evaluator demonstration control, majority-class baselines, and deviation-item analyses. There is no first-principles derivation, no fitted scalar renamed as a prediction, no uniqueness theorem, and no ansatz whose only support is a self-citation. Self-citations (e.g., PILOT, MetaSynth, ADEQA) appear only as related-work context and do not force the reported accuracy deltas. The 9.9 pp gain, the J+RR superiority, the telemetry penalty, and the neutral-usage / consensus-divergence predictors are all measured quantities, not identities. Score 0 is therefore the correct outcome.

Assumptions & free parameters 2 free parameters · 4 assumptions · 1 invented entities

Empirical HCI/ML paper. Load-bearing premises are standard experimental assumptions plus the operationalization of 'individual simulation' via ICL accuracy on three-class labels. No free physical constants; experimental factors (shot counts, models, demo types) are controlled variables, not fitted parameters that define the claim. The main invented construct is the PERSONAJUDGE multi-facet demonstration package itself.

free parameters (2)
  • demonstration count k (1/2/4/8)
    Chosen experimental levels; results plateau after 4, so the recommended 8-shot config is a design choice rather than a data-fitted constant, but the reported peak gain depends on this choice.
  • event-level telemetry serialization schema
    Authors encode raw GUI events into a fixed typed sequence (Table 4). Alternative aggregations (dwell totals, re-read counts) are not tested; the negative IT finding is therefore schema-dependent.
assumptions (4)
  • domain assumption In-context learning with a small number of evaluator-specific demonstrations can transfer that evaluator's judgment style to new instances.
    Core methodological premise of §3; standard ICL literature is cited but success on subjective three-class preference is assumed testable rather than proved.
  • domain assumption Post-hoc think-alouds cued by interaction replay are usable proxies for the criteria that produced the original categorical judgment.
    Stage-2 collection protocol (Appendix A); Limitations explicitly notes possible rationalization.
  • ad hoc to paper Three-class accuracy against the same evaluator's held-out labels is the right primary metric of individual simulation fidelity.
    Chosen over agreement with consensus or binary collapse; justified in §4.5 but remains a design choice.
  • domain assumption Trained professional annotators on HH pairwise items are a sufficient testbed for claims about individual-aware AI assessment.
    Study population and task scope (§4.1, Limitations).
invented entities (1)
  • PERSONAJUDGE multi-facet demonstration (J / J+IT / J+RR / J+IT+RR) independent evidence
    purpose: Package categorical judgment with optional interface telemetry and retrospective reasoning for ICL simulation of a target evaluator.
    The framework is the paper's central construct; independent evidence is the empirical accuracy gains reported against base and cross-evaluator controls.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PERSONAJUDGE: Simulating Individual Human Preference Judgments with Evaluator-Specific Demonstration Data." pith.science (2026). https://pith.science/paper/SFAGBFGO

@misc{pith2026260705742,
  author       = {Pith},
  title        = {Pith review of: PERSONAJUDGE: Simulating Individual Human Preference Judgments with Evaluator-Specific Demonstration Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SFAGBFGO}},
  note         = {Machine review of arXiv:2607.05742}
}
read the original abstract

Large language models increasingly serve as judges in AI evaluation, but current approaches rely on consensus preferences that ignore individual evaluator variation. We propose a novel simulation approach that combines categorical judgments with evaluator-specific auxiliary data--retrospective reasoning traces and interface telemetry--to enable LLM-based simulation of individual evaluators via in-context learning. We conduct a systematic empirical study of this approach using multi-facet data from 32 trained annotators across 4,200 preference judgments in a 4 x 4 x 4 factorial design. Our key findings: (1) The simulation approach achieves up to 9.9 percentage point improvements over the Base Judge; (2) Reasoning traces provide the largest gains with higher collection efforts, while interface telemetry often hurts rather than helps performance despite being cheaper to collect. (3) Simulation difficulty is systematic, predicted by an evaluator's neutral usage (most clearly on Helpfulness) and divergence from consensus; the neutral-usage tendency--rather than simulatability itself--is the cross-task-stable property (r = 0.728). These results establish both the potential and limits of evaluator-specific auxiliary data for personalized evaluation, offering methodological insights for scaling individual aware AI assessment.

Figures

Figures reproduced from arXiv: 2607.05742 by the authors.

Figure 1
Figure 1. PERSONAJUDGE workflow. A human eval￾uator first completes judgment tasks, producing three complementary signals: a categorical judgment, inter￾face telemetry, and retrospective reasoning. PERSONA￾JUDGE organizes these signals into evaluator-specific demonstrations and combines them with the task instruc￾tion and a new task instance. An LLM then uses this information to simulate the target evaluator’s judgment. prefe… view at source ↗
Figure 2
Figure 2. Accuracy comparison across random selec [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Evaluators’ Neutral-selection rates are strongly correlated across tasks (r = 0.728, p < 0.05), whereas simulation fidelity itself is not (r = 0.181). Each point is one of the 10 evaluators participating in both tasks; Neutral rates are computed on each evalua￾tor’s 60 validation items with a human majority label, so they can differ slightly from the full-data rates in [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Stage 1 interface for preference judgment collection, showing the conversation context, two AI responses, [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 5
Figure 5. Figure 5: Stage 2 interface for retrospective reasoning, showing an interaction replay and structured think-aloud [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: Retrospective Reasoning Format ICL Demonstration Format [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 8
Figure 8. Figure 8: Evaluator judgment distribution for the Harm [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

90 extracted references · 90 canonical work pages

  1. [1]

    Constitutional AI: Harmlessness from AI Feedback

    Bai, Yuntao and Kadavath, Saurav and Kundu, Sandipan and Askell, Amanda and Kernion, Jackson and Jones, Andy and Chen, Anna and Goldie, Anna and Mirhoseini, Azalia and McKinnon, Cameron and Chen, Carol and Olsson, Catherine and Olah, Christopher and Hernandez, Danny and Drain, Dawn and Ganguli, Deep and Li, Dustin and Tran-Johnson, Eli and Perez, Ethan an...

    work page Pith review arXiv

  2. [2]

    If in a crowdsourced data annotation pipeline, a

    He, Zeyu and Huang, Chieh-Yang and Ding, Chien-Kuang Cornelia and Rohatgi, Shaurya and Huang, Ting-Hao Kenneth , booktitle=. If in a crowdsourced data annotation pipeline, a

    work page

  3. [3]

    Zheng, Lianmin and Chiang, Wei-Lin and Sheng, Ying and Zhuang, Siyuan and Wu, Zhanghao and Zhuang, Yonghao and Lin, Zi and Li, Zhuohan and Li, Dacheng and Xing, Eric and Zhang, Hao and Gonzalez, Joseph and Stoica, Ion , journal=. Judging

    work page

  4. [4]

    2006 , publisher=

    Cognitive apprenticeship , author=. 2006 , publisher=

    work page 2006

  5. [5]

    A short primer on situated cognition , author=

    work page

  6. [6]

    Mental models , pages=

    Some observations on mental models , author=. Mental models , pages=. 2014 , publisher=

    work page 2014

  7. [7]

    Proceedings of the 2008 conference on empirical methods in natural language processing , pages=

    Cheap and fast--but is it good? evaluating non-expert annotations for natural language tasks , author=. Proceedings of the 2008 conference on empirical methods in natural language processing , pages=

    work page 2008

  8. [8]

    Self-Instruct: Aligning Language Models with Self-Generated Instructions

    Self-instruct: Aligning language models with self-generated instructions , author=. arXiv preprint arXiv:2212.10560 , year=

    work page Pith review arXiv

Show all 90 references
  1. [9]

    Advances in Neural Information Processing Systems , volume=

    Alpacafarm: A simulation framework for methods that learn from human feedback , author=. Advances in Neural Information Processing Systems , volume=

  2. [10]

    Proceedings of the 41st International Conference on Machine Learning , articleno =

    Lee, Harrison and Phatale, Samrat and Mansoor, Hassan and Mesnard, Thomas and Ferret, Johan and Lu, Kellie and Bishop, Colton and Hall, Ethan and Carbune, Victor and Rastogi, Abhinav and Prakash, Sushant , title =. Proceedings of the 41st International Conference on Machine Le...

    2024

  3. [11]

    Proceedings of the AAAI Conference on Human Computation and Crowdsourcing , volume=

    Deja vu: Characterizing worker reliability using task consistency , author=. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing , volume=

  4. [12]

    G -Eval: NLG Evaluation using Gpt-4 with Better Human Alignment

    Liu, Yang and Iter, Dan and Xu, Yichong and Wang, Shuohang and Xu, Ruochen and Zhu, Chenguang. G -Eval: NLG Evaluation using Gpt-4 with Better Human Alignment. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023. doi:10.18653/v1/2023.em...

    2023 doi

  5. [13]

    A companion to cognitive science , pages=

    Protocol analysis , author=. A companion to cognitive science , pages=. 2017 , publisher=

    2017

  6. [14]

    Advances in neural information processing systems , volume=

    Learning to summarize with human feedback , author=. Advances in neural information processing systems , volume=

  7. [15]

    2021 , volume=

    Reiichiro Nakano and Jacob Hilton and Suchir Balaji and Jeff Wu and Ouyang Long and Christina Kim and Christopher Hesse and Shantanu Jain and Vineet Kosaraju and William Saunders and Xu Jiang and Karl Cobbe and Tyna Eloundou and Gretchen Krueger and Kevin Button and Matthew Kn...

    2021

  8. [16]

    Understanding Dataset Difficulty with

    Ethayarajh, Kawin and Choi, Yejin and Swayamdipta, Swabha , booktitle =. Understanding Dataset Difficulty with. 2022 , editor =

    2022

  9. [17]

    Behavioral and brain sciences , volume=

    Does the chimpanzee have a theory of mind? , author=. Behavioral and brain sciences , volume=. 1978 , publisher=

    1978

  10. [18]

    theory of mind

    Mindreaders: the cognitive basis of" theory of mind" , author=. 2010 , publisher=

    2010

  11. [19]

    Large Language Models are not Fair Evaluators

    Wang, Peiyi and Li, Lei and Chen, Liang and Cai, Zefan and Zhu, Dawei and Lin, Binghuai and Cao, Yunbo and Kong, Lingpeng and Liu, Qi and Liu, Tianyu and Sui, Zhifang. Large Language Models are not Fair Evaluators. Proceedings of the 62nd Annual Meeting of the Association for ...

    2024 doi

  12. [20]

    Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , year=

    Towards a Unified Multi-Dimensional Evaluator for Text Generation , author=. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , year=

    2022

  13. [21]

    Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , year=

    FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation , author=. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , year=

    2023

  14. [22]

    Advances in Neural Information Processing Systems , volume=

    Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Models , author=. Advances in Neural Information Processing Systems , volume=

  15. [23]

    Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics , pages=

    NLPositionality: Characterizing Design Biases of Datasets and Models , author=. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics , pages=

  16. [24]

    Justice or Prejudice? Quantifying Biases in

    Jiayi Ye and Yanbo Wang and Yue Huang and Dongping Chen and Qihui Zhang and Nuno Moniz and Tian Gao and Werner Geyer and Chao Huang and Pin-Yu Chen and Nitesh V Chawla and Xiangliang Zhang , booktitle=. Justice or Prejudice? Quantifying Biases in. 2024 , url=

    2024

  17. [25]

    Judging the judges: A systematic study of position bias in

    Shi, Lin and Ma, Chiyu and Liang, Wenhua and Diao, Xingjian and Ma, Weicheng and Vosoughi, Soroush , booktitle=. Judging the judges: A systematic study of position bias in

  18. [26]

    Rank analysis of incomplete block designs:

    Bradley, Ralph Allan and Terry, Milton E , journal=. Rank analysis of incomplete block designs:. 1952 , publisher=

    1952

  19. [27]

    Advances in neural information processing systems , volume=

    Direct preference optimization: Your language model is secretly a reward model , author=. Advances in neural information processing systems , volume=

  20. [28]

    Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    Orpo: Monolithic preference optimization without reference model , author=. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    2024

  21. [29]

    Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology , pages=

    Creating general user models from computer use , author=. Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology , pages=

  22. [30]

    2506.05606 , archivePrefix=

    Ziyi Wang and Yuxuan Lu and Wenbo Li and Amirali Amini and Bo Sun and Yakov Bart and Weimin Lyu and Jiri Gesi and Tian Wang and Jing Huang and Yu Su and Upol Ehsan and Malihe Alikhani and Toby Jia-Jun Li and Lydia Chilton and Dakuo Wang , year=. 2506.05606 , archivePrefix=

    arXiv

  23. [31]

    1993 , publisher=

    The Adaptive Decision Maker , author=. 1993 , publisher=

    1993

  24. [32]

    Proceedings of the 12th International Conference on Intelligent User Interfaces , year=

    Toward Harnessing User Feedback for Machine Learning , author=. Proceedings of the 12th International Conference on Intelligent User Interfaces , year=

  25. [33]

    Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks , year=

    Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processing , author=. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks , year=

  26. [34]

    Advances in Neural Information Processing Systems , year=

    Training Language Models to Follow Instructions with Human Feedback , author=. Advances in Neural Information Processing Systems , year=

  27. [35]

    arXiv preprint arXiv:2204.05862 , year=

    Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback , author=. arXiv preprint arXiv:2204.05862 , year=

    arXiv

  28. [36]

    Organizational Behavior and Human Decision Processes , year=

    Process Tracing Methods: Contributions, Problems, and Neglected Research Questions , author=. Organizational Behavior and Human Decision Processes , year=

  29. [37]

    Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval , year=

    Ready to Buy or Just Browsing? Detecting Web Searcher Goals from Interaction Data , author=. Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval , year=

  30. [38]

    Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval , year=

    Understanding Web Browsing Behaviors through Weibull Analysis of Dwell Time , author=. Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval , year=

  31. [39]

    2011 , publisher=

    Eye Tracking: A Comprehensive Guide to Methods and Measures , author=. 2011 , publisher=

    2011

  32. [40]

    Cognitive Psychology , year=

    Eye Fixations and Cognitive Processes , author=. Cognitive Psychology , year=

  33. [41]

    1993 , publisher=

    Protocol Analysis: Verbal Reports as Data , author=. 1993 , publisher=

    1993

  34. [42]

    Knowledge Based Systems , year=

    The think aloud method: a practical approach to modelling cognitive processes , author=. Knowledge Based Systems , year=

  35. [43]

    Proceedings of the IEEE Symposium on Visual Languages and Human-Centric Computing , year=

    Too Much, Too Little, or Just Right? Ways Explanations Impact End Users' Mental Models , author=. Proceedings of the IEEE Symposium on Visual Languages and Human-Centric Computing , year=

  36. [44]

    Proceedings of the SIGCHI Conference on Human Factors in Computing Systems , year=

    Why and Why Not Explanations Improve the Intelligibility of Context-Aware Intelligent Systems , author=. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems , year=

  37. [45]

    Cognitive Science , year=

    Self-Explanations: How Students Study and Use Examples in Learning to Solve Problems , author=. Cognitive Science , year=

  38. [46]

    Who validates the validators?

    Shankar, Shreya and Zamfirescu-Pereira, JD and Hartmann, Bj. Who validates the validators?. Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology , pages=

  39. [47]

    and Hajishirzi, Hannaneh

    Lambert, Nathan and Pyatkin, Valentina and Morrison, Jacob and Miranda, LJ and Lin, Bill Yuchen and Chandu, Khyathi and Dziri, Nouha and Kumar, Sachin and Zick, Tom and Choi, Yejin and Smith, Noah A. and Hajishirzi, Hannaneh. R eward B ench: Evaluating Reward Models for Langua...

    2025 doi

  40. [48]

    and Le, Quoc V

    Wei, Jason and Wang, Xuezhi and Schuurmans, Dale and Bosma, Maarten and Ichter, Brian and Xia, Fei and Chi, Ed H. and Le, Quoc V. and Zhou, Denny , title =. Proceedings of the 36th International Conference on Neural Information Processing Systems , articleno =. 2022 , isbn =

    2022

  41. [49]

    Advances in Neural Information Processing Systems , volume=

    Large Language Models are Zero-Shot Reasoners , author=. Advances in Neural Information Processing Systems , volume=

  42. [50]

    Advances in Neural Information Processing Systems , volume=

    Language Models are Few-Shot Learners , author=. Advances in Neural Information Processing Systems , volume=

  43. [51]

    Information Processing & Management , volume=

    A Systematic Analysis of Performance Measures for Classification Tasks , author=. Information Processing & Management , volume=

  44. [52]

    arXiv preprint arXiv:2010.16061 , year=

    Evaluation: From Precision, Recall and F-measure to ROC, Informedness, Markedness and Correlation , author=. arXiv preprint arXiv:2010.16061 , year=

    2010 arXiv

  45. [53]

    Journal of Memory and Language , volume=

    Mixed-effects Modeling with Crossed Random Effects for Subjects and Items , author=. Journal of Memory and Language , volume=

  46. [54]

    Journal of the American Statistical Association , volume=

    Multiple Comparisons among Means , author=. Journal of the American Statistical Association , volume=

  47. [55]

    1988 , publisher=

    Statistical Power Analysis for the Behavioral Sciences , author=. 1988 , publisher=

    1988

  48. [56]

    Proceedings of the 14th International Joint Conference on Artificial Intelligence , volume=

    A Study of Cross-validation and Bootstrap for Accuracy Estimation and Model Selection , author=. Proceedings of the 14th International Joint Conference on Artificial Intelligence , volume=

  49. [57]

    A Survey on In-context Learning

    Dong, Qingxiu and Li, Lei and Dai, Damai and Zheng, Ce and Ma, Jingyuan and Li, Rui and Xia, Heming and Xu, Jingjing and Wu, Zhiyong and Chang, Baobao and Sun, Xu and Li, Lei and Sui, Zhifang. A Survey on In-context Learning. Proceedings of the 2024 Conference on Empirical Met...

    2024 doi

  50. [58]

    arXiv preprint arXiv:2504.04332 , year=

    IMPersona: Evaluating Individual Level LM Impersonation , author=. arXiv preprint arXiv:2504.04332 , year=

    arXiv

  51. [59]

    arXiv preprint arXiv:2510.22954 , year=

    Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond) , author=. arXiv preprint arXiv:2510.22954 , year=

  52. [60]

    Rewarded soups: towards

    Rame, Alexandre and Couairon, Guillaume and Dancette, Corentin and Gaya, Jean-Baptiste and Shukor, Mustafa and Soulier, Laure and Cord, Matthieu , journal=. Rewarded soups: towards

  53. [61]

    2024 , editor =

    Chakraborty, Souradip and Qiu, Jiahao and Yuan, Hui and Koppel, Alec and Manocha, Dinesh and Huang, Furong and Bedi, Amrit and Wang, Mengdi , booktitle =. 2024 , editor =

    2024

  54. [62]

    Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages=

    Helpsteer: Multi-attribute helpfulness dataset for steerlm , author=. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages=

    2024

  55. [63]

    Can LLM be a Personalized Judge?

    Dong, Yijiang River and Hu, Tiancheng and Collier, Nigel. Can LLM be a Personalized Judge?. Findings of the Association for Computational Linguistics: EMNLP 2024. 2024. doi:10.18653/v1/2024.findings-emnlp.592

    2024 doi

  56. [64]

    Chen, Lingjiao and Zaharia, Matei and Zou, James , journal=. How is. 2024 , publisher=

    2024

  57. [65]

    Computer Speech & Language , volume=

    Human evaluation of automatically generated text: Current trends and best practice guidelines , author=. Computer Speech & Language , volume=. 2021 , publisher=

    2021

  58. [66]

    Transactions of the Association for Computational Linguistics , volume=

    Dealing with disagreements: Looking beyond the majority vote in subjective annotations , author=. Transactions of the Association for Computational Linguistics , volume=. 2022 , publisher=

    2022

  59. [67]

    Proceedings of the 42nd International Conference on Machine Learning , pages =

    Diverging Preferences: When do Annotators Disagree and do Models Know? , author =. Proceedings of the 42nd International Conference on Machine Learning , pages =. 2025 , editor =

    2025

  60. [68]

    Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    When the majority is wrong: Modeling annotator disagreement for subjective tasks , author=. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    2023

  61. [69]

    We Need to Consider Disagreement in Evaluation

    Basile, Valerio and Fell, Michael and Fornaciari, Tommaso and Hovy, Dirk and Paun, Silviu and Plank, Barbara and Poesio, Massimo and Uma, Alexandra. We Need to Consider Disagreement in Evaluation. Proceedings of the 1st Workshop on Benchmarking: Past, Present and Future. 2021....

    2021 doi

  62. [70]

    Language Resources and Evaluation , volume=

    Perspectivist approaches to natural language processing: a survey , author=. Language Resources and Evaluation , volume=. 2025 , publisher=

    2025

  63. [71]

    Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection

    Sap, Maarten and Swayamdipta, Swabha and Vianna, Laura and Zhou, Xuhui and Choi, Yejin and Smith, Noah A. Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection. Proceedings of the 2022 Conference of the North American Chapter of the Asso...

    2022 doi

  64. [72]

    Can Language Models Reason about Individualistic Human Values and Preferences?

    Jiang, Liwei and Sorensen, Taylor and Levine, Sydney and Choi, Yejin. Can Language Models Reason about Individualistic Human Values and Preferences?. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025. doi:10.1...

    2025 doi

  65. [73]

    Annotation alignment: Comparing

    Movva, Rajiv and Koh, Pang Wei and Pierson, Emma , booktitle=. Annotation alignment: Comparing

  66. [74]

    Learning Personalized Alignment for Evaluating Open-ended Text Generation

    Wang, Danqing and Yang, Kevin and Zhu, Hanlin and Yang, Xiaomeng and Cohen, Andrew and Li, Lei and Tian, Yuandong. Learning Personalized Alignment for Evaluating Open-ended Text Generation. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing....

    2024 doi

  67. [75]

    Judgment and Decision making , volume=

    Applying the decision moving window to risky choice: Comparison of eye-tracking and mouse-tracing methods , author=. Judgment and Decision making , volume=. 2011 , publisher=

    2011

  68. [76]

    2002 , publisher=

    Jasper, JD and Shapiro, Jennifer , journal=. 2002 , publisher=

    2002

  69. [77]

    Cognitive psychology , volume=

    Eye fixations and cognitive processes , author=. Cognitive psychology , volume=. 1976 , publisher=

    1976

  70. [78]

    Judgment and Decision making , volume=

    The role of process data in the development and testing of process models of judgment and decision making , author=. Judgment and Decision making , volume=. 2011 , publisher=

    2011

  71. [79]

    Camburu, Oana-Maria and Rockt. e-. Advances in Neural Information Processing Systems , volume=

  72. [80]

    Advances in Neural Information Processing Systems , volume=

    Towards interpretable natural language understanding with explanations as latent variables , author=. Advances in Neural Information Processing Systems , volume=

  73. [81]

    Park, Joon Sung and Zou, Carolyn Q and Kamphorst, Jonne and Egan, Niles and Shaw, Aaron and Hill, Benjamin Mako and Cai, Carrie and Morris, Meredith Ringel and Liang, Percy and Willer, Robb and others , journal=

  74. [82]

    Hu, Tiancheng and Collier, Nigel , booktitle=

  75. [83]

    arXiv preprint arXiv:2511.21912 , year=

    Tracing How Annotators Think: Augmenting Preference Judgments with Reading Processes , author=. arXiv preprint arXiv:2511.21912 , year=

  76. [84]

    A survey on

    Gu, Jiawei and Jiang, Xuhui and Shi, Zhichao and Tan, Hexiang and Zhai, Xuehao and Xu, Chengjin and Li, Wei and Shen, Yinghan and Ma, Shengjie and Liu, Honghao and others , journal=. A survey on. 2026 , publisher=

    2026

  77. [85]

    Li, Haitao and Dong, Qian and Chen, Junjie and Su, Huixue and Zhou, Yujia and Ai, Qingyao and Ye, Ziyi and Liu, Yiqun , journal=

  78. [86]

    2025 , url =

    Meyer, Yev and Corneil, Dane , title =. 2025 , url =

    2025

  79. [87]

    M eta S ynth: Meta-Prompting-Driven Agentic Scaffolds for Diverse Synthetic Data Generation

    Riaz, Haris and Bhabesh, Sourav Sanjukta and Arannil, Vinayak and Ballesteros, Miguel and Horwood, Graham. M eta S ynth: Meta-Prompting-Driven Agentic Scaffolds for Diverse Synthetic Data Generation. Findings of the Association for Computational Linguistics: ACL 2025. 2025. do...

    2025 doi

  80. [88]

    2509.15447 , archivePrefix=

    Caitlin Cisar and Emily Sheffield and Joshua Drake and Alden Harrell and Subramanian Chidambaram and Nikita Nangia and Vinayak Arannil and Alex Williams , year=. 2509.15447 , archivePrefix=

  81. [89]

    ADEQA : A Question Answer based approach for joint ADE -Suspect Extraction using Sequence-To-Sequence Transformers

    Arannil, Vinayak and Deb, Tomal and Roy, Atanu. ADEQA : A Question Answer based approach for joint ADE -Suspect Extraction using Sequence-To-Sequence Transformers. Proceedings of the 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks. 2023. doi:10....

    2023 doi

  82. [90]

    Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems , articleno =

    He, Zeyu and Naphade, Saniya and Huang, Ting-Hao Kenneth , title =. Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems , articleno =. 2025 , isbn =. doi:10.1145/3706598.3714319 , abstract =

    2025 doi

Pith tools

Reviewed July 11, 2026 · model on record in the stance chip above.