Pith. sign in

REVIEW 4 major objections 6 minor 44 references

Aligning Black-box Language Models with Human Judgments

T0 review · 4 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single linear remapping of LLM output labels to human labels raises agreement by 142% on average across 29 judgment tasks.

desk verdict Simple per-category label remap, useful as a baseline but oversold; the 142% claim needs a majority-class comparison before it means what the authors say. read the letter →

arxiv 2502.04997 v1 pith:72UO6WMR submitted 2025-02-07 cs.CL cs.AIcs.LG

Gerrit J. J. van den Burg , Gen Suzuki , Wei Liu , Murat Sensoy This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords LLM-as-a-judgehumanalignmentlinearmappingridgeregressionblack-boxcalibrationresponsestylebiaslabelremappingevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that most of the disagreement between an LLM's categorical judgments and human judgments is a systematic label bias, not random noise or a reasoning failure. It proposes learning a single linear mapping from the LLM's output labels to human labels using a small calibration set, then applying that mapping to all future judgments from the same model and task. Across 29 tasks and three black-box models, the authors report that this remapping raises LLM-human agreement by an average of 142 percent, using no model logits, no fine-tuning, and as few as 100 calibration examples. If correct, this means a large part of the 'LLM judge' problem can be fixed by a post-hoc correction rather than by building better judges.

What carries the argument

The load-bearing object is the transformation matrix $W \in \mathbb{R}^{m \times n}$, where $m$ is the number of LLM label categories and $n$ the number of human label categories. It is fit by ridge regression, $\hat{W} = (Z^{\top}Z + \lambda I)^{-1} Z^{\top} Y$, on one-hot encoded judgments from a small training set, and applied at inference by $\varphi(z) = \arg\max_{j} (z^{\top}\hat{W})_j$. This maps each LLM label to the human label it is most associated with, and it works even when the LLM and human label sets differ in size or meaning. The mapping is what carries the argument: because it needs only the LLM's final categorical output, the method works on black-box models with no access to logits or weights.

What would settle it

Select a task with two clearly distinct item types, learn the mapping on items of the first type, and evaluate on items of the second type when both types have the same human label distribution; if agreement after remapping does not beat the unaligned baseline, the input-independent mapping assumption fails.

Watch

Extended reading notes

Core claim

The central discovery is that LLM judges and human judges disagree in a structured, mostly one-directional way: LLMs have their own response styles, such as avoiding negative labels and clustering on highly positive ones, while human annotators spread their labels more evenly. The paper claims that a single input-independent linear transformation $W$, learned by ridge regression from one-hot encodings of LLM labels to one-hot encodings of human labels, corrects this bias and substantially improves agreement with humans on held-out test items. In their experiments this gives a 142% average relative improvement across 29 tasks, exceeds inter-human agreement on four of six multi-annotator tasks, and lets a smaller model such as Mixtral 8x7B Instruct match larger proprietary judges after alignment.

Load-bearing premise

The entire method rests on the premise that the disagreement between a given LLM and the humans it is aligned to is one consistent label bias, so that a single per-task remapping learned from about 100 examples also applies to every other item in that task.

Editorial extensions

If this is right

  • LLM-based evaluation pipelines can become human-aligned by adding a small labeled calibration set and a post-hoc remapping step, with no retraining and no access to model internals.
  • Automated judges that currently disagree with humans because of response style can be used in safety-relevant settings such as medical triage only after this kind of alignment, since raw judgments may be systematically skewed.
  • Smaller, cheaper models can substitute for larger proprietary judges once their labels are remapped, reducing the cost of running evaluations at scale.
  • A mapping learned on one task transfers to related tasks with the same label scale, so calibration effort can be shared across similar evaluation setups.
  • In-context examples of human judgments are not a substitute for the alignment step; the structured remapping adds agreement on top of few-shot prompting.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct testable extension would be to condition the mapping on input features, since tasks where the human meaning of a label changes with content are exactly where the single $W$ should break down.
  • The result suggests that LLM judgments may already contain a reliable ordering of items, and that rank-based agreement metrics could show even stronger human alignment than the label accuracy reported here.
  • If LLM response style is the dominant error source, then judge benchmarks should report both raw and calibrated agreement; reporting only raw agreement may systematically undervalue black-box judges that use skewed label scales.
  • Replacing one-hot labels with soft probability vectors whenever logits are available would likely refine the mapping; the paper notes this possibility but leaves it untested.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a post-hoc alignment method for black-box LLM judges. It learns a linear map W from the one-hot encoded LLM labels to human labels via ridge regression (Eq. 2) on a small calibration set, and at inference time outputs the human label with the largest transformed value (Eq. 3). The method is evaluated on 29 tasks from Judge-Bench plus Feedback-QA, using Claude-3 Sonnet, Mixtral 8x7B, and Llama-3 70B, with 10 random splits. The authors report an average relative improvement of 142% in accuracy over raw LLM labels, show that the approach outperforms in-context learning, and claim that aligned smaller models match larger ones and that the aligned judgments exceed inter-human agreement on four of six multi-annotator tasks.

Significance. The paper offers a simple, logit-free, calibration-set-based approach that can be applied to any LLM without fine-tuning. The mathematical derivation is clear, and the evaluation is repeated with standard deviations. If the gains are real and not a base-rate artifact, the method would be a practically useful post-processing step for deploying LLM judges. The paper's main weakness is that the learned map is essentially a per-category majority remap, and the authors never compare against a majority-class baseline or report LLM label distributions; this omission currently undermines the interpretation of the headline improvement. The paper is valuable as a demonstration of label-bias correction, but the claims need to be scaled to what is actually established.

major comments (4)
  1. [§3.1, Eq. (2), Table 1] The learned mapping in Eq. (2) with λ=10^-6 is, for one-hot encoded Z, a per-category majority remap: each row of W is proportional to the empirical distribution of human labels among training examples with that LLM label, and Eq. (3) selects the modal human label for that LLM label. The paper does not report a majority-class baseline (e.g., predicting the overall training majority label for every test item) or the distribution of LLM labels. On heavily skewed tasks such as SummEval Fluency and Medical Safety Response Type, the aligned accuracies (≈80% and ≈70–80%) are close to what a constant majority predictor would achieve, so the 142% relative improvement over raw LLM accuracy may reflect base-rate correction rather than alignment of the LLM's judgments. The authors should add this baseline, report label distributions, and either temper the alignment claim or explicitly frame the contribution as correcting label bias.
  2. [Abstract, §3.1] The headline '142% average improvement' is a relative gain and is inflated when non-aligned accuracy is near zero, as in the single-digit baselines in Table 1. The paper should also report absolute differences, the distribution of per-task relative improvements, and the proportion of tasks where aligned accuracy exceeds the majority baseline. Reporting the mean of relative improvements across tasks without these context statistics makes the main quantitative claim hard to interpret.
  3. [§5, Table 1] The statement that the aligned method 'exceeds inter-human agreement on four out of six tasks' is not backed by an explicit comparison. The six tasks are not enumerated, and it is unclear whether inter-human agreement is measured as accuracy against individual annotators or against a majority vote, and whether the aligned accuracy is computed on the same examples and with the same metric. The authors should list the tasks, show the inter-human agreement for each, and provide confidence intervals or a significance test before making this claim.
  4. [§3.2, Table 3] The in-context learning comparison reports relative improvements (e.g., 111.74%) without a majority baseline as well. Since the alignment mapping is still a per-category majority remap on top of the ICL outputs, the same base-rate concern applies. Please report the ICL label distributions and a majority predictor on the ICL outputs.
minor comments (6)
  1. [§3.2] 'human judments' is a typo for 'human judgments'.
  2. [Table 1 caption] 'the each row' should be 'each row'.
  3. [Figure 1 caption] 'the same graph' should probably be 'the same data'.
  4. [§2.4] The notation for the j-th component of z^T W_hat is garbled in the rendered text; please use a clear subscript.
  5. [§2.4] The paper does not specify how the method handles LLM labels that never appear in the calibration set; for such rows W would be zero, and the argmax in Eq. (3) would be arbitrary. A short statement on this case would be helpful.
  6. [§4] The related work section does not discuss prior work on label-shift or prior-correction methods; a brief connection would help position the contribution.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the alignment mapping is fit on a training split and evaluated on held-out data, so the reported improvements are empirical rather than definitional.

full rationale

The paper's derivation chain is a standard supervised-learning pipeline. The linear map W in Eq. (2) is estimated by ridge regression from a training set of paired (LLM label, human label) examples, and the aligned judgment in Eq. (3) is applied to held-out test instances. The central claims (142% average improvement, exceeding inter-human agreement on four of six tasks, smaller models matching larger ones) are all measured on test data not used to fit W; no test labels enter the fitting procedure. The one-hot encoding, ridge loss, and argmax decoding are fixed design choices rather than outputs of the fit. The paper contains no self-citations used as load-bearing evidence, and the methodology section does not import any uniqueness or modeling ansatz from prior author work. The absence of a majority-class baseline is a legitimate evaluation concern, but it concerns whether the improvement is attributable to the LLM's signal versus base rates; it is not a circularity, since the reported numbers are genuine held-out measurements rather than quantities forced by construction.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The method introduces no new entities; it relies on standard linear algebra and a supervised training set. The key added assumption is the fixed linear map, which is explicit in the method.

free parameters (2)
  • lambda (regularization) = 1e-6
    Set by hand in all experiments to avoid singular matrix inversion; not tuned.
  • calibration set size = 100 per task (or 25% when dataset < 400)
    Number of labeled examples used to learn the linear mapping; authors chose this as a practical small label budget, and the method's improvement depends on it.
assumptions (3)
  • domain assumption Human judgments are the ground truth and each is represented as a one-hot vector.
    The method optimizes agreement to individual human labels and treats them as correct targets; acknowledged inter-human variability is averaged over annotators.
  • ad hoc to paper A single input-independent linear map W captures the relationship between LLM labels and human labels within a task.
    Eq. (1) learns a single W applied to all instances. If the mapping varies with item content, the method cannot correct it; the limitation is noted in Section 6.
  • domain assumption LLM judgments are categorical and convertible to one-hot vectors.
    Section 2.2 builds on this representation, restricting the method to models that emit discrete labels (no logits used).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Aligning Black-box Language Models with Human Judgments." pith.science (2026). https://pith.science/paper/72UO6WMR

@misc{pith2026250204997,
  author       = {Pith},
  title        = {Pith review of: Aligning Black-box Language Models with Human Judgments},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/72UO6WMR}},
  note         = {Machine review of arXiv:2502.04997}
}
read the original abstract

Large language models (LLMs) are increasingly used as automated judges to evaluate recommendation systems, search engines, and other subjective tasks, where relying on human evaluators can be costly, time-consuming, and unscalable. LLMs offer an efficient solution for continuous, automated evaluation. However, since the systems that are built and improved with these judgments are ultimately designed for human use, it is crucial that LLM judgments align closely with human evaluators to ensure such systems remain human-centered. On the other hand, aligning LLM judgments with human evaluators is challenging due to individual variability and biases in human judgments. We propose a simple yet effective framework to align LLM judgments with individual human evaluators or their aggregated judgments, without retraining or fine-tuning the LLM. Our approach learns a linear mapping between the LLM's outputs and human judgments, achieving over 142% average improvement in agreement across 29 tasks with only a small number of calibration examples used for training. Notably, our method works in zero-shot and few-shot settings, exceeds inter-human agreement on four out of six tasks, and enables smaller LLMs to achieve performance comparable to that of larger models.

Figures

Figures reproduced from arXiv: 2502.04997 by the authors.

Figure 1
Figure 1. Cumulative distribution function of response options for the judgment task of [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Test accuracy for Medical Safety (response type) dataset as we increase the number of training ex￾amples per judgment category. ing to guide the LLM towards better alignment with human interpretations. As shown in [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Example (top) prompt for one of the judgment tasks (medical-safety: response type), where [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

44 extracted references · 19 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Gavin Abercrombie and Verena Rieser. 2022. https://aclanthology.org/2022.aacl-short.30 Risk-graded safety for handling medical queries in conversational AI . In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 2:...

    work page 2022

  4. [4]

    Anthropic. 2024. The Claude 3 model family: Opus , Sonnet , Haiku . https://anthropic.com/claude-3-model-card. Retrieved Oct 2024

    work page 2024

  5. [5]

    Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, André F. T. Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya K Surikuchi, Ece Takmaz, and Alberto Testoni. 2024. http...

    arXiv 2024

  6. [6]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Ma teusz Litwin, ...

    arXiv 2020

  7. [7]

    Oana-Maria Camburu, Tim Rockt\" a schel, Thomas Lukasiewicz, and Phil Blunsom. 2018. https://proceedings.neurips.cc/paper_files/paper/2018/file/4c7a167bb329bd92580a99ce422d6fa6-Paper.pdf e-SNLI : Natural language inference with natural language explanations . In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc

    work page 2018

  8. [8]

    Cheng-Han Chiang and Hung-yi Lee. 2023. https://doi.org/10.18653/v1/2023.acl-long.870 Can large language models be an alternative to human evaluations? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15607--15631, Toronto, Canada. Association for Computational Linguistics

    doi:10.18653/v1/2023.acl-long.870 2023

Show all 44 references
  1. [9]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. https://arxiv.org/abs/2110.14168 Training verifiers to solve math word problems . ...

    2021 arXiv

  2. [10]

    Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. https://doi.org/10.18653/v1/N19-1246 DROP : A reading comprehension benchmark requiring discrete reasoning over paragraphs . In Proceedings of the 2019 Conference of the North A ...

    2019 doi

  3. [11]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. https://arxiv.org/abs/2407.21783 The Llama 3 herd of models . Preprint, ArXiv:2407.21783

    2024 arXiv

  4. [12]

    Clarke, Gianluca Demartini, Matthias Hagen, Claudia Hauff, Noriko Kando, Evangelos Kanoulas, Martin Potthast, Benno Stein, and Henning Wachsmuth

    Guglielmo Faggioli, Laura Dietz, Charles L.A. Clarke, Gianluca Demartini, Matthias Hagen, Claudia Hauff, Noriko Kando, Evangelos Kanoulas, Martin Potthast, Benno Stein, and Henning Wachsmuth. 2023. Perspectives on large language models for relevance judgment. In Proceedings of...

    2023

  5. [13]

    Yu Fei, Yifan Hou, Zeming Chen, and Antoine Bosselut. 2023. https://doi.org/10.18653/v1/2023.acl-long.783 Mitigating label biases for in-context learning . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages...

    2023 doi

  6. [14]

    Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021. https://doi.org/10.1162/tacl_a_00437 Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation . Transactions of the Association for Comp...

    2021 doi

  7. [15]

    Fabrizio Gilardi, Meysam Alizadeh, and Ma \"e l Kubli. 2023. ChatGPT outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences, 120(30):e2305016120

    2023

  8. [16]

    Olga Golovneva, Moya Peng Chen, Spencer Poff, Martin Corredor, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyilmaz. 2023. https://openreview.net/forum?id=xYlJRpzZtsY ROSCOE : A suite of metrics for scoring step-by-step reasoning . In The Eleventh International Confere...

    2023

  9. [17]

    Max Grusky, Mor Naaman, and Yoav Artzi. 2018. https://doi.org/10.18653/v1/N18-1065 N ewsroom: A dataset of 1.3 million summaries with diverse extractive strategies . In Proceedings of the 2018 Conference of the North A merican Chapter of the Association for Computational Lingu...

    2018 doi

  10. [18]

    Rishav Hada, Varun Gumma, Adrian Wynter, Harshita Diddee, Mohamed Ahmed, Monojit Choudhury, Kalika Bali, and Sunayana Sitaram. 2024. https://aclanthology.org/2024.findings-eacl.71 Are large language model-based evaluators the solution to scaling up multilingual evaluation? In ...

    2024

  11. [19]

    Zhixiong Han, Yaru Hao, Li Dong, Yutao Sun, and Furu Wei. 2023. https://openreview.net/forum?id=nUsP9lFADUF Prototypical calibration for few-shot learning of language models . In The Eleventh International Conference on Learning Representations

    2023

  12. [20]

    Arthur E Hoerl and Robert W Kennard. 1970. Ridge regression: Biased estimation for nonorthogonal problems. Technometrics, 12(1):55--67

    1970

  13. [21]

    Fan Huang, Haewoon Kwak, Kunwoo Park, and Jisun An. 2024. https://aclanthology.org/2024.lrec-main.277 C hat GPT rates natural language explanation quality like humans: But on which scales? In Proceedings of the 2024 Joint International Conference on Computational Linguistics, ...

    2024

  14. [22]

    Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. https://doi.org/10.18653/v1/D19-1243 Cosmos QA : Machine reading comprehension with contextual commonsense reasoning . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processin...

    2019 doi

  15. [23]

    Jackson and Samuel Messick

    Douglas N. Jackson and Samuel Messick. 1958. Content and style in personality assessment. Psychological Bulletin, 55(4):243

    1958

  16. [24]

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, L'elio Renard Lavaud, Lucile Saulnier, Marie-Ann...

    2024 arXiv

  17. [25]

    Tom Kocmi and Christian Federmann. 2023. https://aclanthology.org/2023.eamt-1.19 Large language models are state-of-the-art evaluators of translation quality . In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pages 193--203, Tam...

    2023

  18. [26]

    Ryan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park, Zae Myung Kim, and Dongyeop Kang. 2024. https://doi.org/10.18653/v1/2024.findings-acl.29 Benchmarking cognitive biases in large language models as evaluators . In Findings of the Association for Computational Linguistics ACL 2...

    2024 doi

  19. [27]

    Zichao Li, Prakhar Sharma, Xing Han Lu, Jackie CK Cheung, and Siva Reddy. 2022. https://arxiv.org/abs/2204.03025 Using interactive feedback to improve the accuracy and explainability of question answering systems post-deployment . Preprint, arXiv:2204.03025

    2022 arXiv

  20. [28]

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.153 G -eval: NLG evaluation using GPT -4 with better human alignment . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language...

    2023 doi

  21. [29]

    Ben Naismith, Phoebe Mulcaire, and Jill Burstein. 2023. Automated evaluation of written discourse coherence using GPT-4 . In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), pages 394--403

    2023

  22. [30]

    Delroy L. Paulhus. 1991. https://doi.org/10.1016/B978-0-12-590241-0.50006-X Measurement and control of response bias . In John P. Robinson, Phillip R. Shaver, and Lawrence S. Wrightsman, editors, Measures of Personality and Social Psychological Attitudes, pages 17--59. Academic Press

    1991 doi

  23. [31]

    Maja Pavlovic and Massimo Poesio. 2024. https://aclanthology.org/2024.nlperspectives-1.11 The effectiveness of LLM s as annotators: A comparative overview and empirical analysis of direct representation . In Proceedings of the 3rd Workshop on Perspectivist Approaches to NLP (N...

    2024

  24. [32]

    Yuval Reif and Roy Schwartz. 2024. https://doi.org/10.18653/v1/2024.naacl-long.378 Beyond performance: Quantifying and mitigating label bias in LLM s . In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human L...

    2024 doi

  25. [33]

    Schoonees, Michel Van de Velden, and Patrick J

    Pieter C. Schoonees, Michel Van de Velden, and Patrick J. F. Groenen. 2015. https://doi.org/10.1007/s11336-015-9458-9 Constrained dual scaling for detecting response styles in categorical data . Psychometrika, 80:968--994

    2015 doi

  26. [34]

    Lindia Tjuatja, Valerie Chen, Tongshuang Wu, Ameet Talwalkwar, and Graham Neubig. 2024. https://doi.org/10.1162/tacl_a_00685 Do LLMs Exhibit Human-like Response Biases? A Case Study in Survey Design . Transactions of the Association for Computational Linguistics, 12:1011--1026

    2024 doi

  27. [35]

    Petter T \"o rnberg. 2023. https://arxiv.org/abs/2304.06588 ChatGPT-4 outperforms experts and crowd workers in annotating political Twitter messages with zero-shot learning . Preprint, arXiv:2304.06588

    2023 arXiv

  28. [36]

    Joost Van Rosmalen, Hester Van Herk, and Patrick J. F. Groenen. 2010. https://www.jstor.org/stable/20618962 Identifying response styles: A latent-class bilinear multinomial logit model . Journal of Marketing Research, 47(1):157--172

    2010

  29. [37]

    Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis. 2024. https://arxiv.org/abs/2404.18796 Replacing judges with juries: Evaluating LLM generations with a panel of diverse models . P...

    2024 arXiv

  30. [38]

    Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2023. https://arxiv.org/abs/2305.17926 Large language models are not fair evaluators . Preprint, arXiv:2305.17926

    2023 arXiv

  31. [39]

    Minghao Wu and Alham Fikri Aji. 2023. https://arxiv.org/abs/2307.03025 Style over substance: Evaluation biases for large language models . Preprint, arXiv:2307.03025

    2023 arXiv

  32. [40]

    Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. 2024. https://openreview.net/forum?id=tr0KidwPLc Evaluating large language models at evaluating instruction following . In The Twelfth International Conference on Learning Representations

    2024

  33. [41]

    Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. https://proceedings.mlr.press/v139/zhao21c.html Calibrate before use: Improving few-shot performance of language models . In Proceedings of the 38th International Conference on Machine Learning, pages 12697...

    2021

  34. [42]

    Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. 2023 a . Large language models are not robust multiple choice selectors. In The Twelfth International Conference on Learning Representations

    2023

  35. [43]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023 b . Judging LLM-as-a-Judge with MT - B ench and C hatbot A rena. Advances in Neural Information Processing Systems, 36:46595--46623

    2023

  36. [44]

    Han Zhou, Xingchen Wan, Lev Proleev, Diana Mincu, Jilin Chen, Katherine A Heller, and Subhrajit Roy. 2024. https://openreview.net/forum?id=L3FHMoKZcS Batch calibration: Rethinking calibration for in-context learning and prompt engineering . In The Twelfth International Confere...

    2024

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.