Pith. sign in

REVIEW 3 major objections 6 minor 4 cited by

Correlated Errors in Large Language Models

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that LLM errors are substantially correlated across models, and that this correlation grows with individual accuracy even after controlling for developer, architecture, and size.

desk verdict A solid, large-scale empirical study that establishes correlated LLM errors convincingly, but the headline claim that more accurate models err more similarly is genuinely threatened by the item-difficulty confound the authors acknowledge and don't resolve. read the letter →

arxiv 2506.07962 v1 pith:HKF426WY submitted 2025-06-09 cs.CL cs.AIcs.CYstat.ML

Elliot Kim , Avi Garg , Kenny Peng , Nikhil Garg This is my paper
classification cs.CLcs.AIcs.CYstat.ML
keywords correlatederrorslargelanguagemodelsalgorithmicmonoculturemodeldiversityLLM-as-a-judgehiringmarketssystemicexclusionMMLU
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to test a common assumption: that diversity in training data, architecture, and providers makes large language models substantively different from one another. Using responses from over 350 models on two leaderboard benchmarks and on a resume-screening task, it finds that when two models are both wrong, they pick the same wrong answer far more often than chance would predict. Strikingly, more accurate models are more correlated in their errors, even across distinct architectures and providers. The authors argue that this convergence in failure modes matters for practice: it biases LLM-as-judge evaluations and changes who gets hired and excluded in simulated labor markets. The underlying concern is that surface diversity across models may not deliver the robustness that multi-agent and high-stakes deployments assume.

What carries the argument

The paper's central object is the 'agreement rate when both models are wrong': for each model pair, the fraction of questions on which both models choose the same incorrect answer, given that both are wrong. Because it is conditioned on errors, this metric isolates correlated failure from the trivial fact that accurate models agree on many correct answers. The paper combines this pairwise metric with a regression framework over model-pair features (same developer, same architecture, size difference, and each model's individual accuracy plus their interaction) to separate lineage effects from accuracy-driven convergence, and it applies the same logic to resume-rating residuals where human labels serve as ground truth.

What would settle it

Compute agreement-on-error rates while controlling for question difficulty and for the semantic plausibility of each wrong answer. If the accuracy-driven increase in agreement vanishes once hard questions and attractive distractors are accounted for, then the claim that more accurate models share a true error process would be weakened; a direct test would also check whether the correlation persists on open-ended generation tasks where models are not constrained to a small answer set.

Watch

Extended reading notes

Core claim

The central claim is that LLM errors are substantially correlated, and that this correlation is highest among the most individually accurate models. On the Helm leaderboard, pairs of models agree on the same wrong answer about 60% of the time when both err, versus a random baseline of one-third; on the HuggingFace leaderboard the mean agreement-on-error rate is 0.423 versus a baseline of 0.127. Regression analysis shows that shared provider, shared base architecture, and similar model size all increase error agreement, but even after conditioning on these factors, more accurate model pairs are more correlated, with the accuracy-by-accuracy interaction term having a substantial positive coefficient in all three datasets. The authors then show two consequences: an LLM used as a judge systematically inflates the measured accuracy of less accurate models, especially those sharing its provider or architecture, and simulated hiring markets in which firms use distinct LLMs still exhibit substantial systemic exclusion of applicants, with applicant welfare and differential-access effects matching theoretical predictions on algorithmic monoculture.

Load-bearing premise

The measure of error correlation assumes that two models picking the same wrong answer reflects a shared underlying error process, rather than both models independently choosing the same 'best looking' wrong answer on hard questions.

Editorial extensions

If this is right

  • LLM-as-judge evaluations are systematically biased: a judge over-inflates the accuracy of models that are less accurate than itself, and inflation is larger for models from the same provider or architecture, so leaderboard comparisons should be calibrated per judge-model pair using ground-truth data.
  • Even when many firms deploy different LLMs in a hiring market, a substantial fraction of applicants are screened out of every opportunity; with 20 distinct LLMs in use, the systemic exclusion rate remains around 20 percent.
  • Firm and worker welfare in hiring markets depend on the degree of error correlation, not just on individual model accuracy; using the latest, most accurate models improves firm-side match quality, while full monoculture benefits matched applicants but worsens differential access.
  • The paper's results imply that simply using multiple different models is not a reliable route to diversity of opinion; for multi-agent and high-stakes systems, the relevant quantity is error correlation, and it should be measured or tracked rather than assumed away.
  • Leaderboards already hold the data needed to track cross-model error correlation, and the authors recommend that such ecosystem-level monitoring become a routine part of model evaluation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The accuracy-correlation link suggests a possible convergence mechanism: training and selection pipelines that push models toward high scores on shared benchmarks may also funnel them into the same residual failure modes, implying that scaling individual accuracy may decrease, not increase, useful model diversity.
  • The paper's metric compares choices among fixed wrong answers; on open-ended generation tasks, where models have more degrees of freedom, the measured correlation could be lower or higher, so a natural extension is to measure agreement on erroneous open-ended outputs rather than only on multiple-choice items.
  • If error correlation is largely shared, then disagreement between two models may itself carry information about which answer is correct; one testable extension is whether the answer a pair of correlated models agrees on is more likely to be correct than a single model's choice.
  • The resume-market simulations treat human labels as ground truth, but resume-job fit is subjective; a further step is to test whether the monoculture predictions hold under alternative expert or panel gold labels.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper investigates the extent to which different large language models make the same errors. Using responses from 349 HuggingFace Open LLM Leaderboard models and 71 HELM models on MMLU-style multiple-choice questions, plus 20 models on a newly constructed resume-job screening task, the authors measure pairwise agreement rates conditional on both models being wrong. They report that models agree on the same wrong answer far more often than a random baseline (e.g., 60% on HELM versus 1/3), that shared company, architecture, and size increase agreement, and—crucially—that pairs of more accurate models have more correlated errors even after conditioning on these shared characteristics. They then examine downstream consequences for LLM-as-judge evaluation, showing that judges inflate the accuracy of less-accurate models, and for hiring markets, finding substantial systemic exclusion and matching-market effects consistent with theoretical predictions about algorithmic monoculture. The paper includes extensive appendices with regression tables, model lists, and additional analyses.

Significance. If the central accuracy-correlation finding is valid, the paper makes an important empirical contribution to the emerging literature on LLM ecosystem diversity and algorithmic monoculture. Its strengths include the unusually large scale of the multiple-choice data (349 and 71 models, with per-question answers), the release of code and data, the use of multiple independent datasets including a realistic resume-screening task, and the explicit testing of prior theoretical predictions from Peng & Garg (2024a) in the labor-market section. The paper also carefully distinguishes its metric from concurrent work by Goel et al. and reports several robustness checks in the appendix. However, the main metric is acknowledged by the authors to conflate error generation with item difficulty and answer closeness, and this confound is not quantified; the paper also treats pair-level observations as independent in its regressions. Because the headline claim is that correlation increases with accuracy, these issues are load-bearing rather than cosmetic.

major comments (3)
  1. [§3.1, Table 1, §6] The central metric, 'agreement rate when both models are wrong,' conflates shared error-generation processes with item-level difficulty and distractor attractiveness. The authors explicitly acknowledge in §6 that 'some incorrect answers may be closer to correct... some questions may also be harder than others,' but they do not quantify how this affects the reported coefficients. Under the plausible mechanism that more accurate models are wrong only on a small, hard subset of items, and that hard items tend to have one most-plausible wrong option, the positive coefficients on Acc.1, Acc.2, and especially the Acc.1:Acc.2 interaction in Table 1 could arise even if error processes are not intrinsically more correlated. A within-difficulty analysis (e.g., stratifying questions by the fraction of models that answer correctly, or by ensemble disagreement) or a null model that conditions on question-specific wrong-answer priors is needed to secure the claim that error correlation increases with accuracy. The alternate metrics in Appendix Tables 3 and 6 do not resolve the issue, because they are also conditional on at least one model erring and inherit the same item-selection effect.
  2. [Appendix C, Tables 1–8] All pair-level regressions treat pairs of models as independent observations, but each model appears in many pairs, so the reported standard errors are understated. The t-statistics are extremely large, so the signs of the coefficients may be robust, but the reported precision is not trustworthy. I request cluster-robust standard errors grouped by model (or a permutation/randomization test over models) for the regressions in Table 1 and Appendix C, with a statement of whether the conclusions change.
  3. [§3.1, Table 8 (RESUMES)] The resume dataset is small (20 models, 190 model pairs) and the accuracy proxy is the correlation with human labels, which are subjective and available for only 450 of 1,800 resume-job pairs. The 'same company' coefficient in Table 8 is not significant and the accuracy coefficients are substantially smaller than in the multiple-choice datasets. Describing the result as holding 'across three datasets' in the abstract and introduction overstates the evidence from this dataset; the resume analysis should be presented as exploratory or the claims should be softened accordingly.
minor comments (6)
  1. [Abstract] The abstract reads 'on one leaderboard dataset, models agree 60% of the time,' which is the HELM result; naming the dataset in the abstract would improve precision.
  2. [§3.2, Figure 1] The color scale in Figure 1 is not described; adding a colorbar or explicitly stating that the entries are agreement rates would help the reader interpret the heatmaps.
  3. [§5.1, Eq. (1)] The definition of systemic exclusion depends on the percentile ranking of each firm; the text should clarify that the interview threshold p=0.25 means the top quarter of each firm's applicant ranking receives interviews, and that this implicitly assumes a fixed applicant pool per firm.
  4. [Appendix A.1] The description of the HELM dataset says it has 14,042 questions, but the main text uses 'Helm' and 'Helm leaderboard'; the occasional 'Helmet' typo should be corrected throughout.
  5. [Table 12] Several HuggingFace models have '?' for the architecture field; the paper should state how these observations are handled in the regression (e.g., omitted, or a separate 'unknown' category).
  6. [Section 2] The phrase 'inchoosinguncorrelated models' in the multi-agent discussion has a missing space; there are several similar spacing errors in the text that should be corrected in copyediting.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's central correlation findings are empirical measurements, and its downstream simulations follow from those measurements rather than from fitted parameters or load-bearing self-citations.

full rationale

The paper's central claims are empirical and self-contained. Agreement-when-both-models-err is measured directly from leaderboard responses (Section 3.1), and the regression in Table 1 fits agreement rates on model characteristics; the positive accuracy coefficients are descriptive, not parameters that are then repredicted. The LLM-as-judge inflation (Section 4) is a computed downstream quantity (judged accuracy minus true accuracy) whose sign is not forced by the agreement metric alone; it depends on the empirical joint distribution of judge and model errors. The hiring analysis (Section 5) explicitly tests theoretical predictions from Peng & Garg (2024a) rather than using those predictions as inputs, and the self-citations to that prior work are hypotheses, not load-bearing evidence for the correlation estimates. In the resume analysis, residuals are defined as model-minus-human and accuracy is measured as correlation-with-human, so both use the same hand labels; however, the paper does not reduce residual correlation to human-correlation by any equation, and the regression coefficients in Table 8 are not forced by construction. The Conclusion's acknowledgement that some incorrect answers are 'closer' to correct and that some questions are harder is a genuine validity threat to the accuracy-correlation finding, but it is a confound, not a circular derivation. No step in the paper equates a prediction to its own input by definition.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the validity of the error correlation metric and the ground truth labels; no new entities are introduced. The free parameters shown affect the hiring and resume portions, not the leaderboard correlation analysis.

free parameters (2)
  • interview threshold p = 0.25
    Fraction of top-rated applicants interviewed at each firm in the hiring simulation; chosen by the authors, not estimated from data. It affects the systemic exclusion magnitudes in Figure 3.
  • number of resume and job clusters = 28
    Chosen via KMeans heuristic to maximize similarity between a resume cluster and a job description cluster; affects which resumes and jobs enter the RESUMES dataset.
assumptions (5)
  • domain assumption MMLU multiple choice questions have correct ground truth labels, and leaderboard answers are valid model predictions.
    Used throughout Section 3 to define errors and accuracy; errors are relative to these labels.
  • domain assumption Human hand labels of resume-job fit are a valid ground truth for applicant quality and firm fit.
    Used in Section 3.1 and Section 5 to define residuals and to measure match probability by true applicant rating.
  • domain assumption Agreement rate when both models are wrong is a valid measure of error correlation, with uniform random choice among wrong answers as the null.
    Defines the main dependent variable in Table 1; the paper notes in the Conclusion that hard questions and answer-choice structure may confound it.
  • domain assumption Pairwise observations in the regressions are independent enough for OLS inference.
    Table 1 regresses each pair of models; overlapping pairs may produce correlated residuals, so standard errors may be understated.
  • domain assumption The stable matching model captures relevant labor market behavior, including firm and applicant preferences.
    Section 5.2 relies on stable matching to compute match probabilities, applicant rank, and differential access outcomes.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Correlated Errors in Large Language Models." pith.science (2026). https://pith.science/paper/HKF426WY

@misc{pith2026250607962,
  author       = {Pith},
  title        = {Pith review of: Correlated Errors in Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HKF426WY}},
  note         = {Machine review of arXiv:2506.07962}
}
read the original abstract

Diversity in training data, architecture, and providers is assumed to mitigate homogeneity in LLMs. However, we lack empirical evidence on whether different LLMs differ meaningfully. We conduct a large-scale empirical evaluation on over 350 LLMs overall, using two popular leaderboards and a resume-screening task. We find substantial correlation in model errors -- on one leaderboard dataset, models agree 60% of the time when both models err. We identify factors driving model correlation, including shared architectures and providers. Crucially, however, larger and more accurate models have highly correlated errors, even with distinct architectures and providers. Finally, we show the effects of correlation in two downstream tasks: LLM-as-judge evaluation and hiring -- the latter reflecting theoretical predictions regarding algorithmic monoculture.

Figures

Figures reproduced from arXiv: 2506.07962 by the authors.

Figure 1
Figure 1. Agreement when both models are wrong. Models are sorted by accuracy. More accurate models have more correlated errors, and almost all model pairs have higher agreement rates than random disagreement on errors would imply. than it—especially for models of the same provider or ar￾chitecture. Furthermore, motivated by concerns regarding algorithmic monoculture and systemic exclusion in hiring markets (Kleinberg & Ragha… view at source ↗
Figure 2
Figure 2. Evaluating LLM-as-judge on HELM. In each plot, one model is used as the judge. Each dot is another model; the y-axis is the accuracy inflation (compared to ground truth) of using the given model as the judge, and the x-axis is the model’s true accuracy. The vertical red line corresponds to the true accuracy of the judge. Each judge tends to inflate the accuracy of models that are less accurate than itself, especiall… view at source ↗
Figure 3
Figure 3. Systemic exclusion (fraction of resumes with no job offers) when p = 0.25: (a) for varying number of distinct LLMs used in a job hiring market, (b) for varying firm preference methods. Lower systemic exclusion is better. 1 2 3 4 5 6 7 8 9 10 True Applicant Rating 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Match Probability Firm Preference Method Same LLM (1) Same Company LLM (3-7) Latest LLM (5) Random LLM (5) Uniformly Random (5)… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: shows how match probability changes with true applicant rating, in each market. As expected from the theoretical literature—among the LLM-based markets due to monoculture—using the same (randomly chosen) single LLM leads to the worst firm welfare, with relatively small…
Figure 5
Figure 5. Figure 5: Market outcomes depending on firm LLM usage: (a) gives average applicant rank, (b) gives the effect of differential application access. Both plots are generated by averaging over 1500 random markets; in each, all firms share a randomly sampled job description. Applican…
Figure 6
Figure 6. Figure 6: Cluster analysis for selecting resumes and job descriptions in RESUMES. We used 20 open-source models available on Amazon Bedrock, specifically models made from Meta, Mistral AI, and Amazon, Anthropic API, OpenAI API (see Appendix 10), and 2 prompts. For each applicant…
Figure 7
Figure 7. Figure 7: The prompts used to score the fit of a resume to a job description. The first prompt is named “firm rate comb short 1“, and the second prompt is named “firm rate comb 2“. The resume and job description texts are taken directly from the dataset. A similar prompt is used…
Figure 8
Figure 8. Figure 8: Evaluating LLM-as-judge on HUGGINGFACE. In each plot, one model is used as the judge. Each dot is another model; the y-axis is the accuracy inflation (compared to ground truth) of using the given model as judge, and the x-axis is the model’s true accuracy. Each judge t…
Figure 9
Figure 9. Figure 9: Correlation heatmaps on RESUMES. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Replication of matching markets experiments given LLM-determined applicant preferences. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. How Much Does Correctness Cost? Budgeted Placement of Strong Correctors in a Weak Multi-Agent Swarm

    cs.AI 2026-07 conditional novelty 6.5 of 10

    Heterogeneous cost-coupled corrector placement in a consensus swarm stays submodular, so cost-benefit greedy is near-optimal, and cost-quality curvature decides few-strong versus many-medium oracles in a task-dependent way.

  2. Are Diversity Metrics Measuring Diversity? A Capability-Controlled Audit of Majority-Vote Gain in LLM Ensembles

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Diversity metrics used to select LLM ensembles are largely capability proxies; after control, only a modest pairwise co-failure association with majority-vote gain remains.

  3. PlanFlip: Attacking Multi-Agent LLM Systems via Planning-Phase Prompt Injection

    cs.AI 2026-04 reject novelty 5.0 of 10

    Planning-phase injection can silently corrupt homogeneous multi-agent LLM pipelines, but the headline claims are undermined by metric and consistency errors.

  4. DIANOIA: Diagnostic Decomposition and Joint Optimization for Multi-Agent Reasoning

    cs.AI 2026-02 reject novelty 3.0 of 10

    Multi-agent reasoning gains can be written as coverage × selection accuracy, which is a conditioning identity rather than a new decomposition; the PRISM system still shows moderate benchmark gains.

Reference graph

Works this paper leans on

52 extracted references · 28 canonical work pages · cited by 4 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    F., Joachims, T., and Antonio, A

    Alvero, A., Lee, J., Regla-Vargas, A., Kizilcec, R. F., Joachims, T., and Antonio, A. L. Large language models, social demography, and hegemony: comparing authorship in human and synthetic text. Journal of Big Data, 11 0 (1): 0 138, 2024

    work page 2024

  3. [3]

    Upwork job postings dataset 2024 (50k records), 2024

    Asaniczka. Upwork job postings dataset 2024 (50k records), 2024. URL https://www.kaggle.com/datasets/asaniczka/upwork-job-postings-dataset-2024-50k-records

    work page 2024

  4. [4]

    Hiring under congestion and algorithmic monoculture: Value of strategic behavior

    Baek, J., Bastani, H., and Chen, S. Hiring under congestion and algorithmic monoculture: Value of strategic behavior. arXiv preprint arXiv:2502.20063, 2025

    arXiv 2025

  5. [5]

    Resume dataset, 2022

    Bhawal, S. Resume dataset, 2022. URL https://www.kaggle.com/datasets/snehaanbhawal/resume-dataset

    work page 2022

  6. [6]

    A., Kumar, A., Jurafsky, D., and Liang, P

    Bommasani, R., Creel, K. A., Kumar, A., Jurafsky, D., and Liang, P. S. Picking on the same person: Does algorithmic monoculture lead to outcome homogenization? Advances in Neural Information Processing Systems, 35: 0 3663--3678, 2022

    work page 2022

  7. [7]

    I., Creel, K

    Bommasani, R., Soylu, D., Liao, T. I., Creel, K. A., and Liang, P. Ecosystem graphs: The social footprint of foundation models. March 2023. URL https://arxiv.org/abs/2303.15772

    arXiv 2023

  8. [8]

    Chen, Z., Li, J., Chen, P., Li, Z., Sun, K., Luo, Y., Mao, Q., Yang, D., Sun, H., and Yu, P. S. Harnessing multiple large language models: A survey on llm ensemble. arXiv preprint arXiv:2502.18036, 2025

    arXiv 2025

Show all 52 references
  1. [9]

    and Hellman, D

    Creel, K. and Hellman, D. The algorithmic leviathan: Arbitrariness, fairness, and opportunity in algorithmic decision-making systems. Canadian Journal of Philosophy, 52 0 (1): 0 26--43, 2022

    2022

  2. [10]

    D., Goel, S., Huq, A., and Tambe, P

    Gaebler, J. D., Goel, S., Huq, A., and Tambe, P. Auditing the use of language models to guide hiring decisions, 2024. URL https://arxiv.org/abs/2404.03086

    2024 arXiv

  3. [11]

    Geirhos, R., Meding, K., and Wichmann, F. A. Beyond accuracy: quantifying trial-by-trial behaviour of cnns and humans by measuring error consistency. Advances in neural information processing systems, 33: 0 13890--13902, 2020

    2020

  4. [12]

    A., Chandra, K

    Goel, S., Struber, J., Auzina, I. A., Chandra, K. K., Kumaraguru, P., Kiela, D., Prabhu, A., Bethge, M., and Geiping, J. Great models think alike and this undermines ai oversight. arXiv preprint arXiv:2502.04313, 2025

    2025 arXiv

  5. [13]

    Auditing work: Exploring the new york city algorithmic bias audit regime

    Groves, L., Metcalf, J., Kennedy, A., Vecchione, B., and Strait, A. Auditing work: Exploring the new york city algorithmic bias audit regime. In The 2024 ACM Conference on Fairness, Accountability, and Transparency, pp.\ 1107--1120, 2024

    2024

  6. [14]

    A survey on llm-as-a-judge

    Gu, J., Jiang, X., Shi, Z., Tan, H., Zhai, X., Xu, C., Li, W., Shen, Y., Ma, S., Liu, H., et al. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594, 2024

    2024 arXiv

  7. [15]

    S., and Chouldechova, A

    Guerdan, L., Barocas, S., Holstein, K., Wallach, H., Wu, Z. S., and Chouldechova, A. Validating llm-as-a-judge systems in the absence of gold labels. arXiv preprint arXiv:2503.05965, 2025

    2025

  8. [16]

    Measuring massive multitask language understanding

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020

    2009 arXiv

  9. [17]

    Scarce resource allocations that rely on machine learning should be randomized

    Jain, S., Creel, K., and Wilson, A. Scarce resource allocations that rely on machine learning should be randomized. arXiv preprint arXiv:2404.08592, 2024 a

    2024 arXiv

  10. [18]

    Algorithmic pluralism: A structural approach to equal opportunity

    Jain, S., Suriyakumar, V., Creel, K., and Wilson, A. Algorithmic pluralism: A structural approach to equal opportunity. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, pp.\ 197--206, 2024 b

    2024

  11. [19]

    Jha, R., Zhang, C., Shmatikov, V., and Morris, J. X. Harnessing the universal geometry of embeddings. arXiv preprint arXiv:2505.12540, 2025

    2025

  12. [20]

    Jiechieu, K. F. F. and Tsopze, N. Skills prediction based on multi-label resume classification using cnn with model predictions explanation. Neural Computing and Applications, 33 0 (10): 0 5069--5087, 2021. ISSN 1433-3058. doi:10.1007/s00521-020-05302-x. URL https://doi.org/10...

    2021 doi

  13. [21]

    Kamoi, R., Das, S. S. S., Lou, R., Ahn, J. J., Zhao, Y., Lu, X., Zhang, N., Zhang, Y., Zhang, H. R., Vummanthala, S. R., Dave, S., Qin, S., Cohan, A., Yin, W., and Zhang, R. Evaluating LLM s at detecting errors in LLM responses. In First Conference on Language Modeling, 2024 a...

    2024

  14. [22]

    When can llms actually correct their own mistakes? a critical survey of self-correction of llms

    Kamoi, R., Zhang, Y., Zhang, N., Han, J., and Zhang, R. When can llms actually correct their own mistakes? a critical survey of self-correction of llms. Transactions of the Association for Computational Linguistics, 12: 0 1417--1440, 2024 b

    2024

  15. [23]

    and Raghavan, M

    Kleinberg, J. and Raghavan, M. Algorithmic monoculture and social welfare. Proceedings of the National Academy of Sciences, 118 0 (22): 0 e2018340118, 2021

    2021

  16. [24]

    A., Manning, C

    Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., Newman, B., Yuan, B., Yan, B., Zhang, C., Cosgrove, C. A., Manning, C. D., Re, C., Acosta-Navas, D., Hudson, D. A., Zelikman, E., Durmus, E., Ladhak, F., Rong,...

    2023

  17. [25]

    Sparse autoencoders for hypothesis generation

    Movva, R., Peng, K., Garg, N., Kleinberg, J., and Pierson, E. Sparse autoencoders for hypothesis generation. arXiv preprint arXiv:2502.04382, 2025

    2025 arXiv

  18. [26]

    Llm evaluators recognize and favor their own generations

    Panickssery, A., Bowman, S., and Feng, S. Llm evaluators recognize and favor their own generations. Advances in Neural Information Processing Systems, 37: 0 68772--68802, 2024

    2024

  19. [27]

    and Garg, N

    Peng, K. and Garg, N. Monoculture in matching markets. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024 a . URL https://openreview.net/forum?id=P5yezHuMSS

    2024

  20. [28]

    and Garg, N

    Peng, K. and Garg, N. Wisdom and foolishness of noisy matching markets. In Proceedings of the 25th ACM Conference on Economics and Computation, pp.\ 675--675, 2024 b

    2024

  21. [29]

    Do these llm benchmarks agree? fixing benchmark evaluation with benchbench

    Perlitz, Y., Gera, A., Arviv, O., Yehudai, A., Bandel, E., Shnarch, E., Shmueli-Scheuer, M., and Choshen, L. Do these llm benchmarks agree? fixing benchmark evaluation with benchbench. arXiv preprint arXiv:2407.13696, 2024

    2024 arXiv

  22. [30]

    Competition and diversity in generative ai

    Raghavan, M. Competition and diversity in generative ai. arXiv preprint arXiv:2412.08610, 2024

    2024 arXiv

  23. [31]

    K., Ahmad, A., Andreetto, M., Prabhakaran, V., Prabhu, U., Dieng, A

    Senthilkumar, N. K., Ahmad, A., Andreetto, M., Prabhakaran, V., Prabhu, U., Dieng, A. B., Bhattacharyya, P., and Dave, S. Beyond aesthetics: Cultural competence in text-to-image models. Advances in Neural Information Processing Systems, 37: 0 13716--13747, 2024

    2024

  24. [32]

    Large language models are inconsistent and biased evaluators

    Stureborg, R., Alikaniotis, D., and Suhara, Y. Large language models are inconsistent and biased evaluators. CoRR, 2024

    2024

  25. [33]

    F., Ilhan, F., Huang, T., Hu, S., and Liu, L

    Tekin, S. F., Ilhan, F., Huang, T., Hu, S., and Liu, L. Llm-topla: Efficient llm ensemble by maximising diversity. arXiv preprint arXiv:2410.03953, 2024

    2024 arXiv

  26. [34]

    Law and the emerging political economy of algorithmic audits

    Terzis, P., Veale, M., and Gaumann, N. Law and the emerging political economy of algorithmic audits. In The 2024 ACM Conference on Fairness, Accountability, and Transparency, pp.\ 1255--1267, 2024

    2024

  27. [35]

    Toups, C., Bommasani, R., Creel, K., Bana, S., Jurafsky, D., and Liang, P. S. Ecosystem-level analysis of deployed machine learning reveals homogeneous outcomes. Advances in Neural Information Processing Systems, 36: 0 51178--51201, 2023

    2023

  28. [36]

    Replacing judges with juries: Evaluating llm generations with a panel of diverse models

    Verga, P., Hofstatter, S., Althammer, S., Su, Y., Piktus, A., Arkhangorodsky, A., Xu, M., White, N., and Lewis, P. Replacing judges with juries: Evaluating llm generations with a panel of diverse models. arXiv preprint arXiv:2404.18796, 2024

    2024 arXiv

  29. [37]

    F., Wang, A., Barocas, S., Chouldechova, A., Atalla, C., Blodgett, S

    Wallach, H., Desai, M., Pangakis, N., Cooper, A. F., Wang, A., Barocas, S., Chouldechova, A., Atalla, C., Blodgett, S. L., Corvi, E., et al. Evaluating generative ai systems is a social science measurement challenge. arXiv preprint arXiv:2411.10939, 2024

    2024 arXiv

  30. [38]

    Wang, A., Morgenstern, J., and Dickerson, J. P. Large language models that replace human participants can harmfully misportray and flatten identity groups. Nature Machine Intelligence, pp.\ 1--12, 2025

    2025

  31. [39]

    Self-preference bias in llm-as-a-judge

    Wataoka, K., Takahashi, T., and Ri, R. Self-preference bias in llm-as-a-judge. arXiv preprint arXiv:2410.21819, 2024

    2024 arXiv

  32. [40]

    D., Wallach, H., Mitchell, M., Wang, A., Salaudeen, O., Bommasani, R., Ganguli, D., Koyejo, S., and Isaac, W

    Weidinger, L., Raji, I. D., Wallach, H., Mitchell, M., Wang, A., Salaudeen, O., Bommasani, R., Ganguli, D., Koyejo, S., and Isaac, W. Toward an evaluation science for generative ai systems. arXiv preprint arXiv:2503.05336, 2025

    2025 arXiv

  33. [41]

    and Kenett, Y

    Wenger, E. and Kenett, Y. We're different, we're the same: Creative homogeneity across llms. arXiv preprint arXiv:2501.19361, 2025

    2025 arXiv

  34. [42]

    and Horton, J

    Wiles, E. and Horton, J. J. Generative ai and labor market matching efficiency. Available at SSRN 5187344, 2025

    2025

  35. [43]

    Algorithmic writing assistance on jobseekers’ resumes increases hires

    Wiles, E., Munyikwa, Z., and Horton, J. Algorithmic writing assistance on jobseekers’ resumes increases hires. Management Science, 2025

    2025

  36. [44]

    and Caliskan, A

    Wilson, K. and Caliskan, A. Gender, race, and intersectional bias in resume screening via language model retrieval. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, volume 7, pp.\ 1578--1590, 2024

    2024

  37. [45]

    and Schein, A

    Wolfram, C. and Schein, A. Layers at similar depths generate similar activations across llm architectures. arXiv preprint arXiv:2504.08775, 2025

    2025 arXiv

  38. [46]

    M., Vecchione, B., Qu, T., Cai, P., Smith, A., Investigators, C

    Wright, L., Muenster, R. M., Vecchione, B., Qu, T., Cai, P., Smith, A., Investigators, C. . S., Metcalf, J., Matias, J. N., et al. Null compliance: Nyc local law 144 and the challenges of algorithm accountability. In The 2024 ACM Conference on Fairness, Accountability, and Tra...

    2024

  39. [47]

    Generative monoculture in large language models

    Wu, F., Black, E., and Chandrasekaran, V. Generative monoculture in large language models. arXiv preprint arXiv:2407.02209, 2024

    2024 arXiv

  40. [48]

    Echoes in ai: Quantifying lack of plot diversity in llm outputs

    Xu, W., Jojic, N., Rao, S., Brockett, C., and Dolan, B. Echoes in ai: Quantifying lack of plot diversity in llm outputs. arXiv preprint arXiv:2501.00273, 2024

    2024 arXiv

  41. [49]

    One llm is not enough: Harnessing the power of ensemble learning for medical question answering

    Yang, H., Li, M., Zhou, H., Xiao, Y., Fang, Q., Zhou, S., and Zhang, R. One llm is not enough: Harnessing the power of ensemble learning for medical question answering. Journal of Medical Internet Research, pp.\ 70080, 2025. doi:10.2196/70080. URL https://preprints.jmir.org/pr...

    2025 doi

  42. [50]

    Generative ai meets open-ended survey responses: Research participant use of ai and homogenization

    Zhang, S., Xu, J., and Alvero, A. Generative ai meets open-ended survey responses: Research participant use of ai and homogenization. Sociological Methods & Research, pp.\ 00491241251327130, 2025

    2025

  43. [51]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36: 0 46595--46623, 2023

    2023

  44. [52]

    Hypothesis Generation with Large Language Models

    Zhou, Y., Liu, H., Srivastava, T., Mei, H., and Tan, C. Hypothesis Generation with Large Language Models . 0 (arXiv:2404.04326), August 2024

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.