Pith. sign in

REVIEW 4 major objections 5 minor 26 references

Peeking Behind Closed Doors: Risks of LLM Evaluation by Private Data Curators

T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Private AI evaluators favor models trained on their own data.

desk verdict A timely conflict-of-interest argument undermined by a misdefined self-bias metric and a statistical coin flip. read the letter →

arxiv 2503.04756 v1 pith:6QZGJPFU submitted 2025-02-09 cs.CY cs.AIcs.CLcs.LG

Hritik Bansal , Pratyush Maini This is my paper · ORCID
classification cs.CYcs.AIcs.CLcs.LG
keywords privateevaluationLLMbiasdatacuratorconflictofinterestELOratingannotatorself-preferenceleaderboard
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that private data-curation companies that sell training data and also grade models create an inherent evaluation bias toward the models trained on their own data. To demonstrate the mechanism, the authors used GPT-4o and Claude-Sonnet-3.5 as stand-ins for the expert annotators of two rival companies, fine-tuned an open-weight base model on each company's answers to 10,000 prompts, and then had each evaluator judge the two fine-tuned models on 805 fresh queries. Each evaluator preferred the model trained on its own data, with self-bias scores of 12.90% for the GPT-4o side and 10.51% for the Claude side. Translated into the ELO ratings used by public leaderboards, the preferences produce an 81-point gap when Claude evaluates both models. The paper concludes that even a good-faith private evaluator can mislead clients and investors, and that separating evaluation from training-data provision is necessary.

What carries the argument

The load-bearing mechanism is the dual-role pipeline: the same private company both supplies instruction-following training data and evaluates the resulting models. The paper operationalizes this by fine-tuning the same base model on answers generated by two different language models, using those same two models as judges over a fresh set of queries, and then converting pairwise preference counts into a self-bias score and into ELO ratings with the same method used by the public leaderboard it criticizes. The self-bias score measures how much more often one evaluator prefers a model than the other evaluator does, normalized by total preferences, which gives the structural conflict a concrete numerical size.

What would settle it

A direct falsifying experiment would recruit two teams of human annotators at a real data-curation firm, have each team write training answers used to fine-tune the same base model, then have each team evaluate both models on fresh prompts while blinded to provenance. If teams do not consistently prefer the model fine-tuned on their own answers—or if GPT-4o's 50.68% versus 49.32% split turns out to be statistically indistinguishable from chance—the paper's central claim loses its empirical support.

Watch

Extended reading notes

Core claim

The paper's central claim is that a private data curator's dual role—providing fine-tuning data to model developers and then evaluating those same developers' models—produces a systematic evaluator bias toward models trained on the curator's own data, even when the curator acts in good faith and does not leak test prompts. The experiments support this by showing that Evaluator Alpha (GPT-4o) preferred its own fine-tuned model 407 times versus 396 for the rival model, while Evaluator Beta (Claude) preferred its own model 489 times versus 314 for the rival. The largest effect appears in ELO simulation: Evaluator Beta assigns its own model a 1040 ELO versus 959 for the rival, an 81-point gap that the paper argues is large enough to shift public perception and investment decisions. The authors present this as the mildest form of bias, since there is no intentional tampering, only the subjective preferences of annotators who wear both hats.

Load-bearing premise

The argument depends on the unverified assumption that the self-preference bias shown by GPT-4o and Claude-Sonnet-3.5 is a reliable stand-in for the preferences of the human expert annotators who create training data and grade models, which the paper does not test with human subjects.

Editorial extensions

If this is right

  • A private leaderboard's ranking reflects the preferences of its own annotators as much as model quality, so two private leaderboards can rank the same pair of models in opposite order.
  • Commercial evaluations should be treated as conflicted unless the curator discloses its training-data relationships, analogous to the Chinese walls used in finance.
  • An ELO gap of 81 points from self-bias is large enough to shift public perception and investment decisions even when the underlying models are otherwise comparable.
  • Separating training-data provision from evaluation, or using multiple independent evaluator pools, would reduce but not eliminate this bias.
  • The near-tie GPT-4o result (50.68% vs 49.32%) shows that the bias magnitude depends on which evaluator model is used, so a single private leaderboard may not even be internally stable across evaluator teams.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If LLM self-preference is a reliable model for human annotator bias, the same distortion should appear in any preference-based leaderboard where the human voters overlap with the people who created a model's training data, including open platforms whose chat logs are released.
  • The paper only tests one base model and two GPT-class evaluators; a natural extension would vary the base model and evaluator families to see whether self-bias scales with model capability, dataset size, or task difficulty.
  • A concrete, policy-relevant test would be to have a private data curator run its leaderboard alongside an independent blinded evaluation on the same models; if the rankings diverge, disclosure alone cannot fix the bias and governance changes would be needed.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper argues that private LLM evaluation by data curation companies poses financial conflict-of-interest and evaluation-bias risks, focusing on the overlap between annotators who create training data and those who evaluate models. It reports a simulation in which GPT-4o and Claude-Sonnet-3.5 act as stand-ins for two companies' expert annotators, two Mistral models are fine-tuned on each evaluator's outputs, and each evaluator then compares the two models on 805 AlpacaEval queries. The authors report self-bias for both evaluators, quantify it with a formula, and simulate ELO ratings that they claim show 'significant differences' based on evaluator preferences. The paper also discusses financial-incentive disclosures and policy remedies such as 'Chinese walls.'

Significance. The topic is timely, and the experimental design directly targets a plausible mechanism for conflict of interest in private evaluation: fine-tuning on a judge's data and then measuring that judge's preference. This directness is a strength, as is the paper's transparency in noting in the Figure 1 caption that the effect is not observed in the current SEAL rankings. However, the quantitative support is undermined by a mis-specified 'self-bias' metric, the absence of any statistical inference, and the lack of validation that LLM self-preference transfers to human annotators. If the analysis were corrected and the claims suitably narrowed, the paper could serve as a useful cautionary note; in its present form, the abstract's and Section 3.2's strong claims are not supported by the data as reported.

major comments (4)
  1. [Section 3.2, Eq. (1)] Equation (1) is labeled 'Self Bias_A' but it actually measures the cross-judge disagreement about model A, namely the difference between Judge A's and Judge B's preference counts for M_A normalized by their sum. It is not a within-judge measure of a judge's tendency to prefer its own fine-tuned model. For Evaluator Alpha (GPT-4o), Table 1 shows that GPT-4o itself preferred Model A only 407/803 times (50.68%), which is statistically indistinguishable from chance (two-sided binomial test p≈0.72). The reported 12.90% in Eq. (3) arises from comparing GPT-4o's 407 preferences for Model A with Claude's 314 preferences for Model A, not from GPT-4o's own self-preference. Consequently, the Section 3.2 statement that 'The results highlight a clear bias aligned with each evaluator's preferences' is not supported for GPT-4o, and the claim that 'both models exhibit self-bias' is contradicted by the raw within-judge preference rates.
  2. [Section 3.3, Table 2] The ELO simulation reports a 1003 versus 996 rating gap for Evaluator Alpha, a difference of only 7 points, yet the text states that these ELO ratings 'show significant differences based on evaluator preferences' and that the differences 'further highlight the significance of the bias.' No uncertainty, confidence interval, or significance test is provided for either ELO estimate or for the 1003–996 gap. Given that the underlying preference counts are 407 versus 396, this gap is plausibly pure noise. The paper should report bootstrap or Bradley–Terry-model uncertainties and state whether the gap exceeds the simulation's noise floor before claiming significance.
  3. [Section 3.1, Experimental Protocol and Evaluators] The experiment uses GPT-4o and Claude-Sonnet-3.5 as simulators for expert human annotators at private data-curation companies, but no human-subject data, prior human-annotation validation, or argument for why LLM self-preference transfers to human annotators is provided. The abstract and Section 2.2 assert that 'the subjective preferences of private expert annotators will lead to inherent evaluation bias,' which is a claim about human behavior. As the current design only demonstrates (at best) a property of LLM evaluators, the external-validity gap is load-bearing. The paper should either include human-subject evidence or explicitly restrict its conclusions to LLM-based evaluators and discuss the conditions under which the mechanism might extend to humans.
  4. [Figure 1 and Section 1.2.2] The caption of Figure 1 contains a direct self-limitation: 'We don't see this effect in the current SEAL rankings.' This admission is in tension with the abstract's claim that private expert annotators' subjective preferences lead to 'inherent evaluation bias.' If the proposed mechanism is 'inherent,' then one would expect some trace of it in the real private leaderboard that motivated the paper; the authors should explain why the SEAL non-effect does not contradict their central claim, or they should weaken the claim from 'inherent' to 'a demonstrated risk under specific conditions.'
minor comments (5)
  1. [Section 3.2, Eqs. (2)-(3)] The label 'GPT-4' in Eq. (3) should be 'GPT-4o' for consistency with the rest of the paper, including Table 1.
  2. [Section 3.1, step 3] The protocol states that 805 queries were used, but Table 1 sums to 803 preference judgments; the discrepancy should be explained (for example, two queries may have produced ties or invalid outputs).
  3. [Section 3.3] The ELO simulation is described only as using 'the same method as employed by the LMSys leaderboard'; the initialization, update rule, handling of ties, and number of iterations should be specified so that the results are reproducible.
  4. [Section 1.1] The sentence 'specific models look better than some models might appear to perform better than they actually do' is garbled and should be rewritten for clarity.
  5. [Table 1] Adding exact binomial confidence intervals for the preference proportions (e.g., 95% CIs for 50.68% and 60.90%) would help readers see which differences are meaningfully distinguishable from chance.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's empirical preference measurements and ELO simulation are direct transformations of measured data, not derivations that reduce to their inputs.

full rationale

The claimed derivation chain is not circular. Section 3.1 constructs two Mistral models fine-tuned on GPT-4o and Claude-Sonnet-3.5 outputs, and Section 3.2 records each evaluator's preferences; this is direct measurement, not a derivation from an assumed conclusion. The 'Self Bias' formula in Equation (1) is a summary statistic computed from those preference counts, so the ELO results in Table 2 are deterministic transformations via the cited LMSys Bradley-Terry procedure of those same counts, not independent predictions. No parameter is fitted to a subset and then 'predicted' on a closely related quantity; no load-bearing argument reduces to a self-citation (the only corroborating citation, Panickssery et al. 2024, is external); and no uniqueness theorem or ansatz is imported from the authors' prior work. The review-level objections—that Equation (1) operationalizes 'self-bias' as cross-judge disagreement rather than a judge's own preference rate, that GPT-4o's 50.68% direct preference is near a statistical tie, that the LLM-judge simulation may not transfer to human annotators, and that the paper itself notes 'We don't see this effect in the current SEAL rankings'—are validity and interpretation concerns, not circularity under the definition of a claim reducing to its inputs by construction.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim depends on transferring LLM-judge self-preference to human annotators and on the statistical adequacy of the small evaluation set. Both premises are asserted rather than validated. No free parameters are fitted to data, and no new theoretical entities are introduced; Company Alpha and Beta are illustrative experimental conditions, not invented mechanisms.

assumptions (4)
  • domain assumption LLM judges (GPT-4o and Claude-Sonnet-3.5) faithfully simulate expert human annotators at private data curation companies.
    Section 3.1 uses these models as simulators for company evaluators, with no human-subject validation or discussion of model-specific response patterns.
  • domain assumption A model fine-tuned on a judge's outputs inherits that judge's stylistic preferences, so any preference differential reflects evaluator bias rather than quality differences.
    The experimental design in Section 3.1 assumes the fine-tuned Mistral models are valid proxies for 'models trained with the private curators' data' and that the preference split isolates bias.
  • standard math The Bradley-Terry / ELO model used by LMSys is an appropriate aggregation of pairwise preferences.
    Section 3.3 explicitly uses the same method as the LMSys leaderboard to convert preferences to ELO ratings.
  • domain assumption 805 prompt comparisons are sufficient to draw quantitative conclusions about self-bias and ELO shifts.
    No confidence intervals or significance tests are reported; the GPT-4o result is a near tie, making this assumption load-bearing for the quantitative claims.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Peeking Behind Closed Doors: Risks of LLM Evaluation by Private Data Curators." pith.science (2026). https://pith.science/paper/6QZGJPFU

@misc{pith2026250304756,
  author       = {Pith},
  title        = {Pith review of: Peeking Behind Closed Doors: Risks of LLM Evaluation by Private Data Curators},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6QZGJPFU}},
  note         = {Machine review of arXiv:2503.04756}
}
read the original abstract

The rapid advancement in building large language models (LLMs) has intensified competition among big-tech companies and AI startups. In this regard, model evaluations are critical for product and investment-related decision-making. While open evaluation sets like MMLU initially drove progress, concerns around data contamination and data bias have constantly questioned their reliability. As a result, it has led to the rise of private data curators who have begun conducting hidden evaluations with high-quality self-curated test prompts and their own expert annotators. In this paper, we argue that despite potential advantages in addressing contamination issues, private evaluations introduce inadvertent financial and evaluation risks. In particular, the key concerns include the potential conflict of interest arising from private data curators' business relationships with their clients (leading LLM firms). In addition, we highlight that the subjective preferences of private expert annotators will lead to inherent evaluation bias towards the models trained with the private curators' data. Overall, this paper lays the foundation for studying the risks of private evaluations that can lead to wide-ranging community discussions and policy changes.

Figures

Figures reproduced from arXiv: 2503.04756 by the authors.

Figure 1
Figure 1. Illustration of private evaluation pipeline showing the relationship between data curators, [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Screenshot showing OpenAI and Cohere as customers of ScaleAI, while ScaleAI also [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. An ELO difference of 44 on the LMSys leaderboard allows for significant bragging rights [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Demonstration of a simple adversarial strategy that can allow a user to maliciously vote [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: The North Cascades view that inspired our discussions on LLM evaluation bias. [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 12 canonical work pages

  1. [1]

    45 real-world llm applications and use cases from top companies

    Evidentally AI. 45 real-world llm applications and use cases from top companies. https://www.evidentlyai.com/blog/llm-applications, 2024. URL https://www.evidentlyai.com/blog/llm-applications. Accessed: 2024-11-22

    work page 2024

  2. [2]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. URL https://arxiv.org/abs/2110.14168

    arXiv

  3. [3]

    Generalization or memorization: Data contamination and trustworthy evaluation for large language models

    Yihong Dong et al. Generalization or memorization: Data contamination and trustworthy evaluation for large language models. arXiv preprint arXiv:2402.15938, 2024. URL https://arxiv.org/abs/2402.15938

    arXiv 2024

  4. [4]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2021. URL https://arxiv.org/pdf/2009.03300

    arXiv 2009

  5. [5]

    Not all llm reasoners are created equal

    Arian Hosseini et al. Not all llm reasoners are created equal. arXiv preprint arXiv:2410.01748, 2024. URL https://arxiv.org/abs/2410.01748

    arXiv 2024

  6. [6]

    Hashimoto

    Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval, 2023. URL https://github.com/tatsu-lab/alpaca_eval

    work page 2023

  7. [7]

    Alpacafarm: A simulation framework for methods that learn from human feedback

    Yao Li et al. Alpacafarm: A simulation framework for methods that learn from human feedback. arXiv preprint arXiv:2408.00118, 2024. URL https://arxiv.org/abs/2408.00118

    arXiv 2024

  8. [8]

    Transition from online elo rating system to bradley-terry model

    LMSYS-Org. Transition from online elo rating system to bradley-terry model. https://lmsys.org/blog/2023-12-07-leaderboard/#transition-from-online-elo-rating-system-to-bradley-terry-model, 2023. URL https://lmsys.org/blog/2023-12-07-leaderboard/#transition-from-online-elo-rating-system-to-bradley-terry-model. Accessed: 2024-11-22

    work page 2023

Show all 26 references
  1. [9]

    Lmsys org

    LMSYS-Org. Lmsys org. https://lmsys.org/, 2024. URL https://lmsys.org/. Accessed: 2024-11-22

    2024

  2. [10]

    Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models

    Seyed Iman Mirzadeh et al. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models. arXiv preprint arXiv:2410.05229, 2024. URL https://arxiv.org/abs/2410.05229v1

    2024 arXiv

  3. [11]

    Mistral-7b-v0.3, 2024

    Mistral-Team. Mistral-7b-v0.3, 2024. URL https://huggingface.co/mistralai/Mistral-7B-v0.3/tree/main

    2024

  4. [12]

    Usa contests

    Art of Problem-Solving. Usa contests. https://artofproblemsolving.com/community/c3158_usa_contests, 2024. URL https://artofproblemsolving.com/community/c3158_usa_contests. Accessed: 2024-11-22

    2024

  5. [13]

    Llm evaluators recognize and favor their own generations

    Arjun Panickssery, Samuel R Bowman, and Shi Feng. Llm evaluators recognize and favor their own generations. arXiv preprint arXiv:2404.13076, 2024

    2024 arXiv

  6. [14]

    Openai valued at \ 157 billion after closing \ 6.6 billion funding round, 10 2024

    Antonio Pequeño IV. Openai valued at \ 157 billion after closing \ 6.6 billion funding round, 10 2024. URL https://www.forbes.com/sites/antoniopequenoiv/2024/10/02/openai-valued-at-157-billion-after-closing-66-billion-funding-round/. Accessed on November 22, 2024

    2024

  7. [15]

    Top 10 real-life applications of large language models

    PixelPlex. Top 10 real-life applications of large language models. https://pixelplex.io/blog/llm-applications/, 2024. URL https://pixelplex.io/blog/llm-applications/. Accessed: 2024-11-22

    2024

  8. [16]

    Do imagenet classifiers generalize to imagenet? arXiv preprint arXiv:1902.10811, 2019

    Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? arXiv preprint arXiv:1902.10811, 2019. URL https://arxiv.org/abs/1902.10811

    1902 arXiv

  9. [17]

    Nlp evaluation in trouble: On the need to measure llm data contamination for each benchmark

    Oscar Sainz, Jon Campos, Iker Garc \' a-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. Nlp evaluation in trouble: On the need to measure llm data contamination for each benchmark. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp.\ ...

    2023

  10. [18]

    Language model leaderboard

    ScaleAI. Language model leaderboard. https://scale.com/leaderboard, 2024. URL https://scale.com/leaderboard. Accessed: 2024-11-22

    2024

  11. [19]

    Taori et al

    R. Taori et al. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023. URL https://github.com/tatsu-lab/stanford_alpaca. Accessed: 2024-11-22

    2023

  12. [20]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark

    Yubo Wang et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406.01574, 2024. URL https://arxiv.org/abs/2406.01574

    2024 arXiv

  13. [21]

    Chinese wall, wikipedia

    Wikipedia. Chinese wall, wikipedia. Wikipedia, The Free Encyclopedia, 2024. URL https://en.wikipedia.org/wiki/Chinese_wall#Finance. Accessed on November 22, 2024

    2024

  14. [22]

    A careful examination of large language model performance on grade school arithmetic

    Hugh Zhang et al. A careful examination of large language model performance on grade school arithmetic. arXiv preprint arXiv:2405.00332, 2024. URL https://arxiv.org/abs/2405.00332

    2024 arXiv

  15. [23]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  16. [24]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  17. [25]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  18. [26]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.