Pith. sign in

REVIEW 4 major objections 6 minor 73 references

ZeroSumEval: Scaling LLM Evaluation with Inter-Model Competition

T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ZeroSumEval claims that pitting LLMs against each other in zero-sum games produces a dynamic, saturation-resistant evaluation, and its tournament results locate a creativity gap in current models.

desk verdict Useful framework with a real creativity finding, but the PyJail headline claim overreaches and the verification design needs rethinking before the rankings are solid. read the letter →

arxiv 2504.12562 v1 pith:6OYB4UMR submitted 2025-04-17 cs.AI cs.CL

Haidar Khan , Hisham A. Alyahya , Yazeed Alnumay , M Saiful Bari , Bülent Yener This is my paper · ORCID
classification cs.AIcs.CL
keywords LLMevaluationzero-sumgamesdynamicbenchmarksmodelcompetitionBradley-Terryratingcreativityscalableverificationjailbreak
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ZeroSumEval argues that the standard way of grading large language models—fixed question sets whose answers are already known—has a shelf life: models memorize or overfit the data, and leaderboards saturate. The paper's alternative is to make evaluation a competition: two models play a zero-sum game against each other, so every run is a fresh encounter and there is no fixed answer key to game. Using seven games, from chess and poker to math-quiz creation and a Python sandbox escape contest, the authors ran more than 7,000 simulations among 13 models and rated them with Bradley-Terry scores. The headline empirical finding is that current models can play established games and answer questions, but they fail when the task is to create new, genuinely hard challenges—math questions that stump an opponent, sandboxes the opponent cannot escape, or persuasive gambits that reveal a secret. If this holds, competitive evaluation is a practical complement to static benchmarks, and it locates a specific blind spot in today's models around open-ended creativity.

What carries the argument

The load-bearing object is the zero-sum game protocol together with its scalable verification step. In the two games that require open-ended generation, MathQuiz and PyJail, the framework's manager first fixes a hidden target—an answer or a flag—and instructs the generating model to produce a challenge whose solution is that target, with the target kept out of the model's context. The generator must then solve its own challenge, matching the hidden target, as proof that the challenge is valid and solvable. This self-solution check is what lets the benchmark grow without human annotation, and it is also the mechanism that incentivizes novelty, because a memorized or trivial question is likely to be solvable by the opponent. Ratings are computed with the Bradley-Terry model, a permutation-invariant paired-comparison method that converts game outcomes into a single scale. The combination of adversarial generation, hidden-target verification, and paired-comparison rating is what carries the paper's claim that the benchmark resists saturation.

What would settle it

A direct test would be to rerun MathQuiz and PyJail with the verification step altered: have a different model, or a symbolic solver, validate the generated challenge instead of the generating model, or reveal the target to the generator during question-writing and measure whether opponents' failure rates rise. If students still fail less than ten percent of the time when the generator never had to solve its own question, the paper's creativity gap holds; if failure rates jump, the gap is an artifact of self-solution verification.

Watch

Extended reading notes

Core claim

The paper's central claim is that evaluating LLMs through zero-sum competition produces measurements that do not saturate, because the test content is generated adversarially by the models themselves rather than fixed by a benchmark provider. In the protocol, a manager picks a hidden target answer, a teacher model must invent a challenge that has that target as its solution while not seeing the target, then solve its own challenge to prove it is valid; only then does a student model face the challenge. This verification step is what makes the evaluation scalable, and the adversarial structure means a model that regurgitates memorized questions tends to lose, since the opponent is likely to know them too. Across 7 games and 13 models the authors report that frontier GPT and Claude models play chess, poker, and liar's dice competently, but all tested models struggle to generate novel and challenging questions: students answer MathQuiz questions incorrectly less than ten percent of the time, and PyJail sandboxes are broken less than two percent of the time. The authors conclude that the dynamic, competitive design gives a more trustworthy capability measurement and reveals a large, specifically creative deficit in current models.

Load-bearing premise

The assumption that carries the whole evaluation is that when a model can solve a challenge it created itself without having seen the answer, that challenge is genuinely valid and appropriately difficult—but this design pushes models toward questions they themselves can easily solve, so the measured creativity gap could partly reflect that push rather than an intrinsic limit.

Editorial extensions

If this is right

  • If the protocol scales as claimed, evaluation budgets can shift from human annotation and static question writing to tournament compute, so benchmarks can keep pace with model improvements rather than being periodically rewritten.
  • Leaderboard scores from competitive evaluation should be harder to game by training on leaked test items, since questions are generated fresh in each match and memorized material is exploitable by the opponent.
  • The measured creativity deficit, if real, gives a concrete target for training: models that can generate novel, hard challenges should rise in the ratings, making question generation a trainable capability rather than an incidental one.
  • Game traces double as interpretability data, showing where a model loses, such as failing to produce a legal chess move after ten moves, which static accuracy numbers hide.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to replace the teacher-solves-own-question verification with an external solver, such as a different model, a symbolic engine, or a human check, and compare measured difficulty; if the creativity gap shrinks or vanishes, the gap is partly an artifact of the verification design.
  • The same adversarial protocol could be pointed at reasoning robustness: models could be tasked to generate the hardest possible counterexample to an opponent's proof, turning evaluation into an automated hunt for reasoning failures.
  • Because the manager chooses the hidden target, the protocol can control difficulty in a way static benchmarks cannot; one could deliberately target questions at known weak spots and map capability boundaries per model rather than reporting a single average.
  • The near-zero PyJail escape rate may understate infiltration skill, because the defender also must solve its own sandbox, which biases generated sandboxes toward those the defender can solve; cross-model generation-attack pairings would separate defensive strength from creative attack.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces ZeroSumEval, an open-source framework for evaluating LLMs through zero-sum games, and presents experiments with 7 games, 13 models, and more than 7,000 simulations. The framework includes classic games (Chess, Poker, Liar's Dice), question-answer games (MathQuiz), conversational games (Gandalf, Debate), and a cybersecurity game (PyJail), with automated scalable verification for generated challenges. The central empirical claims are that frontier models can play common games and answer questions, but struggle to generate novel and challenging questions, and that models cannot reliably jailbreak each other. Model strengths are summarized with Bradley-Terry ratings. The authors argue that the dynamic, competitive nature of ZeroSumEval provides a more robust and trustworthy measurement than static benchmarks.

Significance. If the claims hold, ZeroSumEval is a valuable complement to static benchmarks: it is extensible, dynamic, and less prone to saturation, and the open-source release enables community adoption. The framework's main strengths are its diverse game suite, the use of rule-verified game outcomes for many games, the interpretable game traces, and the large number of simulations. However, the headline capability-gap conclusion depends critically on the scalable verification protocol, which may bias the measured creativity and jailbreak results. The lack of uncertainty quantification on the ratings and the overlap between Debate judges and contestants further limit the strength of the empirical claims. These issues are addressable with additional analysis and experiments.

major comments (4)
  1. [Section 2.2 and Figure 5C] The scalable verification protocol admits a generated challenge only if the generating model can solve it without seeing the target answer. This filters out hard-but-valid questions that the generator cannot solve, so the low student-incorrect rate in MathQuiz (<10% in Figure 5C) may reflect verification-induced easiness rather than an inability to create challenging questions. The verification protocol also incentivizes teachers to generate easy questions, since failing to solve their own question loses the game. The reported 'creativity gap' therefore conflates generation ability with self-verification ability. Please report verification-pass rates separately and re-analyze the main claim with an independent verifier (e.g., a stronger model or human) to separate these abilities.
  2. [Section 4.1] The manuscript reports that models can break their own sandboxes less than 2% of the time, which by the game rules means over 98% of defender-generated sandboxes fail scalable verification and the attacker wins by default. As a result, PyJail outcomes mostly measure the defender's ability to construct a solvable sandbox, not the attacker's ability to jailbreak. The abstract's claim that 'models cannot reliably jailbreak each other' is not supported by these outcomes. Please report attacker success conditional on a defender sandbox that passed verification, and separate generation, verification, and attack success statistics.
  3. [Section 2.4, Figure 1, and Section 4.2] The Bradley-Terry ratings and all comparative claims (e.g., that o3-mini-high performs worst, that claude-3.7-sonnet-thinking is the strongest thinking model, and that CoT helps in Chess but not MathQuiz) are reported without confidence intervals or significance tests. Many ratings in Figure 1 are within a few points of each other (e.g., MathQuiz ratings are mostly in the 990-1012 range), so the qualitative rankings may be within noise. The statement in Section 4.2 that the MathQuiz CoT difference 'was not significant' is not backed by any reported test. Please add bootstrap confidence intervals, credible intervals, or hypothesis tests for all ratings and rating differences.
  4. [Appendix A.1] In the Debate game, the jury consists of gpt-4o and claude-3.5-sonnet, while gpt-4o is also a contestant in the tournament. This means some Debate outcomes are judged by a model that may have an incentive or bias toward its own outputs, which is a form of self-assessment that should be explicitly addressed. Please report robustness of the Debate ratings when the judging model is not a contestant, or provide evidence that the jury's own identity does not affect the scores.
minor comments (6)
  1. [Section 5.1] There is a typo in 'Critisisms' in the first sentence of Section 5.1; it should be 'Criticisms'.
  2. [Appendix A.1] In the Liar's Dice description, 'foud' should be 'found'.
  3. [Throughout] The naming of the PyJail game is inconsistent ('PyJail', 'Pyjail', 'pyjail'); please standardize.
  4. [Section 4.2] The text says 'llama3.3-8b as the worst' in the Llama family, but the model table lists llama3.1-8b, not llama3.3-8b; this appears to be a typo.
  5. [Section 3.1] The paper states '500-1500 games per experiment' and also reports '>7000 simulations across 7 games'; please clarify how these counts relate, since 13 models in a round-robin with multiple games could produce a larger total.
  6. [Figures 6a and 6b] The bar charts in Figure 6 would be easier to read with explicit confidence intervals or error bars, and the 'Rating Difference' axis labels should specify whether the values are raw rating deltas or bootstrap estimates.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: ZeroSumEval's ratings are computed from game outcomes, and the scalable-verification step, while a validity limitation, does not make the reported results equal to their inputs.

full rationale

The paper's central results are Bradley-Terry ratings and outcome rates derived from deterministic game rules (legal chess moves, poker chip counts, exact math-answer matches, passphrase disclosure in Gandalf), not from fitted parameters or from the ratings themselves. The scalable-verification protocol in Section 2.2 requires a teacher/defender to solve its own generated challenge before the game proceeds; this creates a real selection effect and a construct-validity concern for MathQuiz and PyJail, but it is not a circular reduction. The reported 'student incorrect <10%' rate is a measured outcome on a different model than the generator, and the PyJail observation that defenders 'frequently cannot break the sandboxes they generate themselves' is explicitly about the verification step, not a restatement of the attacker-jailbreak conclusion. The abstract's 'models cannot reliably jailbreak each other' is supported mainly by the Gandalf game, which is not defined in terms of the verification protocol. The self-citation to Alzahrani et al. (2024) is used only to motivate prompt sensitivity and is not load-bearing for the benchmark's own claims. The Debate jury includes gpt-4o, which is also a contestant, but that is a mild adjudication bias rather than a case where a predicted quantity is equivalent to an input by construction. Overall, the derivation chain is self-contained: outcomes are observed from simulations, ratings are computed from those outcomes, and no claim reduces to its own inputs.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claims rest on evaluation design choices (retry limits, target range, judge identity) and on the verification assumption that self-solving proves validity. These are reasonable but untested choices, not fitted parameters. No new physical or conceptual entities are introduced.

free parameters (3)
  • max_player_attempts = 3 or 5 depending on game
    Hand-chosen retry limits (3 for Debate/Gandalf/Liar's Dice, 5 for Chess/MathQuiz/PyJail/Poker) affect how easily models can recover from invalid moves, influencing measured capability.
  • MathQuiz target range = (0, 1000]
    Random integer target constrains question difficulty and gives a 1/1000 per-attempt chance of guessing the target, used as a verification threshold.
  • Debate jury models = gpt-4o and claude-3.5-sonnet
    The choice and identity of the LLM jury define Debate outcomes; gpt-4o also competes, a potential self-bias.
assumptions (3)
  • domain assumption A teacher successfully answering its own generated question (without seeing the target) proves the question is valid and solvable.
    Used in Section 2.2 to replace human verification; if models favor questions they can solve, the creativity measurement is biased.
  • standard math Bradley-Terry model assumes a fixed win probability for each model pair and that ratings maximize the likelihood of observed game outcomes.
    Section 2.4 relies on the standard BT model, which may not hold if model behavior drifts across games or contexts.
  • domain assumption The finite tournament game count is sufficient for stable ratings.
    Section 3.1 reports 500-1500 games per experiment but no confidence intervals, so rating stability is assumed rather than demonstrated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ZeroSumEval: Scaling LLM Evaluation with Inter-Model Competition." pith.science (2026). https://pith.science/paper/6OYB4UMR

@misc{pith2026250412562,
  author       = {Pith},
  title        = {Pith review of: ZeroSumEval: Scaling LLM Evaluation with Inter-Model Competition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6OYB4UMR}},
  note         = {Machine review of arXiv:2504.12562}
}
read the original abstract

Evaluating the capabilities of Large Language Models (LLMs) has traditionally relied on static benchmark datasets, human assessments, or model-based evaluations - methods that often suffer from overfitting, high costs, and biases. ZeroSumEval is a novel competition-based evaluation protocol that leverages zero-sum games to assess LLMs with dynamic benchmarks that resist saturation. ZeroSumEval encompasses a diverse suite of games, including security challenges (PyJail), classic games (Chess, Liar's Dice, Poker), knowledge tests (MathQuiz), and persuasion challenges (Gandalf, Debate). These games are designed to evaluate a range of AI capabilities such as strategic reasoning, planning, knowledge application, and creativity. Building upon recent studies that highlight the effectiveness of game-based evaluations for LLMs, ZeroSumEval enhances these approaches by providing a standardized and extensible framework. To demonstrate this, we conduct extensive experiments with >7000 simulations across 7 games and 13 models. Our results show that while frontier models from the GPT and Claude families can play common games and answer questions, they struggle to play games that require creating novel and challenging questions. We also observe that models cannot reliably jailbreak each other and fail generally at tasks requiring creativity. We release our code at https://github.com/facebookresearch/ZeroSumEval.

Figures

Figures reproduced from arXiv: 2504.12562 by the authors.

Figure 1
Figure 1. Cumulative ratings of 13 models on ZEROSUMEVAL. The top perform￾ing models (gpt-4o and claude-3.7-sonnet) show mostly on-par performance across ZEROSUMEVAL games. Thinking model quality varies between model families (e.g. claude-3.7-sonnet-thinking vs deepseek-r1). Surprisingly, o3-mini-high performs worst amongst this cohort of models. evaluation. Training these models involves massive computational efforts (Hoffma… view at source ↗
Figure 2
Figure 2. Example of a chess game trace. Here deepseek-chat (white) executes a knight fork against llama3.3-70b (black) after which black loses the game by failing to produce a legal move. Classic Games (Chess, Poker, Liar’s Dice) Classic board games like chess serve as a benchmark for strategic reasoning and long-term planning. They require models to engage in multi-step thinking, manage trade-offs, and foresee opponent move… view at source ↗
Figure 3
Figure 3. Example of a MathQuiz game trace. At this point in the game, [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: State diagram of the verification pro￾cess involving the ZSEval Manager and the LLM Generator. Blue boxes indicate deter￾ministic steps and green boxes indicate steps involving the LLM. We leverage the DSPy (Khattab et al., 2023) approach to prompt abstraction in our f…
Figure 5
Figure 5. Figure 5: Summarized outcomes from four games (A) Chess, (B) Gandalf, (C) MathQuiz, [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Rating difference between model types. • Reasoning: undergraduate level academic questions are tested via MMLU (Hendrycks et al., 2020), while GPQA (Rein et al., 2023) tests models with graduate level questions. • Mathematics: GSM8K (Cobbe et al., 2021) evaluates model…
Figure 7
Figure 7. Figure 7: Violin plots showing the distribution of moves per Chess game. Some models can [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

73 extracted references · 15 canonical work pages

  1. [1]

    When benchmarks are targets: Revealing the sensitivity of large language model leaderboards

    Norah Alzahrani, Hisham Abdullah Alyahya, Yazeed Alnumay, Sultan Alrashed, Shaykhah Alsubaie, Yusef Almushaykeh, Faisal Mirza, Nouf Alotaibi, Nora Altwairesh, Areeb Alowisheq, et al. When benchmarks are targets: Revealing the sensitivity of large language model leaderboards. arXiv preprint arXiv:2402.01781, 2024

    arXiv 2024

  2. [2]

    Claude 3.5 Sonnet

    Anthropic. Claude 3.5 Sonnet . https://www.anthropic.com/news/claude-3-5-sonnet, 2024. Accessed: 2024-09-17

    work page 2024

  3. [3]

    Open llm leaderboard (2023-2024)

    Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. Open llm leaderboard (2023-2024). https://huggingface.co/spaces/open-llm-leaderboard-old/open_llm_leaderboard, 2023

    work page 2023

  4. [4]

    Elo uncovered: Robustness and best practices in language model evaluation, 2023

    Meriem Boubdir, Edward Kim, Beyza Ermis, Sara Hooker, and Marzieh Fadaee. Elo uncovered: Robustness and best practices in language model evaluation, 2023. URL https://arxiv.org/abs/2311.17295

    arXiv 2023

  5. [5]

    Rank analysis of incomplete block designs: I

    Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39 0 (3/4): 0 324--345, 1952

    1952

  6. [6]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

    arXiv 2020

  7. [7]

    Langchain

    Harrison Chase. Langchain. https://github.com/langchain-ai/langchain, 2023

    work page 2023

  8. [8]

    Humans or llms as the judge? a study on judgement biases, 2024 a

    Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. Humans or llms as the judge? a study on judgement biases, 2024 a . URL https://arxiv.org/abs/2402.10669

    arXiv 2024

Show all 73 references
  1. [9]

    Chatgpt's one-year anniversary: Are open-source large language models catching up?, 2024 b

    Hailin Chen, Fangkai Jiao, Xingxuan Li, Chengwei Qin, Mathieu Ravaut, Ruochen Zhao, Caiming Xiong, and Shafiq Joty. Chatgpt's one-year anniversary: Are open-source large language models catching up?, 2024 b . URL https://arxiv.org/abs/2311.16989

    2024 arXiv

  2. [10]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott G...

    2021 arXiv

  3. [11]

    Gonzalez, and Ion Stoica

    Wei-Lin Chiang, Tim Li, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: New models & elo system update, Dec 2023. URL https://lmsys.org/blog/2023-12-07-leaderboard/

    2023

  4. [12]

    Gonzalez, and Ion Stoica

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating llms by human preference, 2024. URL https://arxiv.org/a...

    2024 arXiv

  5. [13]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

  6. [14]

    Gamebench: Evaluating strategic reasoning abilities of llm agents, 2024

    Anthony Costarelli, Mat Allen, Roman Hauksson, Grace Sodunke, Suhas Hariharan, Carlson Cheng, Wenjie Li, Joshua Clymer, and Arjun Yadav. Gamebench: Evaluating strategic reasoning abilities of llm agents, 2024. URL https://arxiv.org/abs/2406.06613

    2024 arXiv

  7. [15]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zha...

    2024 arXiv

  8. [16]

    Gonzalez, and Wei-Lin Chiang

    Lisa Dunlap, Evan Frick, Tianle Li, Isaac Ong, Joseph E. Gonzalez, and Wei-Lin Chiang. What’s up with llama 3? arena data analysis, May 2024. URL https://lmsys.org/blog/2024-05-08-llama3/

    2024

  9. [17]

    The proposed uscf rating system, its development, theory, and applications

    Arpad E Elo. The proposed uscf rating system, its development, theory, and applications. Chess life, 22 0 (8): 0 242--247, 1967

    1967

  10. [18]

    Open llm leaderboard v2

    Clémentine Fourrier, Nathan Habib, Alina Lozovskaya, Konrad Szafer, and Thomas Wolf. Open llm leaderboard v2. https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard, 2024

    2024

  11. [19]

    Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack...

    2024 arXiv

  12. [20]

    The false promise of imitating proprietary llms, 2023

    Arnav Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song. The false promise of imitating proprietary llms, 2023. URL https://arxiv.org/abs/2305.15717

    2023 arXiv

  13. [21]

    Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. Annotation artifacts in natural language inference data. In Marilyn Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North A merican Chapter of ...

    2018 doi

  14. [22]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020

    2009 arXiv

  15. [23]

    Measuring coding challenge competence with apps

    Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with apps. NeurIPS, 2021 a

    2021

  16. [24]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021 b

    2021

  17. [25]

    Rae, Oriol Vinyals, and Laurent Sifre

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osin...

    2022 arXiv

  18. [26]

    The dataset nutrition label: A framework to drive higher data quality standards, 2018

    Sarah Holland, Ahmed Hosny, Sarah Newman, Joshua Joseph, and Kasia Chmielinski. The dataset nutrition label: A framework to drive higher data quality standards, 2018. URL https://arxiv.org/abs/1805.03677

    2018 arXiv

  19. [27]

    Towards accountability for machine learning datasets: Practices from software engineering and infrastructure, 2021

    Ben Hutchinson, Andrew Smart, Alex Hanna, Emily Denton, Christina Greer, Oddur Kjartansson, Parker Barnes, and Margaret Mitchell. Towards accountability for machine learning datasets: Practices from software engineering and infrastructure, 2021. URL https://arxiv.org/abs/2010.13561

    2021 arXiv

  20. [28]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020. URL https://arxiv.org/abs/2001.08361

    2020 arXiv

  21. [29]

    Building meta's genai infrastructure

    Mathew Oldham Kevin Lee, Adi Gangidi. Building meta's genai infrastructure. https://engineering.fb.com/2024/03/12/data-center-engineering/building-metas-genai-infrastructure/, 2024. Accessed: September 28, 2024

    2024

  22. [30]

    Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts

    Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. Dspy: Compiling declarative language model calls into self-impro...

    arXiv

  23. [31]

    Dynabench: Rethinking benchmarking in nlp, 2021

    Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adi...

    2021 arXiv

  24. [32]

    Microsoft, brookfield to develop more than 10.5 gigawatts of renewable energy

    Spencer Kimball. Microsoft, brookfield to develop more than 10.5 gigawatts of renewable energy. https://www.cnbc.com/2024/05/01/microsoft-brookfield-to-develop-more-than-10point5-gigawatts-of-renewable-energy.html, 2024. Accessed: September 28, 2024

    2024

  25. [33]

    A systematic study and comprehensive evaluation of chatgpt on benchmark datasets, 2023

    Md Tahmid Rahman Laskar, M Saiful Bari, Mizanur Rahman, Md Amran Hossen Bhuiyan, Shafiq Joty, and Jimmy Xiangji Huang. A systematic study and comprehensive evaluation of chatgpt on benchmark datasets, 2023

    2023

  26. [34]

    A systematic survey and critical review on evaluating large language models: Challenges, limitations, and recommendations

    Md Tahmid Rahman Laskar, Sawsan Alqahtani, M Saiful Bari, Mizanur Rahman, Mohammad Abdullah Matin Khan, Haidar Khan, Israt Jahan, Amran Bhuiyan, Chee Wei Tan, Md Rizwan Parvez, et al. A systematic survey and critical review on evaluating large language models: Challenges, limi...

    2024 arXiv

  27. [35]

    Does style matter? disentangling style and substance in chatbot arena, Aug 2024

    Tianle Li, Anastasios Angelopoulos, and Wei-Lin Chiang. Does style matter? disentangling style and substance in chatbot arena, Aug 2024. URL https://lmsys.org/blog/2024-08-28-style-control/

    2024

  28. [36]

    Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity

    Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual M...

    2022 doi

  29. [37]

    A survey on bias and fairness in machine learning

    Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey on bias and fairness in machine learning. ACM Comput. Surv., 54 0 (6), July 2021. ISSN 0360-0300. doi:10.1145/3457607. URL https://doi.org/10.1145/3457607

    2021 doi

  30. [38]

    Google launches a 9 exaflop cluster of cloud tpu v4 pods into public preview

    Ron Miller. Google launches a 9 exaflop cluster of cloud tpu v4 pods into public preview. https://techcrunch.com/2022/05/11/google-launches-a-9-exaflop-cluster-of-cloud-tpu-v4-pods-into-public-preview/, 2022. Accessed: [Your Access Date]

    2022

  31. [39]

    Openai five defeats dota 2 world champions

    OpenAI. Openai five defeats dota 2 world champions. https://openai.com/blog/openai-five-defeats-dota-2-world-champions/, April 2019. Accessed: 2024-09-28

    2019

  32. [40]

    Solving rubik's cube with a robot hand, 2019

    OpenAI, Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, Jonas Schneider, Nikolas Tezak, Jerry Tworek, Peter Welinder, Lilian Weng, Qiming Yuan, Wojciech Zaremba, and Lei Zh...

    2019 arXiv

  33. [41]

    Generative agents: Interactive simulacra of human behavior

    Joon Sung Park, Joseph O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology, pp.\ 1--22, 2023

    2023

  34. [42]

    Disentangling length from quality in direct preference optimization, 2024

    Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from quality in direct preference optimization, 2024. URL https://arxiv.org/abs/2403.19159

    2024 arXiv

  35. [43]

    Large language models sensitivity to the order of options in multiple-choice questions, 2023

    Pouya Pezeshkpour and Estevam Hruschka. Large language models sensitivity to the order of options in multiple-choice questions, 2023. URL https://arxiv.org/abs/2308.11483

    2023 arXiv

  36. [44]

    Gandalf the red: Adaptive security for llms

    Niklas Pfister, Václav Volhejn, Manuel Knott, Santiago Arias, Julia Bazińska, Mykhailo Bichurin, Alan Commike, Janet Darling, Peter Dienes, Matthew Fiedler, David Haber, Matthias Kraft, Marco Lancini, Max Mathys, Damián Pascual-Ortiz, Jakub Podolak, Adrià Romero-López, Kyriaco...

    2025 arXiv

  37. [45]

    Is chatgpt a general-purpose natural language processing task solver? arXiv preprint arXiv:2302.06476, 2023

    Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. Is chatgpt a general-purpose natural language processing task solver? arXiv preprint arXiv:2302.06476, 2023

    2023 arXiv

  38. [46]

    Gpqa: A graduate-level google-proof q&a benchmark

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022, 2023

    2023 arXiv

  39. [47]

    Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Ni...

    2022 arXiv

  40. [48]

    Shah, and Regina Barzilay

    Tal Schuster, Roei Schuster, Darsh J. Shah, and Regina Barzilay. The limitations of stylometry for detecting machine-generated fake news. Computational Linguistics, 46 0 (2): 0 499--510, June 2020. doi:10.1162/coli_a_00380. URL https://aclanthology.org/2020.cl-2.8

    2020 doi

  41. [49]

    Mastering the game of go with deep neural networks and tree search

    David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529 0 (7587): 0 484-...

    2016

  42. [50]

    Mastering chess and shogi by self-play with a general reinforcement learning algorithm, 2017

    David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. Mastering chess and shogi by self-play with a general reinforce...

    2017 arXiv

  43. [51]

    Dspy assertions: Computational constraints for self-refining language model pipelines, 2024

    Arnav Singhvi, Manish Shetty, Shangyin Tan, Christopher Potts, Koushik Sen, Matei Zaharia, and Omar Khattab. Dspy assertions: Computational constraints for self-refining language model pipelines, 2024. URL https://arxiv.org/abs/2312.13382

    2024 arXiv

  44. [52]

    Evaluating large language models with grid-based game competitions: An extensible llm benchmark and leaderboard, 2024

    Oguzhan Topsakal, Colby Jacob Edell, and Jackson Bailey Harper. Evaluating large language models with grid-based game competitions: An extensible llm benchmark and leaderboard, 2024. URL https://arxiv.org/abs/2407.07796

    2024 arXiv

  45. [53]

    Freshllms: Refreshing large language models with search engine augmentation

    Tu Vu, Mohit Iyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denny Zhou, Quoc Le, et al. Freshllms: Refreshing large language models with search engine augmentation. arXiv preprint arXiv:2310.03214, 2023

    arXiv

  46. [54]

    Glue: A multi-task benchmark and analysis platform for natural language understanding

    Alex Wang. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018

    2018 arXiv

  47. [55]

    Superglue: A stickier benchmark for general-purpose language understanding systems

    Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems, 32, 2019

    2019

  48. [56]

    Voyager: An open-ended embodied agent with large language models, 2023

    Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models, 2023. URL https://arxiv.org/abs/2305.16291

    2023 arXiv

  49. [57]

    Beyond the answers: Reviewing the rationality of multiple choice question answering for the evaluation of large language models, 2024 a

    Haochun Wang, Sendong Zhao, Zewen Qiang, Nuwa Xi, Bing Qin, and Ting Liu. Beyond the answers: Reviewing the rationality of multiple choice question answering for the evaluation of large language models, 2024 a . URL https://arxiv.org/abs/2402.01349

    2024 arXiv

  50. [58]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406.01574, 2024 b

    2024 arXiv

  51. [59]

    Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning, Singapore, December 2023

    Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mosquera, Bhargavi Paranjabe, Adina Williams, Tal Linzen, and Ryan Cotterell (eds.). Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learn...

    2023

  52. [60]

    Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models, 2...

    2022 arXiv

  53. [61]

    Jason Wei, Najoung Kim, Yi Tay, and Quoc V. Le. Inverse scaling can become u-shaped, 2023 a . URL https://arxiv.org/abs/2211.02011

    2023 arXiv

  54. [62]

    Chain-of-thought prompting elicits reasoning in large language models, 2023 b

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023 b . URL https://arxiv.org/abs/2201.11903

    2023 arXiv

  55. [63]

    Lew, Noah D

    Lionel Wong, Gabriel Grand, Alexander K. Lew, Noah D. Goodman, Vikash K. Mansinghka, Jacob Andreas, and Joshua B. Tenenbaum. From word models to world models: Translating from natural language to the probabilistic language of thought, 2023. URL https://arxiv.org/abs/2306.12672

    2023 arXiv

  56. [64]

    Smartplay : A benchmark for LLM s as intelligent agents

    Yue Wu, Xuan Tang, Tom Mitchell, and Yuanzhi Li. Smartplay : A benchmark for LLM s as intelligent agents. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=S2oTVrlcp3

    2024

  57. [65]

    Chatarena: Multi-agent language game environments for large language models

    Yuxiang Wu, Zhengyao Jiang, Akbir Khan, Yao Fu, Laura Ruis, Edward Grefenstette, and Tim Rocktäschel. Chatarena: Multi-agent language game environments for large language models. https://github.com/chatarena/chatarena, 2023

    2023

  58. [66]

    Gonzalez, and Ion Stoica

    Shuo Yang, Wei-Lin Chiang, Lianmin Zheng, Joseph E. Gonzalez, and Ion Stoica. Rethinking benchmark and contamination for language models with rephrased samples, 2023

    2023

  59. [67]

    A survey of large language models, 2024

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong W...

    2024 arXiv

  60. [68]

    Large language models are not robust multiple choice selectors, 2024

    Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. Large language models are not robust multiple choice selectors, 2024. URL https://arxiv.org/abs/2309.03882

    2024 arXiv

  61. [69]

    Parkes, and Richard Socher

    Stephan Zheng, Alexander Trott, Sunil Srinivasa, David C. Parkes, and Richard Socher. The ai economist: Optimal economic policy design via two-level deep reinforcement learning, 2021. URL https://arxiv.org/abs/2108.02755

    2021 arXiv

  62. [70]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  63. [71]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  64. [72]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  65. [73]

    Find the smallest prime number greater than 400 that leaves a remainder of 1 when divided by 6 and a remainder of 5 when divided by 7

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.