{"id":"fd9d3cce-b9f0-4a62-a587-4d40059d8f26","arxiv_id":"2504.12562","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"ZeroSumEval ranks 13 LLMs through over 7,000 head-to-head games and finds models fall short at creative generation and jailbreaking.","lead":"This paper introduces ZeroSumEval, a framework that evaluates AI models by making them compete in zero-sum games like chess, poker, math quizzes, and security challenges. It finds that today's models can play familiar games but struggle to create novel, challenging questions or jailbreak each other.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Scalable-verification protocol admits only questions the generator can solve, so the reported creativity/jailbreak gap may be an artifact; PyJail outcomes are dominated by verification failures.","rationale":"The reader's weakest assumption is the same verification confound: a teacher solving its own question does not prove the question is challenging. My stress-test sharpens this into a concrete logical flaw in PyJail, where a self-verification success rate below 2% means the game outcome is mostly a default win for the attacker after the defender fails to create a solvable sandbox. The reported finding that 'models cannot reliably jailbreak each other' is thus not measured by the games as played; the intended capability is rarely exercised. This is a load-bearing concern because the paper's main qualitative contribution is the observed creativity deficit, and the evidence for it comes disproportionately from MathQuiz and PyJail, the two games that rely on self-verification. I do not recommend rejecting the paper outright: the framework, released code, and the objective games (Chess, Poker, Liar's Dice) are useful and the qualitative ordering of frontier models is plausible. However, the central claim of a trustworthy capability measurement requires either an independent verification method or a re-analysis that separates verification failures from genuine attack/generation outcomes. The reader's CONDITIONAL verdict adequately captures this, so I leave the verdict unchanged.","tokens_in":17837,"tokens_out":6124,"duration_ms":69369,"concrete_test":"Re-run the MathQuiz and PyJail protocols with an independent verifier (a different frontier model, e.g., gpt-4o verifying claude-3.7-sonnet questions and vice versa) replacing the generator's self-answer step, while holding prompts, retry counts, and opponent pairings fixed. Compare verification-pass rates and opponent solve rates. If opponent solve rates change materially, or if PyJail attacker solve rates computed only from verification-passed games are high, the reported creativity/jailbreak gap is an artifact of self-verification rather than a model limitation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing assumption is the scalable-verification protocol in Section 2.2: a teacher-generated challenge is admitted only if the same teacher can solve it without seeing the target. This verifies solvability by the generator, not difficulty. In MathQuiz, the verification step filters out hard-but-valid questions the generator cannot solve, so the low student-incorrect rate (<10%, Figure 5C) may reflect verification-induced easiness rather than an inability to create challenging problems. In PyJail the issue is more direct: Section 4.1 reports that models can break their own sandboxes less than 2% of the time, meaning over 98% of defender-generated sandboxes fail verification. The attacker then wins by default, and the intended jailbreak interaction rarely occurs. The abstract's claim that 'models cannot reliably jailbreak each other' is therefore not supported by PyJail outcomes; the game mostly measures whether a defender can construct a solvable sandbox, not whether an attacker can escape it. The headline 'creativity gap' conflates generation ability, self-verification ability, and attack ability, so the central claim that ZeroSumEval reveals a robust capability gap depends on an unvalidated identification of these distinct abilities.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ZeroSumEval, an open-source framework for evaluating LLMs through zero-sum games, and presents experiments with 7 games, 13 models, and more than 7,000 simulations. The framework includes classic games (Chess, Poker, Liar's Dice), question-answer games (MathQuiz), conversational games (Gandalf, Debate), and a cybersecurity game (PyJail), with automated scalable verification for generated challenges. The central empirical claims are that frontier models can play common games and answer questions, but struggle to generate novel and challenging questions, and that models cannot reliably jailbreak each other. Model strengths are summarized with Bradley-Terry ratings. The authors argue that the dynamic, competitive nature of ZeroSumEval provides a more robust and trustworthy measurement than static benchmarks.","tokens_in":18103,"tokens_out":3896,"duration_ms":40009,"significance":"If the claims hold, ZeroSumEval is a valuable complement to static benchmarks: it is extensible, dynamic, and less prone to saturation, and the open-source release enables community adoption. The framework's main strengths are its diverse game suite, the use of rule-verified game outcomes for many games, the interpretable game traces, and the large number of simulations. However, the headline capability-gap conclusion depends critically on the scalable verification protocol, which may bias the measured creativity and jailbreak results. The lack of uncertainty quantification on the ratings and the overlap between Debate judges and contestants further limit the strength of the empirical claims. These issues are addressable with additional analysis and experiments.","major_comments":[{"comment":"The scalable verification protocol admits a generated challenge only if the generating model can solve it without seeing the target answer. This filters out hard-but-valid questions that the generator cannot solve, so the low student-incorrect rate in MathQuiz (<10% in Figure 5C) may reflect verification-induced easiness rather than an inability to create challenging questions. The verification protocol also incentivizes teachers to generate easy questions, since failing to solve their own question loses the game. The reported 'creativity gap' therefore conflates generation ability with self-verification ability. Please report verification-pass rates separately and re-analyze the main claim with an independent verifier (e.g., a stronger model or human) to separate these abilities.","section":"Section 2.2 and Figure 5C"},{"comment":"The manuscript reports that models can break their own sandboxes less than 2% of the time, which by the game rules means over 98% of defender-generated sandboxes fail scalable verification and the attacker wins by default. As a result, PyJail outcomes mostly measure the defender's ability to construct a solvable sandbox, not the attacker's ability to jailbreak. The abstract's claim that 'models cannot reliably jailbreak each other' is not supported by these outcomes. Please report attacker success conditional on a defender sandbox that passed verification, and separate generation, verification, and attack success statistics.","section":"Section 4.1"},{"comment":"The Bradley-Terry ratings and all comparative claims (e.g., that o3-mini-high performs worst, that claude-3.7-sonnet-thinking is the strongest thinking model, and that CoT helps in Chess but not MathQuiz) are reported without confidence intervals or significance tests. Many ratings in Figure 1 are within a few points of each other (e.g., MathQuiz ratings are mostly in the 990-1012 range), so the qualitative rankings may be within noise. The statement in Section 4.2 that the MathQuiz CoT difference 'was not significant' is not backed by any reported test. Please add bootstrap confidence intervals, credible intervals, or hypothesis tests for all ratings and rating differences.","section":"Section 2.4, Figure 1, and Section 4.2"},{"comment":"In the Debate game, the jury consists of gpt-4o and claude-3.5-sonnet, while gpt-4o is also a contestant in the tournament. This means some Debate outcomes are judged by a model that may have an incentive or bias toward its own outputs, which is a form of self-assessment that should be explicitly addressed. Please report robustness of the Debate ratings when the judging model is not a contestant, or provide evidence that the jury's own identity does not affect the scores.","section":"Appendix A.1"}],"minor_comments":[{"comment":"There is a typo in 'Critisisms' in the first sentence of Section 5.1; it should be 'Criticisms'.","section":"Section 5.1"},{"comment":"In the Liar's Dice description, 'foud' should be 'found'.","section":"Appendix A.1"},{"comment":"The naming of the PyJail game is inconsistent ('PyJail', 'Pyjail', 'pyjail'); please standardize.","section":"Throughout"},{"comment":"The text says 'llama3.3-8b as the worst' in the Llama family, but the model table lists llama3.1-8b, not llama3.3-8b; this appears to be a typo.","section":"Section 4.2"},{"comment":"The paper states '500-1500 games per experiment' and also reports '>7000 simulations across 7 games'; please clarify how these counts relate, since 13 models in a round-robin with multiple games could produce a larger total.","section":"Section 3.1"},{"comment":"The bar charts in Figure 6 would be easier to read with explicit confidence intervals or error bars, and the 'Rating Difference' axis labels should specify whether the values are raw rating deltas or bootstrap estimates.","section":"Figures 6a and 6b"}],"recommendation":"major_revision","confidential_remarks":"This is a useful systems contribution with an open-source release, but the central capability-gap and jailbreak claims rest on the scalable verification protocol, which appears to confound generation ability with self-verification ability. The requested re-analysis and uncertainty quantification are feasible within the manuscript's scope, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"ZeroSumEval is a genuinely useful contribution. It packages game-based evaluation into an open, extensible framework with released code, runs more than 7,000 simulations across 7 games and 13 models, and produces a clear qualitative finding: frontier models can play chess and poker and answer math questions, but fail to generate challenges that stump an opponent. The MathQuiz result is notable — students answer correctly over 90% of the time across all models. That is a meaningful observation about current creative limits.\n\nThe soft spots are real but fixable. The scalable verification protocol in Section 2.2 admits only questions the generator itself can solve. For MathQuiz, that is a conservative bias: even when the teacher restricts to questions it can solve, those questions still are not hard for opponents. That actually strengthens the 'can't make hard questions' finding, though it muddies the interpretation of a 'creativity gap' distinct from solving ability. For PyJail, the problem is more serious. The paper reports that defenders can break their own sandboxes less than 2% of the time. Under the game rules, that means over 98% of defender-generated sandboxes fail verification and the attacker wins by default. So the game mostly measures sandbox construction, not jailbreak ability. The abstract's claim that 'models cannot reliably jailbreak each other' is not supported by the data; the accurate conclusion is that models cannot reliably create valid jailbreak challenges. That is a different, less dramatic statement. The authors should either verify sandboxes with an independent solver or report the two failure modes separately.\n\nOther concerns: Bradley-Terry ratings come without confidence intervals or significance tests, so several model comparisons (CoT vs predict, thinking vs CoT) are likely within noise. The Debate game uses gpt-4o as a judge while gpt-4o is also a contestant, an avoidable bias. These are minor for a framework paper but should be cleaned up.\n\nWho this is for: anyone building or using LLM benchmarks. The release is concrete and reproducible, which counts for a lot. It deserves a serious referee; the framework is a contribution even if the headline claims need tempering. If I were handling it, I would send to review and ask for a revision that fixes PyJail and adds uncertainty metrics.","headline":"Useful framework with a real creativity finding, but the PyJail headline claim overreaches and the verification design needs rethinking before the rankings are solid.","tokens_in":18608,"tokens_out":5443,"would_cite":true,"duration_ms":51456,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ZeroSumEval claims that pitting LLMs against each other in zero-sum games produces a dynamic, saturation-resistant evaluation, and its tournament results locate a creativity gap in current models.","keywords":["LLM evaluation","zero-sum games","dynamic benchmarks","model competition","Bradley-Terry rating","creativity evaluation","scalable verification","jailbreak"],"falsifier":"A direct test would be to rerun MathQuiz and PyJail with the verification step altered: have a different model, or a symbolic solver, validate the generated challenge instead of the generating model, or reveal the target to the generator during question-writing and measure whether opponents' failure rates rise. If students still fail less than ten percent of the time when the generator never had to solve its own question, the paper's creativity gap holds; if failure rates jump, the gap is an artifact of self-solution verification.","tokens_in":17676,"feed_emoji":"🎲","tokens_out":8358,"duration_ms":80923,"temperature":0.7,"pith_summary":"ZeroSumEval argues that the standard way of grading large language models—fixed question sets whose answers are already known—has a shelf life: models memorize or overfit the data, and leaderboards saturate. The paper's alternative is to make evaluation a competition: two models play a zero-sum game against each other, so every run is a fresh encounter and there is no fixed answer key to game. Using seven games, from chess and poker to math-quiz creation and a Python sandbox escape contest, the authors ran more than 7,000 simulations among 13 models and rated them with Bradley-Terry scores. The headline empirical finding is that current models can play established games and answer questions, but they fail when the task is to create new, genuinely hard challenges—math questions that stump an opponent, sandboxes the opponent cannot escape, or persuasive gambits that reveal a secret. If this holds, competitive evaluation is a practical complement to static benchmarks, and it locates a specific blind spot in today's models around open-ended creativity.","feed_headline":"Pitting AI models against each other exposes a creativity gap","feed_subtitle":"In 7,000 head-to-head matches across 7 games, models answer questions but fail to generate novel, hard ones.","key_machinery":"The load-bearing object is the zero-sum game protocol together with its scalable verification step. In the two games that require open-ended generation, MathQuiz and PyJail, the framework's manager first fixes a hidden target—an answer or a flag—and instructs the generating model to produce a challenge whose solution is that target, with the target kept out of the model's context. The generator must then solve its own challenge, matching the hidden target, as proof that the challenge is valid and solvable. This self-solution check is what lets the benchmark grow without human annotation, and it is also the mechanism that incentivizes novelty, because a memorized or trivial question is likely to be solvable by the opponent. Ratings are computed with the Bradley-Terry model, a permutation-invariant paired-comparison method that converts game outcomes into a single scale. The combination of adversarial generation, hidden-target verification, and paired-comparison rating is what carries the paper's claim that the benchmark resists saturation.","core_discovery":"The paper's central claim is that evaluating LLMs through zero-sum competition produces measurements that do not saturate, because the test content is generated adversarially by the models themselves rather than fixed by a benchmark provider. In the protocol, a manager picks a hidden target answer, a teacher model must invent a challenge that has that target as its solution while not seeing the target, then solve its own challenge to prove it is valid; only then does a student model face the challenge. This verification step is what makes the evaluation scalable, and the adversarial structure means a model that regurgitates memorized questions tends to lose, since the opponent is likely to know them too. Across 7 games and 13 models the authors report that frontier GPT and Claude models play chess, poker, and liar's dice competently, but all tested models struggle to generate novel and challenging questions: students answer MathQuiz questions incorrectly less than ten percent of the time, and PyJail sandboxes are broken less than two percent of the time. The authors conclude that the dynamic, competitive design gives a more trustworthy capability measurement and reveals a large, specifically creative deficit in current models.","pith_inferences":["A testable extension is to replace the teacher-solves-own-question verification with an external solver, such as a different model, a symbolic engine, or a human check, and compare measured difficulty; if the creativity gap shrinks or vanishes, the gap is partly an artifact of the verification design.","The same adversarial protocol could be pointed at reasoning robustness: models could be tasked to generate the hardest possible counterexample to an opponent's proof, turning evaluation into an automated hunt for reasoning failures.","Because the manager chooses the hidden target, the protocol can control difficulty in a way static benchmarks cannot; one could deliberately target questions at known weak spots and map capability boundaries per model rather than reporting a single average.","The near-zero PyJail escape rate may understate infiltration skill, because the defender also must solve its own sandbox, which biases generated sandboxes toward those the defender can solve; cross-model generation-attack pairings would separate defensive strength from creative attack."],"forward_implications":["If the protocol scales as claimed, evaluation budgets can shift from human annotation and static question writing to tournament compute, so benchmarks can keep pace with model improvements rather than being periodically rewritten.","Leaderboard scores from competitive evaluation should be harder to game by training on leaked test items, since questions are generated fresh in each match and memorized material is exploitable by the opponent.","The measured creativity deficit, if real, gives a concrete target for training: models that can generate novel, hard challenges should rise in the ratings, making question generation a trainable capability rather than an incidental one.","Game traces double as interpretability data, showing where a model loses, such as failing to produce a legal chess move after ten moves, which static accuracy numbers hide."],"supporting_citations":[{"why":"It established the multi-agent game-arena approach to LLM evaluation that this work builds on and extends.","marker":"(Wu et al., 2023)"},{"why":"It supplied the broadest prior game-based battery, which the paper compares against on flexibility and scope.","marker":"(Costarelli et al., 2024)"},{"why":"It provided an existing game-competition leaderboard that this work positions as less extensible.","marker":"(Topsakal et al., 2024)"},{"why":"It supplied the head-to-head human-preference arena that motivates replacing human judging with automated competition.","marker":"(Chiang et al., 2024)"},{"why":"It defined the passphrase-guarding Gandalf game that the paper adopts as one of its seven games.","marker":"(Pfister et al., 2025)"},{"why":"It provided the paired-comparison rating model used to convert match outcomes into model ratings.","marker":"(Bradley & Terry, 1952)"},{"why":"It provided the prompt-abstraction layer that lets the framework swap models and strategies without hand-written prompts.","marker":"(Khattab et al., 2023)"},{"why":"It supplied the assertion-based retry loop that gives models feedback when they attempt invalid game moves.","marker":"(Singhvi et al., 2024)"}],"fun_headline_variants":["Zero-sum games: LLMs can't create, only answer","Head-to-head AI reveals: models can't craft fresh questions","Pit LLMs against each other: they lose at making challenges","ZeroSumEval: competition eval exposes a creativity gap","Model versus model: zero-sum test shows they can't create"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The assumption that carries the whole evaluation is that when a model can solve a challenge it created itself without having seen the answer, that challenge is genuinely valid and appropriately difficult—but this design pushes models toward questions they themselves can easily solve, so the measured creativity gap could partly reflect that push rather than an intrinsic limit.","fun_headline_variants_meta":{"raw":{"variants":["Zero-sum games: LLMs can't create, only answer","Head-to-head AI reveals: models can't craft fresh questions","Pit LLMs against each other: they lose at making challenges","ZeroSumEval: competition eval exposes a creativity gap","Model versus model: zero-sum test shows they can't create"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000845,"raw_usage":{"total_tokens":3712,"prompt_tokens":1012,"completion_tokens":2700,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":628,"completion_tokens_details":{"reasoning_tokens":2614}},"tokens_in":628,"tokens_out":2700,"duration_ms":21948,"temperature":1.0,"reasoning_tokens":2614,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:27:56.450534+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test would be to rerun MathQuiz and PyJail with the verification step altered: have a different model, or a symbolic solver, validate the generated challenge instead of the generating model, or reveal the target to the generator during question-writing and measure whether opponents' failure rates rise. If students still fail less than ten percent of the time when the generator never had to solve its own question, the paper's creativity gap holds; if failure rates jump, the gap is an artifact of self-solution verification.","supporting_citations":[],"review_version":1}