Pith. sign in

REVIEW 3 major objections 4 minor 53 references

Relay-Bench: Evaluating LLMs on Multi-Domain Reasoning Chains

T0 review · 3 major / 4 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Relay-Bench claims multi-domain reasoning chains remain a bottleneck, with the best model scoring 43.3%.

desk verdict Genuinely new benchmark construction with unusually candid reporting, but the paper never shows it measures multi-domain chaining rather than bloat and decoding exhaustion. read the letter →

arxiv 2607.18438 v1 pith:CXQYCJ2K submitted 2026-07-20 cs.CL cs.AIcs.LG

Liam Swayne This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords LLMbenchmarkmulti-domainreasoningcompositeproblemschainspromptencodingcontextbloatPass@1evaluationexact-stringgrading
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Relay-Bench is a new benchmark built on composite problems: each test item chains two to thirteen single-domain subproblems into one prompt so that the final answer depends on all of them. The paper's central claim is that this measures LLMs' ability to reason across several domains in combination, and that the benchmark is still far from saturated — the leading model, GPT-5.5 (xHigh), scores 43.3%, while comparable once-unsaturated suites have been pushed well past 90%. To keep the test hard, the author layers two extra forms of difficulty onto many problems: 7,813 characters of irrelevant user preferences and a token-substitution encoding that can require more than 10,000 decoding operations. The test set is text-only, tool-permissive, exact-string graded, and kept private, which the paper argues makes it cheap, reproducible, and hard to contaminate. The paper itself notes that the 30-problem set produces wide confidence intervals and that no ablation separates the composite-reasoning load from the encoding and context-bloat layers.

What carries the argument

The load-bearing object is the composite-problem dependency graph: a chain of independently solvable subproblems whose answers are fed into later subproblems, with the final answer dependent on every node and no cyclic dependencies. Two augmentation layers carry the difficulty: a fixed 7,813-character block of irrelevant user preferences, and an automated token-substitution encoder that replaces each word, symbol, or number with a case-sensitive three-letter string in angle brackets, forcing models to decode long prompts (up to 10,000+ operations) before solving. Exact-string grading and a private held-out test set are the supporting mechanisms that make Pass@1 scoring and contamination cont

What would settle it

Run the same composite problems through two conditions — standard encoded and bloated versus decoded with the user-preferences block removed — and compare Pass@1; if the decoded condition approaches the pass rates of the individual subproblems, the multi-domain chain is not the main source of difficulty.

Watch

Extended reading notes

Core claim

Every Relay-Bench item is a composite problem: subproblems from distinct domains (visual reasoning in text form, coding, math, information extraction, problem-solving, general knowledge, data analysis) are strung together with dependencies, and the final answer depends on all of them. Models are allowed every tool, including code execution and web search, and answers must follow exact-string formatting rules. On this set, GPT-5.5 achieves 43.3%, Gemini 3.1 Pro 40.0%, and Claude Opus 4.7 16.7% at Pass@1; the paper interprets the leading score as evidence that the benchmark has one to two years of useful headroom. The paper's discovery is thus a construction: composite multi-domain prompts, de

Load-bearing premise

The load-bearing premise is that low scores come from the challenge of combining multiple reasoning domains, since the paper does not run an ablation separating that challenge from the 7,813-character context bloat and token-substitution encoding it layers on top.

Editorial extensions

If this is right

  • Frontier-model differentiation can be preserved for an estimated one to two years after single-domain benchmarks saturate, based on the leader-progression rate of comparable benchmarks.
  • Difficulty can be pushed into the problems themselves rather than into multi-modal inputs or Pass^n scoring, so cheap Pass@1 evaluation suffices for a low-saturation benchmark.
  • Text-only prompts with exact-string answer rules make evaluation reproducible across model families, including preview or beta models without tool-calling or multi-modal support.
  • Keeping the full test set private removes the need for URL blocklists and reduces contamination risk from web search during evaluation.
  • Because the test set has only 30 problems, Wilson confidence intervals are wide, so small gaps in leaderboard scores should not be read as meaningful.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the dominant failure driver is context bloat or decoding burden rather than cross-domain composition, Relay-Bench's scores would overstate the gap in multi-domain reasoning; a decoded, bloat-free version of the same composites would separate these mechanisms.
  • The one-to-two-year saturation estimate extrapolates progress on earlier benchmarks; a model with substantially longer effective context or cheaper tool orchestration could close the gap faster by absorbing the mechanical load.
  • Claude Opus 4.7's explicit refusals on encoded problems suggest that safety or instruction-following behavior may be entangled with measured ability; refusing and being unable are different failure modes that the aggregate score treats alike.
  • The exact-string capitalization and symbol rules mean the benchmark also tests instruction-following reliability, so any interpretation of model differences should note this conflation, which is a design choice rather than an accident.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. Relay-Bench introduces a text-only, tool-permissive benchmark consisting of 30 private 'composite problems,' each made of 2–13 subproblems spanning domains such as math, coding, visual reasoning, information extraction, and data analysis. The paper evaluates Claude Opus 4.7, Gemini 3.1 Pro, and GPT-5.5 with maximum thinking effort and all tools enabled, reporting Pass@1 scores of 16.7%, 40.0%, and 43.3% respectively. It claims the benchmark is unsaturated, that GPT-5.5 leads, and that the rate of leader progression on comparable benchmarks suggests Relay-Bench could take one to two years to saturate. The methodology includes deliberate context bloat (7,813 characters of irrelevant user preferences) and a token-substitution encoding step requiring thousands of decoding operations, plus deterministic exact-string grading and Wilson confidence intervals.

Significance. If validated, Relay-Bench would fill a real gap: it is text-only, tool-permissive, deterministically graded, and designed to be hard enough to differentiate frontier models after single-domain benchmarks have saturated. The paper has several genuine strengths: the grading is machine-checkable and avoids LLM judges; the test set is private, reducing contamination risk; the authors report Wilson CIs and a candid limitations section; and the cost analysis in Table 1 is useful practical information. However, the central construct—'reasoning across multiple domains in combination'—is not isolated from the deliberately added encoding and context-bloat layers, and the 30-problem test set produces confidence intervals that are too wide to support the paper's ranking and saturation claims. The contributions are potentially valuable, but the current evidence does not establish that the benchmark measures what its abstract claims.

major comments (3)
  1. [§3.2, Appendix B] The benchmark's load-bearing claim is that low scores measure 'reasoning across multiple domains in combination.' This is never isolated from the two difficulty layers the authors add deliberately: 7,813 characters of irrelevant user preferences and a token-substitution dictionary that forces more than 10,000 translation operations per long prompt. §3.2 itself cites Du et al. for the proposition that context bloat alone degrades LLM performance, and §4.1 reports that Claude Opus 4.7 refused every encoded long problem. No result is reported split by encoded vs. unencoded or bloated vs. unbloated problems, and no unencoded baseline is given for the composite problems. Consequently, the 43.3% leader score and the 'unsaturated' status could be driven by decoding exhaustion, instruction-following fatigue, or context-window effects rather than cross-domain chaining. The authors should add a co
  2. [§5, Fig. 7] With only 30 test problems, the Wilson 95% CIs are too wide to support the paper's ordering claims. GPT-5.5 (43.3%, CI 27.4–60.8) and Gemini 3.1 Pro (40.0%, CI 24.6–57.7) are statistically indistinguishable, and Claude Opus 4.7's CI (7.3–33.6) is wider than its point estimate. The abstract's claim that 'GPT-5.5 leads Relay-Bench' and the Fig. 2 caption's ranking therefore overstate the evidence. The saturation projection ('one to two years') is not derived from any stated extrapolation method and cannot be supported by a 30-item sample with overlapping intervals. Please either enlarge the test set, report formal overlap tests, or soften the claims to the ordering the data can actually support.
  3. [§3.1] The difficulty calibration loop is endogenous: composite problems that did not cause model failure were augmented or subsumed until the tested models failed. Because the same model families (Claude Opus 4.7, Gemini 3.1 Pro, GPT-5.4 Mini) were used to generate subproblems and to tune difficulty, the benchmark's hardness is partly optimized against the evaluated cohort. This does not by itself invalidate the scores, but it weakens the saturation claim and any inference about future model progress. The paper should report how many problems were augmented in response to each model's success, include a holdout of problems generated without this loop, or evaluate a model released after the May 1, 2026 cohort cutoff to break the selection dependence.
minor comments (4)
  1. [§4.2, Table 1] The text says Claude Opus 4.7's input token usage 'exceeded two million on six problems,' but Table 1 lists total input token usage for Claude as 1,665,250. Clarify whether the two-million figure is per-problem, per-run, or includes cached tokens across tool-call iterations.
  2. [Fig. 4] The stacked bar caption should specify that the total bar height is the number of parseable responses and that the 'incorrect' segment includes only parseable wrong answers. Currently 'Correct answers Incorrect, parseable answers' is ambiguous.
  3. [Footnote 1] The sentence 'All three models evaluated in this paper were tested on the problem' should say 'on the public example problem' and should note explicitly that only GPT-5.5 answered it correctly. As written, the antecedent of 'the problem' is unclear.
  4. [Appendix B] The encoded prompt is extremely long; a decoded excerpt or a pointer to a machine-readable file would help readers verify the translation and decode process without manually reconstructing hundreds of dictionary entries.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: reported scores are direct measurements; iterative item selection and prompt encoding raise validity questions but do not reduce any claim to its own inputs.

full rationale

The paper's central output is a set of measured Pass@1 scores on a private test set. No equation fits a parameter to those scores and then re-predicts a closely related quantity. The iterative augmentation described in §3.1 ('Model performance was measured periodically throughout the creation of the benchmark to determine which problems required revision to differentiate model performance. Composite problems that did not cause model failure were further augmented with additional subproblems or subsumed into larger composite problems') is a benchmark-construction procedure, not a derived result: it selects difficult items, but the reported 43.3% score is a direct measurement, not a consequence of a fitted model. The §3.2 encoding and context-bloat layer is a plausible confound for the claim that low scores reflect multi-domain reasoning, but a missing ablation is a construct-validity threat, not a circular reduction. External comparisons (HLE, ARC-AGI-2, τ2-Bench) provide independent reference points, and the saturation extrapolation is based on the progression of other benchmarks rather than on re-labeling the paper's own fitted outputs. Self-references in the paper (e.g., the footnote about the public example problem) are not load-bearing: no external uniqueness theorem or prior self-citation is invoked to force the paper's interpretation. Overall, no step in the derivation chain reduces by construction to its own inputs, so the circularity score is 0.

Assumptions & free parameters 4 free parameters · 5 assumptions · 1 invented entities

The central claim rests on a measurement instrument that is entirely private, calibrated by an unreported loop against the tested models, and supplemented by a difficulty layer (context bloat + encoding) that is never isolated from the construct it claims to measure. These are assumptions, not derivations — the scores themselves are honestly measured outcomes.

free parameters (4)
  • Problem difficulty calibration (augmentation loop)
    §3.1: subproblems were measured against models during construction and revised/augmented until they caused failure; this hand-tunes the difficulty level to the evaluated cohort and directly drives the 'unsaturated' claim.
  • Context bloat length = 7813 characters
    §3.2: the fabricated user-preference block size is a chosen constant that contributes to measured difficulty; its effect on scores is never ablated.
  • Encoding dictionary design (case-reuse collisions)
    §3.2: substitution dictionary size and intentional reuse of the same string with different capitalization are hand-chosen adversarial parameters; the longest problems require >10,000 translation operations, inseparable from reasoning difficulty in the reported scores.
  • Saturation-timeline projection = 1–2 years
    Fig. 2/§5: the projected saturation date is extrapolated from leader progression of GAIA and other benchmarks, not measured on Relay-Bench itself.
assumptions (5)
  • domain assumption Composite-problem difficulty is attributable to cross-domain reasoning rather than to context bloat or decoding burden
    Load-bearing for the benchmark's stated purpose; §3.2 adds two unablated difficulty mechanisms (planted context, token encoding) that can explain low scores without any multi-domain reasoning effect.
  • domain assumption The private test set cannot be retrieved by tool-enabled models during evaluation
    §1 justifies the private set by citing a model (Claude Mythos) that surreptitiously located a benchmark test set; the paper provides no check (e.g., search-log probe) that Relay-Bench itself was never retrieved.
  • domain assumption Exact-string grading maps to correctness of reasoning
    §1: answers are constrained to exact strings with granular capitalization rules; a model that solves a problem but violates format rules is scored wrong, conflating instruction-following with reasoning failure by design.
  • domain assumption Third-party leaderboard scores for HLE, ARC-AGI-2, and τ²-Bench accurately represent model capability
    Fig. 2's saturation comparison rests on Artificial Analysis URLs ([1,2,3]); these externally hosted numbers are not independently verified in the paper.
  • domain assumption Context bloat degrades LLM performance (Du et al. [45])
    The bloat mechanism is justified by a cited prior result; this is a legitimate citation, but it is also a premise for why scores are low.
invented entities (1)
  • Relay-Bench private test set (30 composite problems)
    purpose: The measurement instrument; the benchmark's value proposition depends on this hidden artifact being valid, non-leaked, correctly keyed to gold answers, and representative of generalist multi-domain work.
    No public release, no code, no per-problem data; only one worked example in Appendix A. The private set is exactly the part an outsider cannot inspect, so its construction quality, gold-answer accuracy, and contamination status are unverifiable.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Relay-Bench: Evaluating LLMs on Multi-Domain Reasoning Chains." pith.science (2026). https://pith.science/paper/CXQYCJ2K

@misc{pith2026260718438,
  author       = {Pith},
  title        = {Pith review of: Relay-Bench: Evaluating LLMs on Multi-Domain Reasoning Chains},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CXQYCJ2K}},
  note         = {Machine review of arXiv:2607.18438}
}
read the original abstract

Introducing Relay-Bench, an unsaturated, holistic, text-only benchmark that measures LLMs' ability to complete an assortment of tasks from distinct domains in a single prompt. The leading model, GPT-5.5 (xHigh), scores 43.3%. The test set entirely consists of composite problems: groups of single-domain subproblems that are strung together into challenges that require reasoning across multiple domains in combination. Many of these problems then have layers of complexity added through prompt encoding and deliberate context bloat. Domains tested include visual reasoning, coding, math, information extraction (with a focus on web search), problem-solving, general knowledge, and data analysis. No restrictions are imposed outside of the model harness, and models are explicitly encouraged to leverage code-execution, web searches, and all available tools. All problems are composed of two to thirteen subproblems and do not require multi-modal input or output.

Figures

Figures reproduced from arXiv: 2607.18438 by the authors.

Figure 2
Figure 2. All models evaluated score lower on Relay-Bench than selected widely used benchmarks, including Human￾ity’s Last Exam (HLE), which has not reached saturation more than 15 months after release [1, 2, 3]. GPT-5.5 leads Relay-Bench with a score of 43.3%. The average Relay-Bench score across the three models tested is 33.3%, compared to averages of 42.9% on HLE, 79.3% on ARC-AGI-2, and 92.7% on τ 2 -Bench. The rate of l… view at source ↗
Figure 3
Figure 3. Each dot represents a model’s best GAIA accuracy at release, colored by developer. The line of best fit (turquoise) for record-setting scores on GAIA extrapolates saturation to December 15th, 2025 [34]. 3 METHODOLOGY Every problem in the test set is a composite problem, com￾posed of two to thirteen subproblems. Each subproblem is akin to a single test set problem in existing question￾answering benchmarks (e.g., Fron… view at source ↗
Figure 4
Figure 4. Counts of correct and incorrect parseable responses. The total height of each bar is the number of parseable re￾sponses returned by a model. Claude Opus 4.7 (Max) Gemini 3.1 Pro (High) GPT-5.5 (xHigh) 0 25 50 75 100 16.7 33.3 50.0 36 50 86 Rate (%) Hallucination Rates Across Benchmarks Relay-Bench AA-Omniscience [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: The fraction of parseable responses that were incor￾rect on Relay-Bench and the AA-Omniscience hallucination￾rate evaluation [46]. Claude Opus 4.7 returned the fewest parseable answers, but had the lowest hallucination rate because only one of them was incorrect ( [PI…
Figure 6
Figure 6. Figure 6: Models’ Pass@1 accuracy against Relay-Bench evalu￾ation cost. The upper left region is optimal (low cost, high ac￾curacy). GPT-5.5 Pareto-dominates both other models tested, occupying the frontier alone. 5 LIMITATIONS AND FUTURE WORK Relay-Bench is held back due to res…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

53 extracted references · 21 linked inside Pith

  1. [1]

    ARC Prize Leaderboard

    ARC Prize. ARC Prize Leaderboard. https://arcprize.org /leaderboard. Accessed: May 22, 2026

    2026

  2. [2]

    τ 2-Bench Telecom Benchmark Leaderboard

    Artificial Analysis. τ 2-Bench Telecom Benchmark Leaderboard. https://artificialanalysis.ai/evaluations/tau2-b ench?models=gpt-5-5%2Cgemini-3-1-pro-preview%2Cc laude-opus-4-7 . Accessed: May 22, 2026

    2026

  3. [3]

    Humanity’s Last Exam Benchmark Leader- board

    Artificial Analysis. Humanity’s Last Exam Benchmark Leader- board. https://artificialanalysis.ai/evaluations/ humanitys- last- exam?models=gpt- 5- 5%2Cgemini- 3 -1-pro-preview%2Cclaude-opus-4-7 . Accessed: May 22, 2026

    2026

  4. [4]

    Introducing Claude Opus 4.7

    Anthropic. Introducing Claude Opus 4.7. Apr. 16, 2026. https: //www.anthropic.com/news/claude-opus-4-7

    2026

  5. [5]

    Claude Mythos Preview System Card

    Anthropic. Claude Mythos Preview System Card. 2026. https: //www-cdn.anthropic.com/08ab9158070959f88f29651 4c21b7facce6f52bc.pdf. Accessed: May 27, 2026

    2026

  6. [6]

    The Gemini Team, Google.Gemini 3.1 Pro: A smarter model for your most complex tasks. Feb. 19, 2026. https://blog.googl e/innovation-and-ai/models-and-research/gemini-m odels/gemini-3-1-pro/

    2026

  7. [7]

    Introducing GPT-5.5

    OpenAI. Introducing GPT-5.5. Apr. 23, 2026. https://openai .com/index/introducing-gpt-5-5/

    2026

  8. [8]

    Grok 4.3 Beta

    xAI. Grok 4.3 Beta. Apr. 17, 2026. https://grok.com/relea se-notes/apr-17-2026

    2026

Show all 53 references
  1. [9]

    Qwen3.7: The Agent Frontier

    Qwen Team, Alibaba Cloud. Qwen3.7: The Agent Frontier. May 19, 2026. https://qwen.ai/blog?id=qwen3.7

    2026

  2. [10]

    Gonzalez, and Ion Stoica

    Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline. arXiv:2406.11939, 2024. https:// arxiv.org/abs/2406.11939

    2024 arXiv

  3. [11]

    WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild

    Bill Yuchen Lin et al. WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild . arXiv:2406.04770, 2024. h t t p s : / / a r x i v . o r g / a b s / 2 4 06.04770

    2024 arXiv

  4. [12]

    AgentFrontier: Expanding the Capabil- ity Frontier of LLM Agents with ZPD-Guided Data Synthesis

    Xuanzhong Chen et al. AgentFrontier: Expanding the Capabil- ity Frontier of LLM Agents with ZPD-Guided Data Synthesis. arXiv:2510.24695, 2025. https://arxiv.org/abs/2510.2 4695

    2025

  5. [13]

    Pan et al

    Melissa Z. Pan et al. Measuring Agents in Production . arXiv:2512.04123, 2025. https://arxiv.org/abs/2512.0 4123

    2025 arXiv

  6. [14]

    GPQA: A Graduate-Level Google-Proof Q&A Benchmark

    David Rein et al. GPQA: A Graduate-Level Google-Proof Q&A Benchmark. arXiv:2311.12022, 2023. https://arxiv.org/ab s/2311.12022

    2023 arXiv

  7. [15]

    Jimenez et al

    Carlos E. Jimenez et al. SWE-bench: Can Language Models Re- solve Real-World GitHub Issues?arXiv:2310.06770, 2023. http s://arxiv.org/abs/2310.06770

    2023 arXiv

  8. [16]

    SpreadsheetBench: Towards Challenging Real World Spreadsheet Manipulation

    Zeyao Ma et al. SpreadsheetBench: Towards Challenging Real World Spreadsheet Manipulation. arXiv:2406.14991, 2024. http s://arxiv.org/abs/2406.14991

    2024 arXiv

  9. [17]

    Chatbot Arena: An Open Platform for EvaluatingLLMsbyHumanPreference

    Wei-Lin Chiang et al. Chatbot Arena: An Open Platform for EvaluatingLLMsbyHumanPreference . arXiv:2403.04132, 2024. https://arxiv.org/abs/2403.04132

    2024 arXiv

  10. [18]

    τ-bench: A Benchmark for Tool-Agent-User Inter- action in Real-World Domains

    Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. τ-bench: A Benchmark for Tool-Agent-User Inter- action in Real-World Domains. arXiv:2406.12045, 2024. https: //arxiv.org/abs/2406.12045

    2024 arXiv

  11. [19]

    GPQA Diamond Benchmark Leaderboard

    Artificial Analysis. GPQA Diamond Benchmark Leaderboard. https://artificialanalysis.ai/evaluations/gpqa -diamond. Accessed: May 23, 2026

    2026

  12. [20]

    MATH-500 Benchmark Leaderboard

    Artificial Analysis. MATH-500 Benchmark Leaderboard. http s://artificialanalysis.ai/evaluations/math- 500 . Accessed: May 23, 2026

    2026

  13. [21]

    https: //artificialanalysis.ai/evaluations/aime- 2025

    Artificial Analysis.AIME2025BenchmarkLeaderboard . https: //artificialanalysis.ai/evaluations/aime- 2025 . Accessed: May 23, 2026

    2025

  14. [23]

    MMMU: A Massive Multi-discipline Multi- modalUnderstandingandReasoningBenchmarkforExpertAGI

    Xiang Yue et al. MMMU: A Massive Multi-discipline Multi- modalUnderstandingandReasoningBenchmarkforExpertAGI . CVPR, 2024. https://arxiv.org/abs/2311.16502. 6

    2024 arXiv

  15. [24]

    MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts

    Pan Lu et al. MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts. ICLR, 2024. https: //arxiv.org/abs/2310.02255

    2024 arXiv

  16. [25]

    arXiv:2307.16789, 2023

    Yujia Qin et al.ToolLLM: Facilitating Large Language Models to Master16000+Real-worldAPIs . arXiv:2307.16789, 2023. https: //arxiv.org/abs/2307.16789

    2023 arXiv

  17. [26]

    API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs

    Minghao Li et al. API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs. arXiv:2304.08244, 2023. https://ar xiv.org/abs/2304.08244

    2023 arXiv

  18. [27]

    Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

    Lianmin Zheng et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv:2306.05685, 2023. https://arxiv. org/abs/2306.05685

    2023 arXiv

  19. [28]

    Hashimoto

    Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators. arXiv:2404.04475, 2024. https: //arxiv.org/abs/2404.04475

    2024 arXiv

  20. [29]

    GAIA: a benchmark for General AI As- sistants

    Grégoire Mialon et al. GAIA: a benchmark for General AI As- sistants. arXiv:2311.12983, 2023. https://arxiv.org/abs/ 2311.12983

    2023 arXiv

  21. [30]

    APEX-Agents

    Bertie Vidgen et al. APEX-Agents. arXiv:2601.14242, 2026. ht tps://arxiv.org/abs/2601.14242

    2026

  22. [31]

    Are Your LLMs Capable of Stable Reasoning? arXiv:2412.13147, 2024

    Junnan Liu et al. Are Your LLMs Capable of Stable Reasoning? arXiv:2412.13147, 2024. https://arxiv.org/abs/2412.131 47

    2024 arXiv

  23. [32]

    Project Glasswing

    Anthropic. Project Glasswing. 2026. https://www.anthropi c.com/project/glasswing. Accessed: May 26, 2026

    2026

  24. [33]

    Claude Mythos Preview: Anthropic’s Frontier Model Explained

    Claude Fast. Claude Mythos Preview: Anthropic’s Frontier Model Explained. 2026. https://claudefa.st/blog/mod els/claude-mythos. Accessed: May 26, 2026

    2026

  25. [34]

    Holistic Agent Leader- board: GAIA

    Princeton Language and Intelligence. Holistic Agent Leader- board: GAIA. https://hal.cs.princeton.edu/gaia . Ac- cessed: May 26, 2026

    2026

  26. [35]

    arXiv:2411.04872, 2024

    Elliot Glazer, Ege Erdil, Tamay Besiroglu, et al.FrontierMath:A Benchmark for Evaluating Advanced Mathematical Reasoning in AI. arXiv:2411.04872, 2024. https://arxiv.org/abs/24 11.04872

    2024 arXiv

  27. [36]

    LiveCodeBench: Holistic and Contami- nation Free Evaluation of Large Language Models for Code

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. LiveCodeBench: Holistic and Contami- nation Free Evaluation of Large Language Models for Code . arXiv:2403.07974, 2024. https://arxiv.org/abs/24...

    2024 arXiv

  28. [37]

    Investigating Data Contamina- tion in Modern Benchmarks for Large Language Models

    Chunyuan Deng et al. Investigating Data Contamina- tion in Modern Benchmarks for Large Language Models . arXiv:2311.09783, 2024. https://arxiv.org/abs/2311.0 9783

    2024 arXiv

  29. [38]

    Anthropic API documentation

    Anthropic. Anthropic API documentation. https://docs.ant hropic.com/en/api/overview. Accessed: May 27, 2026

    2026

  30. [39]

    Gemini API documentation

    Google. Gemini API documentation. https://ai.google.de v/gemini-api/docs. Accessed: May 27, 2026

    2026

  31. [40]

    OpenAI API reference

    OpenAI. OpenAI API reference. https://platform.openai. com/docs/api-reference. Accessed: May 27, 2026

    2026

  32. [41]

    Adaptive thinking

    Anthropic. Adaptive thinking. https://platform.claude. com/docs/en/build-with-claude/adaptive-thinking . Accessed: May 27, 2026

    2026

  33. [42]

    Gemini thinking

    Google. Gemini thinking. https://ai.google.dev/gemini -api/docs/thinking. Accessed: May 27, 2026

    2026

  34. [43]

    Reasoning models

    OpenAI. Reasoning models. https://developers.openai. com/api/docs/guides/reasoning. Accessed: May 27, 2026

    2026

  35. [44]

    AA-Omniscience: Evaluating Cross- Domain Knowledge Reliability in Large Language Models

    Declan Jackson, William Keating, George Cameron, and Micah Hill-Smith. AA-Omniscience: Evaluating Cross- Domain Knowledge Reliability in Large Language Models . arXiv:2511.13029, 2025. https://arxiv.org/abs/2511.1 3029

    2025

  36. [45]

    Context Length Alone Hurts LLM Performance Despite Perfect Retrieval

    Yufeng Du et al. Context Length Alone Hurts LLM Performance Despite Perfect Retrieval. arXiv:2510.05381, 2025. https://ar xiv.org/abs/2510.05381

    2025

  37. [46]

    AA-Omniscience: Knowledge and Halluci- nation Benchmark

    Artificial Analysis. AA-Omniscience: Knowledge and Halluci- nation Benchmark. https://artificialanalysis.ai/ev aluations/omniscience?models=gemini-3-1-pro-pre view%2Cclaude-opus-4-7%2Cgpt-5-5#omniscience-hal lucination-rate-tabs . Accessed: May 29, 2026

    2026

  38. [47]

    A. T. Kalai, O. Nachum, S. S. Vempala, and E. Zhang. Why Language Models Hallucinate. arXiv:2509.04664, 2025. https: //arxiv.org/abs/2509.04664

    2025 arXiv

  39. [48]

    Claude Fable 5 and Claude Mythos 5

    Anthropic. Claude Fable 5 and Claude Mythos 5. May 28, 2026. https://www.anthropic.com/news/claude-fable-5-m ythos-5

    2026

  40. [49]

    Artificial Analysis Intelligence Index

    Artificial Analysis. Artificial Analysis Intelligence Index. http s://artificialanalysis.ai/evaluations/artificial -analysis-intelligence-index . Accessed: June 11, 2026

    2026

  41. [51]

    Artificial Analysis. Stirrup. https://github.com/Artific ialAnalysis/Stirrup. Accessed: June 30, 2026

    2026

  42. [52]

    Artificial Analysis Intelligence Index: Methodology

    Artificial Analysis. Artificial Analysis Intelligence Index: Methodology. https://artificialanalysis.ai/method ology/intelligence- benchmarking . Accessed: June 30, 2026

    2026

  43. [53]

    AssetOpsBench: Stirrup Agent

    IBM. AssetOpsBench: Stirrup Agent. https://github.com/I BM/AssetOpsBench/blob/main/docs/stirrup-agent.md . Accessed: June 30, 2026

    2026

  44. [54]

    Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning

    NVIDIA. Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning. ht tps://arxiv.org/abs/2606.15007 . Accessed: June 30, 2026. 7 APPENDIX A / PUBLIC EXAMPLE PROBLEM A benchmark for evaluating language models on mathematics compe...

    2026

  45. [2025]

    https://arxiv.org/abs/2501.14249

    arXiv

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.