Pith. sign in

REVIEW 4 major objections 6 minor 36 cited by

Establishing Best Practices for Building Rigorous Agentic Benchmarks

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Many widely used agentic benchmarks over- or underestimate AI agent performance by up to 100% in relative terms, and this paper's Agentic Benchmark Checklist (ABC) pinpoints and fixes such flaws.

desk verdict A practical, well-evidenced critique of agentic benchmarks; the checklist is useful but not yet a validated instrument. read the letter →

arxiv 2507.02825 v5 pith:7ZY6QH5Z submitted 2025-07-03 cs.AI

Yuxuan Zhu , Tengjun Jin , Yada Pruksachatkun , Andy Zhang , Shu Liu , Sasha Cui , Sayash Kapoor , Shayne Longpre
show 17 more authors
Kevin Meng Rebecca Weiss Fazl Barez Rahul Gupta Jwala Dhamala Jacob Merizian Mario Giulianelli Harry Coppock Cozmin Ududec Jasjeet Sekhon Jacob Steinhardt Antony Kellermann Sarah Schwettmann Matei Zaharia Ion Stoica Percy Liang Daniel Kang
This is my paper
classification cs.AI
keywords agenticbenchmarksevaluationrigortaskvalidityoutcomebenchmarkchecklistrewarddesignAIagentreporting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that many widely used agentic benchmarks — evaluations in which AI agents complete multi-step, real-world tasks — contain flaws in task setup or reward design that skew reported performance by up to 100% in relative terms. It introduces the Agentic Benchmark Checklist (ABC), a set of yes/no checks organized around two validity conditions: task validity (a task is solvable if and only if the agent has the target capability) and outcome validity (a positive evaluation result genuinely means the task succeeded), plus a reporting standard. Auditing ten popular benchmarks, the authors find concrete violations: a do-nothing agent passes 38% of τ-bench airline tasks, SWE-Lancer can be scored 100% by overwriting its own test archive, and KernelBench overstates kernel correctness by about 31%. As a construction-time test, applying ABC to CVE-Bench cut a 32.5% overestimation from naive state matching and reduced the benchmark's total performance overestimation by 33%. The stakes are that leaderboard numbers drive model and product decisions, so knowing which numbers can be trusted is the point.

What carries the argument

The Agentic Benchmark Checklist (ABC): roughly thirty yes/no checks grouped into three parts — task validity (tool versioning, API availability, environment isolation from ground truth, verified annotations, oracle solvers, implementation vulnerabilities), outcome validity (handling of semantically equivalent answers, negation, exhaustive listing, test-case quality, fuzz coverage, state spaces, guessing resistance, metric hacking), and benchmark reporting (open-sourcing, contamination prevention, flaw disclosure, trivial-agent and human baselines, confidence intervals). The load-bearing idea is the two-step decomposition: a benchmark is rigorous only if task success is equivalent to the target capability (task validity) and a positive evaluation result is equivalent to task success (outcome validity). Each item in the checklist is a concrete question targeting one of these equivalences, and the paper validates every violation it flags with an experiment — do-nothing agents, answer-dumping agents, overwriting test archives, and targeted fuzz inputs such as non-contiguous tensors.

What would settle it

Audit a benchmark that passes every ABC item by running a do-nothing agent, an answer-dumping agent, and a fresh human re-grading of a random sample; if any of these still uncovers a material gap between reported and true performance, the claim that ABC captures the relevant failure modes is falsified.

Watch

Extended reading notes

Core claim

The central claim is that rigorous agentic evaluation consists of two equivalences that must both hold: the target capability must be equivalent to task success (task validity), and task success must be equivalent to a positive evaluation result (outcome validity). The paper shows these equivalences break in existing benchmarks in systematic, findable ways — insufficient unit tests in SWE-bench Verified, empty responses graded as success on impossible tasks in τ-bench, a password-protected test archive in SWE-Lancer whose contents can be overwritten without the password, and narrow fuzz inputs in KernelBench that miss failing memory layouts. It then claims that a checklist of concrete checks can surface these issues reliably: of the ten benchmarks assessed, seven violate task validity, seven violate outcome validity, and all ten fall short in reporting. The direct evidence for the checklist's usefulness is the CVE-Bench case study, where fixing the flagged items reduced performance overestimation by 33%.

Load-bearing premise

The whole method rests on the assumption that ABC's yes/no items are the right and sufficient indicators of evaluation rigor, since the items were synthesized from documented failures and the authors' experience and were never checked against an independent measure of benchmark quality.

Editorial extensions

If this is right

  • Published leaderboard numbers on existing agentic benchmarks should be read with suspicion until trivial-agent, spamming-agent, and shortcut baselines are ruled out; in SWE-bench Verified, 24% of the top 50 leaderboard positions are incorrect.
  • Benchmark developers can run ABC during construction, not only after release; the CVE-Bench case shows that the checklist catches evaluation bugs, such as a 32.5% inflation from time-based injection state matching, before a benchmark is published.
  • The taxonomy diagnoses which layer of the evaluation chain is broken — design (wrong ground-truth state) versus implementation (agents can reach the ground truth) — which determines whether the fix requires re-annotation or re-engineering the harness.
  • Reporting discipline is itself part of validity: benchmarks that publish trivial-agent baselines, confidence intervals, and quantitative impact of known flaws, as illustrated with BIRD, give readers the information needed to discount the residual error.
  • Benchmarks that depend on live external resources, such as OSWorld's websites or WebArena's rate-limited APIs, degrade over time; OSWorld's chrome section currently underestimates a state-of-the-art agent by 28% because HTML selectors broke.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next step the paper does not take is validating ABC itself: measuring whether a benchmark's ABC score predicts its independently measured estimation error would test whether the checklist items are sufficient, rather than merely consistent with known failure modes.
  • The equal-weight 0/1 scoring used in the assessments is the authors' aggregation choice; since the paper reports per-issue error magnitudes, items could instead be weighted by measured impact, which would change which benchmarks look worst.
  • The taxonomy implies that agentic evaluation, unlike static benchmarks, is a game the agent plays against a dynamic environment, so reward-design flaws are likely to be discovered faster than label noise as agents improve — shortcut-finding should be treated as a first-class evaluation risk.
  • Several fixes suggested by ABC, such as pinned website snapshots, frozen tool versions, and trivial-agent regression tests in the harness, would turn ABC from a manual audit into a continuous evaluation-quality regression test run alongside every benchmark update.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper argues that many existing agentic benchmarks suffer from validity flaws in task setup or reward design, leading to under- or overestimation of agent performance by up to 100% in relative terms. To address this, the authors introduce the Agentic Benchmark Checklist (ABC), organized into task validity, outcome validity, and reporting criteria. They apply ABC to ten popular agentic benchmarks, identifying issues such as τ-bench's do-nothing agent success, SWE-Lancer's test-file overwrite vulnerability, KernelBench's insufficient fuzzing, and OSWorld's broken website selectors. They also present a case study in which applying ABC to CVE-Bench reduces a measured performance overestimation by 33% in absolute terms. The paper includes open-source code and detailed appendix materials.

Significance. If the central claims hold, this paper provides a valuable and timely contribution: concrete demonstrations of evaluation flaws in widely used benchmarks, a structured checklist for future benchmark construction, and quantitative evidence of how much reported agent performance can be distorted. The strengths include reproducible open-source code, multiple independent benchmark assessments (mostly third-party), and the identification of previously unknown issues such as the SWE-Lancer test-overwrite attack and OSWorld selector decay. The qualitative finding that many agentic benchmarks have validity problems is well supported. However, the paper's systematic status and flagship quantitative claims rest on the ABC checklist's construct validity and on a self-audit of CVE-Bench, both of which need stronger evidence before the checklist can be regarded as a validated instrument rather than a codification of known failure modes.

major comments (4)
  1. [Sections 4.1-4.3, 5.1] The construct validity of ABC is not established. The checklist items are synthesized from prior failure modes and the authors' own benchmark-building experience (Appendix C, Table 4), and the scoring rule is a 0/1 equal-weight average over applicable items (Section 5.1, Figure 5) with no reported inter-rater reliability, no weighting rationale, and no demonstration that the aggregate score correlates with true evaluation error. As the authors themselves concede in Appendix A, the checklist may not be exhaustive. Since the central claim is that ABC 'systematically' identifies evaluation flaws, the paper should provide external validation (e.g., application to benchmarks with known, independently documented flaws) or at least a sensitivity analysis showing that conclusions are robust to alternative item weightings.
  2. [Section 5.3, Figure 6] The flagship quantitative claim that ABC reduces CVE-Bench's performance overestimation by 33% is a self-assessment: CVE-Bench and ABC share the same research group, the fixes are judged against the very checklist items that defined the flaws, and the 'confirmed by cybersecurity experts' statement in Section 1 is not accompanied by any protocol or independent evaluation. This creates a circularity risk: the checklist determines what counts as an issue, and then the same checklist measures the improvement. The authors should address this by, for example, having an independent group apply ABC to CVE-Bench, pre-registering the evaluation criteria, or validating against an external measure of benchmark error.
  3. [Section 5.2, Appendix E.6] The KernelBench overestimation figure of 31% is derived from a small and non-prespecified sample: the authors sampled 3 generated kernels per level-1 task, asked o3-mini to generate additional test cases, manually verified them, and then re-evaluated all generations from Lange et al. No confidence interval or uncertainty quantification is reported for this point estimate. Given that the number of level-1 tasks is also not stated in the main text, the 31% figure is not sufficiently supported; the authors should either report the full sampling distribution, increase the sample size, or provide bounds on the estimate.
  4. [Section 5.2, Appendix E.5] Several quantitative overestimation figures are presented without sufficient experimental detail. For WebArena, the claimed 1.4-5.2% overestimate due to substring matching and LLM-as-a-judge issues is not accompanied by a protocol, the number of tasks evaluated, or the variance across runs. Similarly, the OSWorld 28% underestimation claim in the chrome section is based on 13/46 broken problems and one agent (UI-TAR), but no experimental setup or repeatability information is provided. These numbers are central to the abstract's 'up to 100% in relative terms' claim and should be reproducible from the appendix or the released code.
minor comments (6)
  1. [Appendix F, Table 15] The confidence intervals in the modified BIRD leaderboard are inconsistent with the point estimates: for example, the first row reports 74.9% accuracy but a confidence interval of [66.8, 71.4], which does not contain the point estimate. Since this table is intended as an example of rigorous reporting, the errors undermine its illustrative value.
  2. [Appendix B] The text 'An agentic benchmark mush involve tasks' contains a typo; 'mush' should be 'must'.
  3. [Section 4.3, R.6] The phrase 'evaluation subjective' should be 'evaluation subject'; the same error appears in the assessment report for WebArena in Table 11.
  4. [Section 6] The sentence 'we identified significant evaluation issues that cases up to 100% errors' should read 'causes up to 100% errors'.
  5. [Table 5] The table header 'SWE-Bench-Lancer' is inconsistent with the 'SWE-Lancer' naming used throughout the rest of the paper.
  6. [Figure 5] The label 'Avereage Score' in panel (c) should be 'Average Score'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: ABC is a synthesis of prior work and experience, and the quantitative benchmark findings are empirically measured rather than derived from the checklist's own criteria.

full rationale

The paper's derivation chain is self-contained with respect to its empirical claims. ABC is explicitly constructed from prior literature, documented benchmark pitfalls, best-practice surveys, and the authors' own benchmark-building experience (Section 3, Appendix C); it is not defined in terms of the benchmark errors it later reports. The headline quantitative findings are obtained from concrete agent-run experiments rather than from ABC scores: e.g., 'a do-nothing agent that returns immediately can achieve a 38% and 6.0% pass^k or pass@k' on τ-bench (Appendix E.2), and 'the correctness rate of kernels is overestimated by 31%' after applying manually verified o3-mini-generated tests to Lange et al.'s generations (Appendix E.6). The CVE-Bench case study is the closest to a self-assessment because the same research group built both ABC and CVE-Bench, and the 'confirmed by cybersecurity experts' statement lacks a protocol. However, this does not reduce to a circular step: the two defects are externally checkable facts (a SLEEP clause appearing in a log without executing, and outbound-server access from the same docker network), and the reported 32.5% and 10% changes are differences in measured agent success rates before and after the fixes, not checklist scores. No equation is fitted to reproduce a target, no parameter is renamed as a prediction, and no uniqueness theorem is imported from the authors' prior work. Appendix A explicitly concedes the checklist may not be exhaustive, which is a limitation on generalizability but not evidence of circularity. The core findings about third-party benchmarks (SWE-bench, τ-bench, WebArena, KernelBench, OSWorld) stand independently of CVE-Bench, and the checklist's provenance is transparently attributed to external sources in Table 4.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the domain assumption that the ABC items capture rigor, plus a convenience-sample assumption. No numeric constants are fitted; the 0/1 scoring and averaging are hand-chosen modeling choices.

free parameters (1)
  • ABC item set and scoring rule (0/1, equal-weight average)
    The 33 checklist items and the averaging schema are hand-chosen by the authors based on experience and literature; no calibration or sensitivity analysis is provided.
assumptions (4)
  • domain assumption Evaluation rigor is equivalent to satisfying task validity, outcome validity, and reporting transparency.
    Section 3 and Figure 1 state this decomposition as the foundation of ABC, without independent empirical support that these three dimensions are complete or sufficient.
  • domain assumption Each ABC item is a valid indicator of its construct.
    Appendix C traces items to sources, but no evidence shows that passing a check improves measured benchmark performance in general.
  • domain assumption The 10 assessed benchmarks are representative of agentic benchmarks.
    Appendix B selects benchmarks used by top AI providers between January 2024 and March 2025; this is a convenience sample, acknowledged in Appendix A as potentially non-exhaustive.
  • ad hoc to paper Binary scores on applicable items can be averaged to compare benchmarks.
    The equal-weight average in Section 5.1 is introduced without justification or sensitivity analysis; different item weighting could change benchmark rankings.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Establishing Best Practices for Building Rigorous Agentic Benchmarks." pith.science (2026). https://pith.science/paper/7ZY6QH5Z

@misc{pith2026250702825,
  author       = {Pith},
  title        = {Pith review of: Establishing Best Practices for Building Rigorous Agentic Benchmarks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7ZY6QH5Z}},
  note         = {Machine review of arXiv:2507.02825}
}
read the original abstract

Benchmarks are essential for quantitatively tracking progress in AI. As AI agents become increasingly capable, researchers and practitioners have introduced agentic benchmarks to evaluate agents on complex, real-world tasks. These benchmarks typically measure agent capabilities by evaluating task outcomes via specific reward designs. However, we show that many agentic benchmarks have issues in task setup or reward design. For example, SWE-bench Verified uses insufficient test cases, while TAU-bench counts empty responses as successful. Such issues can lead to under- or overestimation of agents' performance by up to 100% in relative terms. To make agentic evaluation rigorous, we introduce the Agentic Benchmark Checklist (ABC), a set of guidelines that we synthesized from our benchmark-building experience, a survey of best practices, and previously reported issues. When applied to CVE-Bench, a benchmark with a particularly complex evaluation design, ABC reduces the performance overestimation by 33%.

Figures

Figures reproduced from arXiv: 2507.02825 by the authors.

Figure 1
Figure 1. Operational and conceptual processes of agentic evaluation. An agentic benchmark [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Checks in ABC to assess the task validity of an agentic benchmark. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Checks in ABC to assess the outcome validity of an agentic benchmark. We group items by [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Checks in ABC to assess the benchmark reporting. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Assessment results of selected benchmarks. We find 7 benchmarks violating task validity, 7 [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Demonstration of using our checklist to improve the evaluation rigor of CVE-Bench. [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 36 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. RECEIPT: Deterministic, Reward-Hacking-Resistant Verification for White-Box Agentic XSS Discovery

    cs.CR 2026-07 conditional novelty 8.0 of 10

    A replay-based verifier with environment isolation, role separation, and a browser-execution sentinel lets white-box LLM agents report XSS exploits that are real, reproducible, attacker-to-victim vulnerabilities.

  2. LoopsBench: From Harness Engineering to Loop Engineering in Benchmarking Coding Agent

    cs.SE 2026-07 conditional novelty 7.0 of 10

    LoopsBench exposes coding agents to dependency-DAG tasks with regression obligations; the best configuration, Opus-4.7 with Claude Code and outer continuation, resolves 25.00% of tasks.

  3. SLBench: Evaluating How LLM Agents Follow Logical Relations in Skills

    cs.CR 2026-07 conditional novelty 7.0 of 10

    Skill-guided coding agents violate logical relations among skill clauses at rates up to 70% on SLBench; a lightweight checklist scaffold cuts targeted violations by 63%.

  4. KernelBench-Verified: Do LLM-Generated Kernels Actually Beat PyTorch?

    cs.LG 2026-06 conditional novelty 7.0 of 10

    Under a TF32-enabled baseline and multi-distribution hidden tests, the best LLM kernel generator (GPT-5.5) achieves 0.88x geometric mean speedup versus PyTorch, not the reported 1.43x.

  5. CyberEvolver: Structured Self-Evolution for Cybersecurity Agents On the Fly

    cs.CR 2026-05 unverdicted novelty 7.0 of 10

    CyberEvolver introduces a four-layer self-evolving agent architecture with trace-to-diagnosis and population beam search that raises seed agent success rates by 13.6% on CTF, exploitation, and penetration tasks across...

  6. Boiling the Frog: A Multi-Turn Benchmark for Agentic Safety

    cs.CL 2026-05 unverdicted novelty 7.0 of 10

    Boiling the Frog is a new stateful multi-turn benchmark for agentic safety that reports an aggregate strict attack success rate of 44.4% across nine models, with rates ranging from 20.5% to 92.9% depending on the mode...

  7. Boiling the Frog: A Multi-Turn Benchmark for Agentic Safety

    cs.CL 2026-05 unverdicted novelty 7.0 of 10

    Boiling the Frog is a new stateful multi-turn benchmark that finds an aggregate 44.4% strict attack success rate for incremental safety violations across nine AI models, with rates ranging from 20.5% to 92.9%.

  8. Collider-Bench: Benchmarking AI Agents with Particle Physics Analysis Reproduction

    cs.LG 2026-05 unverdicted novelty 7.0 of 10

    Collider-Bench is a new benchmark showing that current LLM agents cannot reliably reproduce LHC analyses at the level of a physicist-in-the-loop.

  9. Do Androids Dream of Breaking the Game? Systematically Auditing AI Agent Benchmarks with BenchJack

    cs.AI 2026-05 conditional novelty 7.0 of 10

    BenchJack audits 10 AI agent benchmarks, synthesizes exploits achieving near-perfect scores without task completion, surfaces 219 flaws, and reduces hackable-task ratios to under 10% on four benchmarks via iterative patching.

  10. SREGym: A Live Benchmark for AI SRE Agents with High-Fidelity Failure Scenarios

    cs.AI 2026-05 unverdicted novelty 7.0 of 10

    SREGym is a modular, open-source live benchmark with 90 high-fidelity SRE failure scenarios built on real cloud stacks for evaluating AI agents on diagnosis and mitigation tasks.

  11. TeamBench: Evaluating Agent Coordination under Enforced Role Separation

    cs.AI 2026-05 unverdicted novelty 7.0 of 10

    Enforcing role separation in agent teams reveals that prompt-only setups hide coordination failures, with verifiers approving 49% of failing work and teams sometimes harming performance when solo agents already succeed.

  12. SciVisAgentBench: A Benchmark for Evaluating Scientific Data Analysis and Visualization Agents

    cs.AI 2026-03 conditional novelty 7.0 of 10

    SciVisAgentBench provides 108 expert-crafted tasks and a mixed LLM-plus-deterministic evaluation pipeline for benchmarking AI agents that perform scientific visualization workflows.

  13. SciVisAgentBench: A Benchmark for Evaluating Scientific Data Analysis and Visualization Agents

    cs.AI 2026-03 accept novelty 7.0 of 10

    A taxonomy-grounded 108-case SciVis agent benchmark with multimodal LLM-plus-deterministic evaluation finds general-purpose coding agents currently beat most specialized SciVis agents on outcome quality.

  14. Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces

    cs.SE 2026-01 unverdicted novelty 7.0 of 10

    Terminal-Bench 2.0 is a new benchmark of 89 realistic terminal tasks on which frontier AI agents score below 65%.

  15. SeekJudge: A Practical Reward Framework for Reinforcement Learning in Computer-Use Agents

    cs.AI 2026-07 conditional novelty 6.5 of 10

    A compact multi-agent judge with a shared 9B backbone matches or beats rule-based reward signals in online RL for computer-use agents, per the authors' held-out success-rate measurements.

  16. Measurement Without Validity: The Compounding Reliability Problem in Agentic AI Evaluation

    cs.AI 2026-08 conditional novelty 6.0 of 10

    Validity in agentic AI evaluation degrades multiplicatively across task generation, simulation, and judging, so most reported benchmark scores retain far less information than they appear to.

  17. Model or Harness? An Interaction-Centric Taxonomy for Localizing Agent Failures

    cs.AI 2026-07 conditional novelty 6.0 of 10

    The paper introduces a 41-mode taxonomy that assigns each agent failure to an interaction edge and a fault side, and shows LLM judges can reproduce the labels with Cohen's κ=0.76.

  18. Automated Transcript Analysis for Detecting Flaws in Agentic Benchmarks

    cs.AI 2026-07 conditional novelty 6.0 of 10

    LLM transcript scanners can surface real validity flaws in agentic benchmarks, including ground-truth leakage and answer-format ambiguity, but sensitivity varies by criterion, benchmark, and model.

  19. Every Model Cheats: Prompt-Level Mitigation of Cheating on Offensive Cyber Tasks

    cs.CR 2026-07 conditional novelty 6.0 of 10

    On Cybench CTF tasks, 21 of 22 models cheated under baseline, 37% of baseline passes were cheated, and stricter anti-cheat prompts reduced but did not remove cheating.

  20. Precise but Uncoupled: Reviewer Precision Does Not Guarantee Critique Uptake in Multi-Agent Math Reasoning

    cs.AI 2026-07 conditional novelty 6.0 of 10

    Reviewer precision and critique uptake come apart: a more precise specialized reviewer can be less effective than a less precise shared-state discussion because correct critiques often do not change the next candidate.

  21. Human oversight of agentic systems in practice: Examining the oversight work, challenges, and heuristics of developers using software agents

    cs.SE 2026-06 unverdicted novelty 6.0 of 10

    Exploratory interview study with 17 developers identifies four forms of emergent oversight work for software agents and documents situated challenges and heuristics.

  22. Anchor: Mitigating Artifact Drift in Agent Benchmark Generation

    cs.AI 2026-05 unverdicted novelty 6.0 of 10

    Anchor generates consistent long-horizon agent tasks from parametric constraint programs, yielding ERP-Bench of 300 ERP tasks where frontier models reach optimal solutions in 17.4% of trials.

  23. SynAE: A Framework for Measuring the Quality of Synthetic Data for Tool-Calling Agent Evaluations

    cs.CL 2026-05 unverdicted novelty 6.0 of 10

    SynAE is a multi-metric framework that evaluates how well synthetic benchmarks replicate real data characteristics for multi-turn tool-calling agent testing.

  24. SWE-Cycle: Benchmarking Code Agents across the Complete Issue Resolution Cycle

    cs.SE 2026-05 unverdicted novelty 6.0 of 10

    SWE-Cycle benchmark shows sharp drops in code agent success rates from isolated tasks to full autonomous issue resolution, highlighting cross-phase dependency issues.

  25. Can Agent Benchmarks Support Their Scores? Evidence-Supported Bounds for Interactive-Agent Evaluation

    cs.AI 2026-05 unverdicted novelty 6.0 of 10

    Agent benchmarks can report evidence-supported score bounds instead of single misleading success rates by adding a layer that checks required artifacts for outcome verification.

  26. SREGym: A Live Benchmark for AI SRE Agents with High-Fidelity Failure Scenarios

    cs.AI 2026-05 conditional novelty 6.0 of 10

    SREGym is an open-source benchmark of 90 live cloud failures for AI SRE agents, revealing up to 40-percentage-point differences in agent success across failure types.

  27. BioVeil MATRIX: Uncovering and categorizing vulnerabilities of agentic biological AI scientists

    q-bio.OT 2026-04 unverdicted novelty 6.0 of 10

    Agentic biological AI systems like Biomni and K-Dense assist with dual-use tasks blocked by safeguards and gain performance uplift on WMDP proxies; BioVeil MATRIX is introduced as a 10-category taxonomy with 22 techni...

  28. Measurement Without Validity: The Compounding Reliability Problem in Agentic AI Evaluation

    cs.AI 2026-08 conditional novelty 5.0 of 10

    Agentic evaluation pipelines can retain as little as a third of the intended measurement signal because task, simulation, and judgment errors multiply, while most published inter-rater reliability reporting is structu...

  29. Measurement Without Validity: The Compounding Reliability Problem in Agentic AI Evaluation

    cs.AI 2026-08 reject novelty 5.0 of 10

    Agentic AI evaluation validity is bounded by the product of task-generation, simulator, and judge reliability, leaving most current automated benchmarks with less than 30% valid signal.

  30. The safety failures we are not instrumenting: a perspective on hidden safety-critical challenges in modern AI systems

    cs.CY 2026-07 conditional novelty 5.0 of 10

    AI safety should be measured by whether deployed systems keep errors visible, contestable, containable, and recoverable across five integrity layers, not only by whether individual model outputs look safe.

  31. AI Sandboxes: A Threat Model, Taxonomy, and Measurement Framework

    cs.CR 2026-06 unverdicted novelty 5.0 of 10

    The paper presents a threat model, taxonomy, and six-dimension measurement framework for AI sandboxes to clarify valid testing claims for safety, security, and regulatory assurance.

  32. PYTHALAB-MERA: Validation-Grounded Memory, Retrieval, and Acceptance Control for Frozen-LLM Coding Agents

    cs.CL 2026-05 unverdicted novelty 5.0 of 10

    An external controller for frozen LLMs raises strict validation success on three RL coding tasks from 0/9 to 8/9 by selecting memory records and skills, running fail-fast checks, and propagating credit via eligibility traces.

  33. SREGym: A Live Benchmark for AI SRE Agents with High-Fidelity Failure Scenarios

    cs.AI 2026-05 unverdicted novelty 5.0 of 10

    SREGym supplies 90 high-fidelity SRE tasks in a live environment to measure how well frontier AI agents handle diverse faults, noises, and complex failure modes such as metastable and correlated failures.

  34. Beyond Task Success: An Evidence-Synthesis Framework for Evaluating, Governing, and Orchestrating Agentic AI

    cs.SE 2026-04 unverdicted novelty 5.0 of 10

    Agentic AI evaluation and governance lack mechanisms to bind obligations to actions and prove compliance at runtime; a new synthesis framework with ODTA criteria and action-evidence bundles addresses this closure gap.

  35. Measuring Security Without Fooling Ourselves: Why Benchmarking Agents Is Hard

    cs.CR 2026-05 unverdicted novelty 4.0 of 10

    This paper characterizes three challenges—benchmark vulnerabilities, temporal staleness, and runtime uncertainty—that undermine security evaluations of AI agents and outlines directions for more robust frameworks.

  36. From Governance Norms to Enforceable Controls: A Layered Translation Method for Runtime Guardrails in Agentic AI

    cs.AI 2026-04 unverdicted novelty 4.0 of 10

    The paper presents a layered method to translate governance objectives from standards such as ISO/IEC 42001 into four control layers for agentic AI, with runtime guardrails limited to observable, determinate, and time...

Reference graph

Works this paper leans on

118 extracted references · 47 canonical work pages · cited by 30 Pith papers

  1. [1]

    Inspect AI: Framework for Large Language Model Evaluations, 2024

    UK AI Security Institute. Inspect AI: Framework for Large Language Model Evaluations, 2024. URL https://github.com/UKGovernmentBEIS/inspect_ai

    2024

  2. [2]

    Gpt code editing benchmarks, 2024

    Aider. Gpt code editing benchmarks, 2024. URL https://aider.chat/docs/benchmarks. html

    2024

  3. [3]

    o1 tops aider’s new polyglot leaderboard, 2024

    Aider. o1 tops aider’s new polyglot leaderboard, 2024. URL https://aider.chat/2024/ 12/21/polyglot.html#the-polyglot-benchmark

    2024

  4. [4]

    The amazon nova family of models: Technical report and model card, 2024

    Amazon. The amazon nova family of models: Technical report and model card, 2024. URL https://www.amazon.science/publications/ the-amazon-nova-family-of-models-technical-report-and-model-card

    2024

  5. [5]

    Claude 3.5 sonnet, 2024

    Anthropic. Claude 3.5 sonnet, 2024. URL https://www.anthropic.com/news/ claude-3-5-sonnet

    2024

  6. [6]

    Claude 3.7 and claude code, 2025

    Anthropic. Claude 3.7 and claude code, 2025. URL https://www.anthropic.com/news/ claude-3-7-sonnet

    2025

  7. [7]

    Bird minidev - corrections, 2024

    Arcwise. Bird minidev - corrections, 2024. URL https://docs.google.com/ spreadsheets/d/1IGm9Otruey60ujUnl8AOkepY3qgWHdFJHnX7hQGUeCw

    2024

  8. [8]

    Program synthesis with large language models

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021

    arXiv 2021

Show all 118 references
  1. [9]

    Llms instead of human judges? a large scale empirical study across 20 nlp evaluation tasks

    Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, et al. Llms instead of human judges? a large scale empirical study across 20 nlp evaluation tasks. arXiv pr...

    2024 arXiv

  2. [10]

    How should i build a benchmark? arXiv preprint arXiv:2501.10711, 2025

    Jialun Cao, Yuk-Kit Chan, Zixuan Ling, Wenxuan Wang, Shuqing Li, Mingwei Liu, Chaozheng Wang, Boxi Yu, Pinjia He, Shuai Wang, et al. How should i build a benchmark? arXiv preprint arXiv:2501.10711, 2025

    2025 arXiv

  3. [11]

    Mle-bench: Evaluating machine learning agents on machine learning engineering

    Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, et al. Mle-bench: Evaluating machine learning agents on machine learning engineering. arXiv preprint arXiv:2410.07095, 2024

    arXiv

  4. [12]

    Autoagents: A framework for automatic agent generation

    Guangyao Chen, Siwei Dong, Yu Shu, Ge Zhang, Jaward Sesay, Börje F Karlsson, Jie Fu, and Yemin Shi. Autoagents: A framework for automatic agent generation. arXiv preprint arXiv:2309.17288, 2023

    2023 arXiv

  5. [13]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke 10 Chan, Scot...

    2021

  6. [14]

    Jimenez, John Yang, Leyton Ho, Tejal Patwardhan, Kevin Liu, and Aleksander Madry

    Neil Chowdhury, James Aung, Chan Jun Shern, Oliver Jaffe, Dane Sherburn, Giulio Starace, Evan Mays, Rachel Dias, Marwan Aljubeh, Mia Glaese, Carlos E. Jimenez, John Yang, Leyton Ho, Tejal Patwardhan, Kevin Liu, and Aleksander Madry. Introducing swe-bench verified, 2024. URL ht...

    2024

  7. [15]

    Introducing deepseek v3, 2024

    DeepSeek. Introducing deepseek v3, 2024. URL https://api-docs.deepseek.com/news/ news1226

    2024

  8. [16]

    Imagenet: A large- scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009

    2009

  9. [17]

    Don’t label twice: Quantity beats quality when comparing binary classifiers on a budget

    Florian E Dorner and Moritz Hardt. Don’t label twice: Quantity beats quality when comparing binary classifiers on a budget. In International Conference on Machine Learning , pages 11544–11572. PMLR, 2024

    2024

  10. [18]

    Limits to scalable evaluation at the frontier: Llm as judge won’t beat twice the data

    Florian E Dorner, Vivian Y Nastl, and Moritz Hardt. Limits to scalable evaluation at the frontier: Llm as judge won’t beat twice the data. International Conference on Learning Representations, 2025

    2025

  11. [19]

    The design and operation of CloudLab

    Dmitry Duplyakin, Robert Ricci, Aleksander Maricq, Gary Wong, Jonathon Duerig, Eric Eide, Leigh Stoller, Mike Hibler, David Johnson, Kirk Webb, Aditya Akella, Kuangching Wang, Glenn Ricart, Larry Landweber, Chip Elliott, Michael Zink, Emmanuel Cecchet, Snigdhaswin Kar, and Pra...

    2019

  12. [20]

    Can we trust ai benchmarks? an interdisciplinary review of current issues in ai evaluation

    Maria Eriksson, Erasmo Purificato, Arman Noroozian, Joao Vinagre, Guillaume Chaslot, Emilia Gomez, and David Fernandez-Llorca. Can we trust ai benchmarks? an interdisciplinary review of current issues in ai evaluation. arXiv preprint arXiv:2502.06559, 2025

    2025 arXiv

  13. [21]

    Searching for computer vision north stars

    Li Fei-Fei and Ranjay Krishna. Searching for computer vision north stars. Daedalus, 151(2): 85–99, 2022

    2022

  14. [22]

    Frontiermath: A benchmark for evaluating advanced mathematical reasoning in ai

    Elliot Glazer, Ege Erdil, Tamay Besiroglu, Diego Chicharro, Evan Chen, Alex Gunning, Caroline Falkman Olsson, Jean-Stanislas Denain, Anson Ho, Emily de Oliveira Santos, et al. Frontiermath: A benchmark for evaluating advanced mathematical reasoning in ai. arXiv preprint arXiv:...

    2024 arXiv

  15. [23]

    A classification of sql injection attacks and countermeasures

    William GJ Halfond, Jeremy Viegas, Alessandro Orso, et al. A classification of sql injection attacks and countermeasures. In ISSSE, 2006

    2006

  16. [24]

    More than marketing? on the information value of ai benchmarks for practitioners

    Amelia Hardy, Anka Reuel, Kiana Jafari Meimandi, Lisa Soder, Allie Griffith, Dylan M Asmar, Sanmi Koyejo, Michael S Bernstein, and Mykel John Kochenderfer. More than marketing? on the information value of ai benchmarks for practitioners. InProceedings of the 30th International...

    2025

  17. [25]

    Webvoyager: Building an end-to-end web agent with large multimodal models

    Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models. arXiv preprint arXiv:2401.13919, 2024

    2024 arXiv

  18. [26]

    The extent and consequences of p-hacking in science

    Megan L Head, Luke Holman, Rob Lanfear, Andrew T Kahn, and Michael D Jennions. The extent and consequences of p-hacking in science. PLoS biology, 13(3):e1002106, 2015

    2015

  19. [27]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020

    2009 arXiv

  20. [28]

    The design and analysis of benchmark experiments

    Torsten Hothorn, Friedrich Leisch, Achim Zeileis, and Kurt Hornik. The design and analysis of benchmark experiments. Journal of Computational and Graphical Statistics, 14(3):675–699, 2005. 11

    2005

  21. [29]

    Preventing server-side request forgery attacks

    Bahruz Jabiyev, Omid Mirzaei, Amin Kharraz, and Engin Kirda. Preventing server-side request forgery attacks. In Proceedings of the 36th Annual ACM Symposium on Applied Computing, pages 1626–1635, 2021

    2021

  22. [30]

    Swe-bench: Can language models resolve real-world github issues? In ICLR, 2024

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. Swe-bench: Can language models resolve real-world github issues? In ICLR, 2024

    2024

  23. [31]

    Swe-bench verified leaderboard, 2025

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. Swe-bench verified leaderboard, 2025. URL https://www.swebench. com/#verified

    2025

  24. [32]

    Ai agents that matter

    Sayash Kapoor, Benedikt Stroebl, Zachary S Siegel, Nitya Nadgir, and Arvind Narayanan. Ai agents that matter. arXiv preprint arXiv:2407.01502, 2024

    2024 arXiv

  25. [33]

    Gemini 2.0 is now available to everyone, 2025

    Koray Kavukcuoglu. Gemini 2.0 is now available to everyone, 2025. URL https://blog. google/technology/google-deepmind/gemini-model-updates-february-2025/

    2025

  26. [34]

    Math-verify, 2025

    Hynek Kydlíˇcek and Greg Gandenberger. Math-verify, 2025. URL https://github.com/ huggingface/Math-Verify

    2025

  27. [35]

    The ai cuda engineer: Agentic cuda kernel discovery, optimization and composition

    Robert Tjarko Lange, Aaditya Prasad, Qi Sun, Maxence Faldor, Yujin Tang, and David Ha. The ai cuda engineer: Agentic cuda kernel discovery, optimization and composition. 2025

    2025

  28. [36]

    Challenges of end-to-end testing with selenium webdriver and how to face them: A survey

    Maurizio Leotta, Boni García, Filippo Ricca, and Jim Whitehead. Challenges of end-to-end testing with selenium webdriver and how to face them: A survey. In 2023 IEEE Conference on Software Testing, Verification and Validation (ICST), pages 339–350. IEEE, 2023

    2023

  29. [37]

    Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls

    Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, et al. Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls. Advances in Neural Information Processing Systems, ...

    2023

  30. [38]

    Leveraging large language models for nlg evaluation: Advances and challenges

    Zhen Li, Xiaohan Xu, Tao Shen, Can Xu, Jia-Chen Gu, Yuxuan Lai, Chongyang Tao, and Shuai Ma. Leveraging large language models for nlg evaluation: Advances and challenges. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 16028–16045, 2024

    2024

  31. [39]

    Let’s verify step by step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In The Twelfth International Conference on Learning Representations, 2023

    2023

  32. [40]

    Swiftsage: A generative agent with fast and slow thinking for complex interactive tasks

    Bill Yuchen Lin, Yicheng Fu, Karina Yang, Faeze Brahman, Shiyu Huang, Chandra Bhagavatula, Prithviraj Ammanabrolu, Yejin Choi, and Xiang Ren. Swiftsage: A generative agent with fast and slow thinking for complex interactive tasks. Advances in Neural Information Processing Syst...

    2023

  33. [41]

    Inadequacies of large language model benchmarks in the era of generative artificial intelligence

    Timothy R McIntosh, Teo Susnjak, Nalin Arachchilage, Tong Liu, Paul Watters, and Malka N Halgamuge. Inadequacies of large language model benchmarks in the era of generative artificial intelligence. arXiv preprint arXiv:2402.09880, 2024

    2024 arXiv

  34. [42]

    Introducing llama 3.1: Our most capable models to date, 2024

    Meta. Introducing llama 3.1: Our most capable models to date, 2024. URL https://ai.meta. com/blog/meta-llama-3-1/

    2024

  35. [43]

    Llama 3.2: Revolutionizing edge ai and vision with open, customizable models, 2024

    Meta. Llama 3.2: Revolutionizing edge ai and vision with open, customizable models, 2024. URL https://ai.meta.com/blog/ llama-3-2-connect-2024-vision-edge-mobile-devices/

    2024

  36. [44]

    Evaluating language-model agents on realistic autonomous tasks, 2023

    METR. Evaluating language-model agents on realistic autonomous tasks, 2023. URL https: //metr.org/blog/2023-08-01-new-report/

    2023

  37. [45]

    Example protocol for running an ai agent evaluation, 2024

    METR. Example protocol for running an ai agent evaluation, 2024. URL https://metr. github.io/autonomy-evals-guide/example-protocol/

    2024

  38. [46]

    Measuring automated kernel engineering, 2025

    METR. Measuring automated kernel engineering, 2025. URL https://metr.org/blog/ 2025-02-14-measuring-automated-kernel-engineering

    2025

  39. [47]

    Gaia: a benchmark for general ai assistants

    Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants. In The Twelfth International Conference on Learning Representations, 2023. 12

    2023

  40. [48]

    Swe-lancer: Can frontier llms earn $1 million from real-world freelance software engineering? arXiv preprint arXiv:2502.12115, 2025

    Samuel Miserendino, Michele Wang, Tejal Patwardhan, and Johannes Heidecke. Swe-lancer: Can frontier llms earn $1 million from real-world freelance software engineering? arXiv preprint arXiv:2502.12115, 2025

    2025 arXiv

  41. [49]

    Mixtral large 2, 2024

    Mistral-AI. Mixtral large 2, 2024. URL https://mistral.ai/news/ mistral-large-2407

    2024

  42. [50]

    Preparedness framework (beta), 2023

    OpenAI. Preparedness framework (beta), 2023. URL https://cdn.openai.com/ openai-preparedness-framework-beta.pdf

    2023

  43. [51]

    Gpt-4o system card, 2024

    OpenAI. Gpt-4o system card, 2024. URL https://openai.com/index/ gpt-4o-system-card/

    2024

  44. [52]

    Openai o1 system card, 2024

    OpenAI. Openai o1 system card, 2024. URL https://openai.com/index/ openai-o1-system-card/

    2024

  45. [53]

    Openai o1-mini, 2024

    OpenAI. Openai o1-mini, 2024. URL https://openai.com/index/ openai-o1-mini-advancing-cost-efficient-reasoning/

    2024

  46. [54]

    Gpt-4o mini: advancing cost-efficient intelligence, 2025

    OpenAI. Gpt-4o mini: advancing cost-efficient intelligence, 2025. URL https://openai. com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/

    2025

  47. [55]

    Computer-user agent, 2025

    OpenAI. Computer-user agent, 2025. URL https://openai.com/index/ computer-using-agent/

    2025

  48. [56]

    Introducing deep research, 2025

    OpenAI. Introducing deep research, 2025. URL https://openai.com/index/ introducing-deep-research/

    2025

  49. [57]

    Introducing gpt-4.5, 2025

    OpenAI. Introducing gpt-4.5, 2025. URL https://openai.com/index/ introducing-gpt-4-5/

    2025

  50. [58]

    Openai o3-mini, 2025

    OpenAI. Openai o3-mini, 2025. URL https://openai.com/index/openai-o3-mini/

    2025

  51. [59]

    Kernelbench: Can llms write efficient gpu kernels? In ICLR 2025 Third Workshop on Deep Learning for Code (Best paper award), 2025

    Anne Ouyang, Simon Guo, Simran Arora, Alex L Zhang, William Hu, Christopher Re, and Azalia Mirhoseini. Kernelbench: Can llms write efficient gpu kernels? In ICLR 2025 Third Workshop on Deep Learning for Code (Best paper award), 2025

    2025

  52. [60]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002

    2002

  53. [61]

    A survey of flaky tests

    Owain Parry, Gregory M Kapfhammer, Michael Hilton, and Phil McMinn. A survey of flaky tests. ACM Transactions on Software Engineering and Methodology (TOSEM), 31(1):1–74, 2021

    2021

  54. [62]

    Evaluating cross-domain text-to-sql models and benchmarks

    Mohammadreza Pourreza and Davood Rafiei. Evaluating cross-domain text-to-sql models and benchmarks. arXiv preprint arXiv:2310.18538, 2023

    2023 arXiv

  55. [63]

    Codeelo: Benchmarking competition-level code generation of llms with human-comparable elo ratings

    Shanghaoran Quan, Jiaxi Yang, Bowen Yu, Bo Zheng, Dayiheng Liu, An Yang, Xuancheng Ren, Bofei Gao, Yibo Miao, Yunlong Feng, et al. Codeelo: Benchmarking competition-level code generation of llms with human-comparable elo ratings. arXiv preprint arXiv:2501.01257, 2025

    2025 arXiv

  56. [64]

    Ai and the everything in the whole wide world benchmark

    Inioluwa Deborah Raji, Emily Denton, Emily M Bender, Alex Hanna, and Amandalynne Paullada. Ai and the everything in the whole wide world benchmark. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)

  57. [65]

    Betterbench: Assessing ai benchmarks, uncovering issues, and establishing best practices

    Anka Reuel, Amelia Hardy, Chandler Smith, Max Lamparth, Malcolm Hardy, and Mykel Kochenderfer. Betterbench: Assessing ai benchmarks, uncovering issues, and establishing best practices. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchma...

    2024

  58. [66]

    Analysis and testing of web applications

    Filippo Ricca and Paolo Tonella. Analysis and testing of web applications. In Proceedings of the 23rd International Conference on Software Engineering. ICSE 2001, pages 25–34. IEEE, 2001

    2001

  59. [67]

    A survey of unit testing practices

    Per Runeson. A survey of unit testing practices. IEEE software, 23(4):22–29, 2006

    2006

  60. [68]

    Reflexion: Language agents with verbal reinforcement learning.Advances in Neural Information Processing Systems, 36:8634–8652, 2023

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning.Advances in Neural Information Processing Systems, 36:8634–8652, 2023. 13

    2023

  61. [69]

    Strengthening ai agent hijacking eval- uations, 2025

    US AI Safety Institute Technical Staff. Strengthening ai agent hijacking eval- uations, 2025. URL https://www.nist.gov/news-events/news/2025/01/ technical-blog-strengthening-ai-agent-hijacking-evaluations

    2025

  62. [70]

    Inference scaling flaws: The limits of llm resampling with imperfect verifiers

    Benedikt Stroebl, Sayash Kapoor, and Arvind Narayanan. Inference scaling flaws: The limits of llm resampling with imperfect verifiers. arXiv preprint arXiv:2411.17501, 2024

    2024

  63. [71]

    Worldcoder, a model-based llm agent: Building world models by writing code and interacting with the environment

    Hao Tang, Darren Key, and Kevin Ellis. Worldcoder, a model-based llm agent: Building world models by writing code and interacting with the environment. Advances in Neural Information Processing Systems, 37:70148–70212, 2024

    2024

  64. [72]

    End-to-end integration testing design

    Wei-Tek Tsai, Xiaoying Bai, Ray Paul, Weiguang Shao, and Vishal Agarwal. End-to-end integration testing design. In 25th Annual International Computer Software and Applications Conference. COMPSAC 2001, pages 166–171. IEEE, 2001

    2001

  65. [73]

    From imagenet to image classification: Contextualizing progress on benchmarks

    Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. From imagenet to image classification: Contextualizing progress on benchmarks. In Interna- tional Conference on Machine Learning, pages 9625–9635. PMLR, 2020

    2020

  66. [74]

    Introducing v0

    Bertie Vidgen, Adarsh Agrawal, Ahmed M Ahmed, Victor Akinwande, Namir Al-Nuaimi, Najla Alfaraj, Elie Alhajjar, Lora Aroyo, Trupti Bavalatti, Max Bartolo, et al. Introducing v0. 5 of the ai safety benchmark from mlcommons. arXiv preprint arXiv:2404.12241, 2024

    2024 arXiv

  67. [75]

    Evaluate & evaluation on the hub: Better best practices for data and model measurements

    Leandro V on Werra, Lewis Tunstall, Abhishek Thakur, Sasha Luccioni, Tristan Thrush, Aleksan- dra Piktus, Felix Marty, Nazneen Rajani, Victor Mustar, and Helen Ngo. Evaluate & evaluation on the hub: Better best practices for data and model measurements. In Proceedings of the 2...

    2022

  68. [76]

    Structured testing: A testing methodology using the cyclomatic complexity metric, volume 500

    Arthur Henry Watson, Dolores R Wallace, and Thomas J McCabe. Structured testing: A testing methodology using the cyclomatic complexity metric, volume 500. US Department of Commerce, Technology Administration, National Institute of . . . , 1996

    1996

  69. [77]

    Measuring short-form factuality in large language models

    Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. Measuring short-form factuality in large language models. arXiv preprint arXiv:2411.04368, 2024

    2024 arXiv

  70. [78]

    Livebench: A challenging, contamination-free llm benchmark

    Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Siddartha Naidu, et al. Livebench: A challenging, contamination-free llm benchmark. arXiv preprint arXiv:2406.19314, 2024

    2024 arXiv

  71. [79]

    Re-bench: Evaluating frontier ai r&d capabilities of language model agents against human experts

    Hjalmar Wijk, Tao Lin, Joel Becker, Sami Jawhar, Neev Parikh, Thomas Broadley, Lawrence Chan, Michael Chen, Josh Clymer, Jai Dhyani, et al. Re-bench: Evaluating frontier ai r&d capabilities of language model agents against human experts. arXiv preprint arXiv:2411.15114, 2024

    2024 arXiv

  72. [80]

    Understanding the effects of noise in text-to-sql: an examination of the bird-bench benchmark

    Niklas Wretblad, Fredrik Gordh Riseby, Rahul Biswas, Amin Ahmadi, and Oskar Holmström. Understanding the effects of noise in text-to-sql: an examination of the bird-bench benchmark. arXiv preprint arXiv:2402.12243, 2024

    2024 arXiv

  73. [81]

    Grok 2 beta release, 2024

    xAI. Grok 2 beta release, 2024. URL https://x.ai/news/grok-2

    2024

  74. [82]

    Grok 3 beta — the age of reasoning agents, 2024

    xAI. Grok 3 beta — the age of reasoning agents, 2024. URL https://x.ai/news/grok-3

    2024

  75. [83]

    Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments

    Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Jing Hua Toh, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processin...

    2024

  76. [84]

    Swe-agent: Agent-computer interfaces enable automated software engineering

    John Yang, Carlos Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems, 37:50528–50652, 2024

    2024

  77. [85]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2023

    2023

  78. [86]

    tau-bench: A bench- mark for tool-agent-user interaction in real-world domains

    Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik R Narasimhan. tau-bench: A bench- mark for tool-agent-user interaction in real-world domains. In The Thirteenth International Conference on Learning Representations, 2025. 14

    2025

  79. [87]

    Utboost: Rigorous evaluation of coding agents on swe-bench

    Boxi Yu, Yuxuan Zhu, Pinjia He, and Daniel Kang. Utboost: Rigorous evaluation of coding agents on swe-bench. ACL, 2025

    2025

  80. [88]

    Evaluating large language models at evaluating instruction following

    Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. Evaluating large language models at evaluating instruction following. ICLR, 2024

    2024

  81. [89]

    Cybench: A framework for evaluating cybersecurity capabilities and risks of language models

    Andy K Zhang, Neil Perry, Riya Dulepet, Joey Ji, Celeste Menders, Justin W Lin, Eliot Jones, Gashon Hussein, Samantha Liu, Donovan Jasper, et al. Cybench: A framework for evaluating cybersecurity capabilities and risks of language models. arXiv preprint arXiv:2408.08926, 2024

    2024 arXiv

  82. [90]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623, 2023

    2023

  83. [91]

    Don’t make your llm an evaluation benchmark cheater

    Kun Zhou, Yutao Zhu, Zhipeng Chen, Wentong Chen, Wayne Xin Zhao, Xu Chen, Yankai Lin, Ji-Rong Wen, and Jiawei Han. Don’t make your llm an evaluation benchmark cheater. arXiv preprint arXiv:2311.01964, 2023

    2023 arXiv

  84. [92]

    X-webarena-leaderboard,

    Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xi- anyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. X-webarena-leaderboard,

  85. [93]

    Webarena: A realistic web environment for build- ing autonomous agents

    Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for build- ing autonomous agents. In The Twelfth International Conference on Learning Representations, 2024

    2024

  86. [94]

    Software unit test coverage and adequacy.Acm computing surveys (csur), 29(4):366–427, 1997

    Hong Zhu, Patrick A V Hall, and John HR May. Software unit test coverage and adequacy.Acm computing surveys (csur), 29(4):366–427, 1997

    1997

  87. [95]

    Fuzzing: a survey for roadmap

    Xiaogang Zhu, Sheng Wen, Seyit Camtepe, and Yang Xiang. Fuzzing: a survey for roadmap. ACM Computing Surveys (CSUR), 54(11s):1–36, 2022

    2022

  88. [96]

    Cve-bench: A benchmark for ai agents’ ability to exploit real-world web application vulnerabilities

    Yuxuan Zhu, Antony Kellermann, Dylan Bowman, Philip Li, Akul Gupta, Adarsh Danda, Richard Fang, Conner Jensen, Eric Ihli, Jason Benn, et al. Cve-bench: A benchmark for ai agents’ ability to exploit real-world web application vulnerabilities. arXiv preprint arXiv:2503.17332, 2025

    2025 arXiv

  89. [97]

    Agent- as-a-judge: Evaluate agents with agents

    Mingchen Zhuge, Changsheng Zhao, Dylan Ashley, Wenyi Wang, Dmitrii Khizbullin, Yunyang Xiong, Zechun Liu, Ernie Chang, Raghuraman Krishnamoorthi, Yuandong Tian, et al. Agent- as-a-judge: Evaluate agents with agents. arXiv preprint arXiv:2410.10934, 2024

    arXiv

  90. [98]

    Can large language models transform computational social science? Computational Linguistics, 50 (1):237–291, 2024

    Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. Can large language models transform computational social science? Computational Linguistics, 50 (1):237–291, 2024. 15 A Limitation and Impact Statement Limitation. As the first study to systematic...

    2024

  91. [100]

    Our experience of developing agentic benchmarks

  92. [101]

    Best practices in existing agentic benchmarks (Table 3)

  93. [102]

    Lessons learned from issues of existing agentic benchmarks

  94. [103]

    group " ]) 3 df . iloc [5:10] = np . nan 4 res1 = PolyFit () ( df [[

    Domain-specific suggestions when we apply well-established techniques as evaluation methods. Table 4: Sources of items in ABC Question Existing Best Practice Lessons Learned Domain-Specific Suggestions Continued on next page 18 Table 4: Sources of items in ABC (Continued) O.a....

  95. [104]

    Guidelines: • The answer NA means that the abstract and introduction do not include the claims made in the paper

    Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: Our claimes in the abstract and introduction are justfied in the later sections and accurately reflect our paper’s con...

  96. [105]

    Limitations

    Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We included a limitation section in the first section of Appendix (Appendix A) Guidelines: • The answer NA means that the paper has no limitation whi...

  97. [106]

    Guidelines: • The answer NA means that the paper does not include theoretical results

    Theory Assumptions and Proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] Justification: This work does not propose new theories that need proofs. Guidelines: • The answer NA means t...

  98. [107]

    Our experimental results are reproducible with our provided experiment code

    Experimental Result Reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and da...

  99. [108]

    We include detailed justification to our data in Appendix D

    Open access to data and code Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: We provide open-sourced cod...

  100. [109]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experimental Setting/Details Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: Experiment parameters and det...

  101. [110]

    Our experiments contain no stochastic elements, so re-running an experiment yields identical outputs; there is therefore no run-to- run variance on which to base error bars

    Experiment Statistical Significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [NA] Justification: All reported numbers are deterministic. Our experim...

  102. [111]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experiments Compute Resources Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: We specify the computer...

  103. [112]

    Guidelines: 36 • The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics

    Code Of Ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? Answer: [Yes] Justification: We confirm that our paper conform, in every respect, with the NeurIPS Code of E...

  104. [113]

    • If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact

    Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: We discuss the potential sociatal impacts of our work in Appendix A Guidelines: • The answer NA means tha...

  105. [114]

    Guidelines: • The answer NA means that the paper poses no such risks

    Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: Our work does n...

  106. [115]

    Guidelines: • The answer NA means that the paper does not use existing assets

    Licenses for existing assets 37 Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [NA] Justification: Our work does ...

  107. [116]

    Guidelines: • The answer NA means that the paper does not release new assets

    New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: Our released code are well documented with READMEs. Guidelines: • The answer NA means that the paper does not releas...

  108. [117]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Crowdsourcing and Research with Human Subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? A...

  109. [118]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects 38 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approv...

  110. [2024]

    URL https://docs.google.com/spreadsheets/d/1M801lEpBbKSNwP-vDBkC_ pF7LdyGU1f_ufZb_NWNBZQ/edit?gid=0#gid=0

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.