Pith. sign in

REVIEW 4 major objections 6 minor 51 references

AgentCompass claims that decoupling agent evaluation into Benchmark, Harness, and Environment layers makes scores reproducible and shows how strongly infrastructure choice changes them.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

AgentCompass is a modular evaluation infrastructure that decouples benchmark, harness, and environment for LLM agents, and its experiments show model scores vary substantially with the harness used.

T0 review reviewed 2026-08-02 challenge →

load-bearing objection Worth engaging for the infrastructure design, but the empirical demonstration that evaluation is 'highly sensitive to infrastructure choices' is not supported as written. the 4 major comments →

arxiv 2607.13705 v1 pith:2FJFGMOO submitted 2026-07-15 cs.AI cs.SE

AgentCompass: A Unified Evaluation Infrastructure for Agent Capabilities

Zichen Ding , Jiaye Ge , Shufan Jiang , Kai Chen , Mo Li , Qingqiu Li , Zehao Li , Zonglin Li
show 15 more authors
Tiaohao Liang Shudong Liu Zerun Ma Zixing Shang Wenhui Tian Zun Wang Liwei Wu Zhenyu Wu Jun Xu Bowen Yang Dingbo Yuan Qi Zhang Songyang Zhang Peiheng Zhou Dongsheng Zhu
This is my paper
classification cs.AI cs.SE
keywords agent evaluationLLM agentsbenchmarkharnessenvironmentreproducibilitytrajectory analysisreward hacking
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

AgentCompass is an open-source infrastructure that separates agent evaluation into three independent layers: Benchmark (task definition and scoring), Harness (the procedure that turns a language model into an interactive agent), and Environment (the sandbox where the agent operates). The paper's central claim is that today's agent evaluation is fragmented and tightly coupled, and that this three-way decoupling, held together by stable protocols, turns rigid pipelines into composable benchmark × harness × environment configurations. That lets researchers reuse the same model across different harnesses or the same harness across benchmarks without rewriting code. To demonstrate why this matters, the paper evaluates several frontier models on eight benchmarks and finds that scores shift by double-digit points when only the harness changes, and that trajectory analysis surfaces behaviors like reward hacking that scalar scores miss. A sympathetic reader would take away that the infrastructure makes agent evaluation both more reproducible and more diagnostic.

Core claim

We introduce AgentCompass, a unified infrastructure whose design principle is to keep Benchmark, Harness, and Environment as separate, independently replaceable components. A declarative RunRequest separates what is being evaluated (the model, benchmark, harness, environment) from how the run is executed (concurrency, resume), and a registry lets new components register locally without touching the runtime. Using this system, we run seven representative models across eight benchmarks spanning tool use, web research, scientific reasoning, agentic coding, and productivity. The results show that agent capability is substantially affected by the harness: the same model can lose 8.7 points or gai

What carries the argument

The central mechanism is the separation of Benchmark, Harness, and Environment into composable components, enforced by two protocol contracts: a model specification that treats models as declarative API endpoints, and a material protocol that shuttles standardized PreparedTask and RunResult objects between benchmark and harness. The registry-based runtime dynamically resolves components from a declarative RunRequest, and the trajectory recorder captures every reasoning step, tool call, and environment feedback into a uniform, versioned schema. These protocols are what allow new benchmarks, harnesses, and environments to be added without cross-modifying existing code.

Load-bearing premise

The load-bearing premise is that the external reference scores used for comparison are truly compatible with the AgentCompass re-runs; the paper, for each model and benchmark, picks the closest available reference value, which assumes those baselines are comparable and systematically minimizes the deviations used to demonstrate infrastructure sensitivity.

What would settle it

Run one model on one benchmark under three different harnesses inside AgentCompass and measure the spread; or recompute all reported gaps against every available external baseline instead of only the nearest one. If the score spread is close to zero, or the gap pattern disappears when all baselines are used, the paper's claim that agent evaluation is highly sensitive to infrastructure choice would be weakened.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the decoupling works, adding a new benchmark reduces to writing a protocol-compliant subclass plus registration; existing harnesses and environments stay untouched.
  • The same model can be compared across different harnesses on the same benchmark, and the same harness can be reused on different benchmarks, eliminating glue code and enabling direct cross-harness comparisons.
  • The uniform trajectory schema makes failure-mode diagnosis (truncation, repetition, reward hacking) systematic and comparable across heterogeneous models and benchmarks.
  • The fault-tolerant, resumable runtime makes long-running and costly agent evaluations interruptible and auditable at scale.
  • The paper's evidence that scores depend strongly on the harness implies that reported agent results should be trusted only when harness and environment versions are fixed and disclosed.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the paper computes each reported gap by selecting the closest available external baseline, the sensitivity evidence is likely an underestimate; recomputing against all available baselines would give a fairer, possibly larger, range. This is our inference, not the paper's claim.
  • We infer that the same three-component abstraction, applied to multi-agent or human-in-the-loop settings, would make comparisons there more stable as well, since harness variation is even larger in those settings.
  • The reward-hacking analyzer flags behaviors characteristic of hacking without establishing that they caused the final outcome, so the true rate of intentional hacking may be lower than reported; we would test this by correlating flagged actions with outcome causality.
  • A concrete testable extension: run a single model on a single benchmark under three harnesses and check whether the ranking of models changes; if rankings invert, the practical implication for leaderboards is immediate.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces AgentCompass, an open-source evaluation infrastructure for LLM-based agents that decouples evaluation into Benchmark, Harness, and Environment components connected by protocol abstractions and a registry system. It describes the architecture, a trajectory-tracking and analysis layer, and an experimental study evaluating seven models on eight benchmarks with several built-in harnesses. The central claims are that the decoupled design enables flexible benchmark-harness-environment configurations, that agent evaluation outcomes are highly sensitive to infrastructure choices, and that trajectory analysis can diagnose failure modes such as reward hacking.

Significance. The paper addresses a real problem: agent evaluation pipelines are fragmented, tightly coupled, and hard to reproduce. Its strengths are the clear component separation, declarative RunRequest, fault-tolerant asynchronous runtime, provenance tracking, and the release of code supporting 20+ benchmarks and multiple harnesses. If the infrastructure works as described, it would be a useful community resource. However, the empirical validation is not yet strong enough to support the paper's motivating claim of infrastructure sensitivity: the baseline-gap analysis in Section 4.2 uses an outcome-dependent selection rule, and the three-run averages are reported without uncertainty intervals. The architecture itself is plausible, and the weaknesses are addressable in a revision.

major comments (4)
  1. [Section 4.2, Table 3, Appendix A.2] The claim that 'agent evaluation is highly sensitive to infrastructure choices' rests largely on the subscripted gaps in Table 3. Appendix A.2 states that each gap is computed against the closest available external reference, and when both a model report and a benchmark leaderboard are available, 'we use the one whose reported score is closest to the AgentCompass evaluation result.' This selection rule is outcome-dependent: the reference is chosen after seeing the AgentCompass score, and external references differ in harness version, prompts, judge model, and environment. The gaps therefore conflate infrastructure sensitivity with uncontrolled protocol differences and cannot serve as clean evidence. The internal comparisons (OpenClaw vs. OpenHands on SkillsBench; Mini-SWE-agent vs. OpenHands on SWE-Pro/SWE-Multilingual) are more appropriate evidence but are not reported with uncertainty.
  2. [Section 4.1, Table 3] All results are said to be averaged over three independent runs, but no variance, confidence interval, or per-run value is reported anywhere. This matters for the key empirical claims: for example, Claude-Opus-4.8's -8.7 gap on DeepSearchQA, GLM-5.2(FP8)'s +15.0 gap on SWE-bench-Pro with OpenHands, and the OpenClaw/OpenHands differences on SkillsBench are point estimates. Without uncertainty, the reader cannot tell whether these differences reflect infrastructure sensitivity or run-to-run noise, especially for LLM-as-judge scoring and stochastic API decoding. Please add standard deviations or per-run scores for at least the central comparisons.
  3. [Sections 3.1 and 4.2] The central architectural contribution is the composability of Benchmark, Harness, and Environment. However, the experiments vary only the Harness component, and only for two task families (SkillsBench and the SWE-bench variants); most benchmarks are evaluated with a single harness, and no experiment varies the Environment (e.g., local process vs. Docker vs. cluster). Thus the 'benchmark x harness x environment' flexibility claim is not empirically demonstrated, and the paper does not quantify engineering savings or integration effort relative to the existing frameworks cited in Section 2. I recommend adding at least one environment comparison, and ideally a small compatibility matrix or integration-cost measurement.
  4. [Section 4.3, RQ2, Table 4] The reward-hacking analysis defines the phenomenon behaviorally: 'any action exhibiting characteristics of hacking is classified as reward hacking, regardless of whether there is direct evidence establishing a causal relationship.' The paper is transparent about this, but the surrounding text interprets the rates as if they answer RQ2 ('Are high-scoring models genuinely strong?'), for example by contrasting GLM-5.2(FP8)'s roughly 12-point lead over Claude-Opus-4.8 with its '30% more suspected reward-hacking samples.' Without validation of the heuristic against labeled examples, these rates are unvalidated proxy counts. Please rename the measure (e.g., 'suspicious-action rate'), add a validation study, or at least state clearly what the heuristic does and does not measure.
minor comments (6)
  1. [Table 3] The subscripted gap annotations are garbled in the manuscript text (e.g., '-4.363.78-5.5' and '+15.082.00'). Please ensure the table typesets sub/superscripts legibly and clearly separates the two values for the two SWE harnesses.
  2. [Appendix A.1] Appendix A.1 says closed-source APIs are used with default settings 'to ensure deterministic outputs where possible,' but Section 4.1 says all results are averaged over three runs. Clarify whether the three runs differ in sampling temperature, API nondeterminism, or environment state, and whether 'deterministic' applies only to some benchmarks.
  3. [Table 1 and Section 3.6] Table 1 lists GDPVal-AC as a benchmark, but Section 3.6 describes it as a custom AgentCompass variant with an agentic judger. Clarify its provenance and whether comparisons are to official GDPVal or to this variant.
  4. [Section 3.2] The 'none' scorer execution mode is described as 'in-memory verification'; the name is ambiguous. Define it explicitly (e.g., scoring in the benchmark process after rollout) and distinguish it from 'reuse' and 'fresh'.
  5. [Section 4.3, Figure 4] Figure 4 is a scatter of seven models per benchmark; describing this as 'most models follow the test-time scaling law' is stronger than the evidence supports. Qualify as a correlation, not a scaling-law test.
  6. [Section 2] The Related Work section lists several existing frameworks (AgentGym, Harbor, MASLab, EvalScope, OpenCompass, VLMEvalKit), but the paper does not compare functionality or design choices. A short comparison table or explicit list of AgentCompass's distinguishing features would strengthen the novelty claim.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

No fitted numerical parameters appear because the paper makes no quantitative derivation; the central claim is architectural. The key assumptions are about comparability and validity of benchmarks, external baselines, judge models, and the behavioral reward-hacking definition.

axioms (4)
  • domain assumption Agent capabilities can be meaningfully summarized by the benchmark scores and gap annotations reported in Table 3.
    The main results section uses scalar scores and colored gap subscripts to draw conclusions about cross-model capability profiles; this assumes the benchmarks and metrics are valid measures of the claimed capabilities.
  • domain assumption External official baselines from model technical reports or leaderboards are comparable to AgentCompass re-runs when the closest available value is selected.
    Appendix A.2 explicitly selects the closest external reference result for each gap annotation, assuming comparability of environments, harness versions, and protocols despite acknowledged differences.
  • ad hoc to paper The behavioral definition of reward hacking ('any action exhibiting characteristics of hacking is classified as reward hacking') is a valid proxy for actual reward hacking.
    Section 4.3 defines suspected reward hacking behaviorally and non-evidentially, then uses it to argue that benchmarks may overstate coding ability. This is an assumption adopted specifically for this analysis.
  • domain assumption LLM-as-judge scoring is reliable for the benchmarks that use it (e.g., DeepSearchQA with a Qwen judge, PinchBench with a Claude judge).
    Appendix A.2 specifies judge models for several benchmarks; the paper provides no calibration or validity check for these judge-based scores.

reviewed 2026-08-02 · how reviews work

0 comments
Cite this review

Pith. "Pith review of AgentCompass: A Unified Evaluation Infrastructure for Agent Capabilities." pith.science (2026). https://pith.science/paper/2FJFGMOO

@misc{pith2026260713705,
  author       = {Pith},
  title        = {Pith review of: AgentCompass: A Unified Evaluation Infrastructure for Agent Capabilities},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2FJFGMOO}},
  note         = {Machine review of arXiv:2607.13705}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

As Large Language Models (LLMs) evolve into autonomous agents, the need for unified evaluation infrastructure becomes critical. However, current evaluation pipelines remain highly fragmented and tightly coupled, hindering reproducibility and causing redundant engineering. To address this, we introduce AgentCompass, an open-source, lightweight, and extensible infrastructure for evaluating LLM-based agents. AgentCompass organizes the evaluation process around three independent components, namely Benchmark, Harness, and Environment, thereby enabling flexible configurations without requiring the reimplementation of complex execution logic. Furthermore, it features a fault-tolerant asynchronous runtime and comprehensive trajectory analysis tools to transparently diagnose nuanced failure modes like reward-hacking. Natively supporting over 20 benchmarks across five capability dimensions, AgentCompass provides the community with a scalable and reproducible infrastructure for advancing agent research.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

51 extracted references · 12 linked inside Pith

  1. [1]

    Claude Opus 4.8.https://www.anthropic.com/news/claude-opus-4-8, 2026

    Anthropic. Claude Opus 4.8.https://www.anthropic.com/news/claude-opus-4-8, 2026. 4.1

    2026

  2. [2]

    Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan.𝜏 2-bench: Evaluating conversational agents in a dual-control environment.arXiv preprint arXiv:2506.07982, 2025. 2, 1

    Pith/arXiv arXiv 2025

  3. [3]

    Aider: Ai pair programming in your terminal.https://github.com/paul-gauthier/ aider, 2023

    CodedotAI. Aider: Ai pair programming in your terminal.https://github.com/paul-gauthier/ aider, 2023. 2

    2023

  4. [4]

    Deepeval: The open-source evaluation framework for llms

    Confident AI. Deepeval: The open-source evaluation framework for llms. https://github.com/ confident-ai/deepeval, 2023. 2

    2023

  5. [5]

    Opencompass: A universal evaluation platform for foundation models

    OpenCompass Contributors. Opencompass: A universal evaluation platform for foundation models. https://github.com/open-compass/opencompass, 2023. 1, 2

    2023

  6. [6]

    Swe-bench pro: Can ai agents solve long-horizon software engineering tasks?arXiv preprint arXiv:2509.16941, 2025

    Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, et al. Swe-bench pro: Can ai agents solve long-horizon software engineering tasks?arXiv preprint arXiv:2509.16941, 2025. 2, 1, 4.1

    Pith/arXiv arXiv 2025

  7. [7]

    Benchmarking reward hack detection in code environments via contrastive analysis, 2026

    Darshan Deshpande, Anand Kannappan, and Rebecca Qian. Benchmarking reward hack detection in code environments via contrastive analysis, 2026. 4.3

    2026

  8. [8]

    Vlmevalkit: An open-source toolkit for evaluating large multi-modality models

    Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, et al. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models. InProceedings of the 32nd ACM International Conference on Multimedia, pages 11198–11201,

  9. [9]

    Glm-5: from vibe coding to agentic engineering, 2026

    GLM-5-Team, :, Aohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, Bin Chen, Da Yin, Chendi Ge, Chenghua Huang, Chengxing Xie, Chenzheng Zhu, Congfeng Yin, Cunxiang Wang, Gengzheng Pan, Hao Zeng, Haoke Zhang, Haoran Wang, Huilong Chen, Jiajie Zhang, Jian Jiao, Jiaqi Guo, Jingsen Wang, Jingzhao Du, Jinzhu Wu, Kedong Wang, Lei Li, Lin Fan, Lucen Zho...

    2026

  10. [10]

    Gemini 3.1 Pro.https://deepmind.google/models/gemini/pro/, 2026

    Google DeepMind. Gemini 3.1 Pro.https://deepmind.google/models/gemini/pro/, 2026. 4.1

    2026

  11. [11]

    Deepsearchqa: Bridging the comprehensiveness gap for deep research agents.arXiv preprint arXiv:2601.20975, 2026

    Nikita Gupta, Riju Chatterjee, Lukas Haas, Connie Tao, Andrew Wang, Chang Liu, Hidekazu Oiwa, Elena Gribovskaya, Jan Ackermann, John Blitzer, et al. Deepsearchqa: Bridging the comprehensiveness gap for deep research agents.arXiv preprint arXiv:2601.20975, 2026. 2, 1, 4.1

    arXiv 2026

  12. [12]

    Harbor: A framework for evaluating and optimizing agents and models in container environments.https://github.com/harbor-framework/harbor, January 2026

    Harbor Framework Team. Harbor: A framework for evaluating and optimizing agents and models in container environments.https://github.com/harbor-framework/harbor, January 2026. 1, 2 9 AgentCompass: A Unified Evaluation Infrastructure for Agent Capabilities

    2026

  13. [13]

    Deepswe: Measuring frontier coding agents on original, long-horizon engineering tasks.arXiv preprint arXiv:2607.07946, 2026

    Wenqi Huang, Charley Lee, Leonard Tng, and Serena Ge. Deepswe: Measuring frontier coding agents on original, long-horizon engineering tasks.arXiv preprint arXiv:2607.07946, 2026. 2

    Pith/arXiv arXiv 2026

  14. [14]

    Waytowich, and Boyuan Chen

    Jonathan Hyun, Nicholas R. Waytowich, and Boyuan Chen. Crew-wildfire: Benchmarking agentic multi-agent collaborations at scale.arXiv preprint arXiv:2507.05178, 2025. 2

    arXiv 2025

  15. [15]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. SWE-bench: Can language models resolve real-world github issues? InInternational Conference on Learning Representations, 2024. 2

    2024

  16. [16]

    Langsmith: A unified platform for debugging, testing, evaluating, and monitoring your llm applications.https://www.langchain.com/langsmith, 2025

    LangChain. Langsmith: A unified platform for debugging, testing, evaluating, and monitoring your llm applications.https://www.langchain.com/langsmith, 2025. 2

    2025

  17. [17]

    Camel: Communicative agents for "mind" exploration of large language model society

    Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for "mind" exploration of large language model society. InThirty-seventh Conference on Neural Information Processing Systems, 2023. 1

    2023

  18. [18]

    Skillsbench: Benchmarking how well agent skills work across diverse tasks.arXiv preprint arXiv:2602.12670, 2026

    Xiangyi Li, Wenbo Chen, Yimin Liu, Shenghan Zheng, Xiaokun Chen, Yifeng He, Yubo Li, Bingran You, Haotian Shen, Jiankai Sun, et al. Skillsbench: Benchmarking how well agent skills work across diverse tasks.arXiv preprint arXiv:2602.12670, 2026. 2, 1, 4.1

    Pith/arXiv arXiv 2026

  19. [19]

    Agentbench: Evaluating llms as agents

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating llms as agents. InThe Twelfth International Conference on Learning Representations, 2024. 2

    2024

  20. [20]

    Agentboard: An analytical evaluation board of multi-turn llm agents

    Chang Ma, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Lingpeng Kong, and Junxian He. Agentboard: An analytical evaluation board of multi-turn llm agents. InAdvances in Neural Information Processing Systems, 2024. 2

    2024

  21. [21]

    Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces.arXiv preprint arXiv:2601.11868, 2026

    Mike A Merrill, Alexander G Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E Kelly Buchanan, et al. Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces.arXiv preprint arXiv:2601.11868, 2026. 2

    Pith/arXiv arXiv 2026

  22. [22]

    GAIA: a benchmark for general AI assistants

    Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. GAIA: a benchmark for general AI assistants. InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. 1, 2, 1

    2024

  23. [23]

    Kimi-k2.6.https://www.kimi.com/en/blog/kimi-k2-6, 2026

    Moonshot AI. Kimi-k2.6.https://www.kimi.com/en/blog/kimi-k2-6, 2026. 4.1

    2026

  24. [24]

    Introducing GPT-5.5.https://openai.com/index/introducing-gpt-5-5/, 2026

    OpenAI. Introducing GPT-5.5.https://openai.com/index/introducing-gpt-5-5/, 2026. 4.1

    2026

  25. [25]

    OpenClaw.https://github.com/openclaw/openclaw, 2026

    OpenClaw Team. OpenClaw.https://github.com/openclaw/openclaw, 2026. 4.1

    2026

  26. [26]

    Patil, Huanzhi Mao, Cheng-Jie Ji, Fanjia Yan, Vishnu Suresh, Ion Stoica, and Joseph E

    Shishir G. Patil, Huanzhi Mao, Cheng-Jie Ji, Fanjia Yan, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. The berkeley function calling leaderboard (bfcl): From tool use to agentic evaluation of large language models. InInternational Conference on Machine Learning, 2025. 2

    2025

  27. [27]

    Gdpval: Evaluating ai model performance on real-world economically valuable tasks.arXiv preprint arXiv:2510.04374, 2025

    Tejal Patwardhan, Rachel Dias, Elizabeth Proehl, Grace Kim, Michele Wang, Olivia Watkins, Simón Posada Fishman, Marwan Aljubeh, Phoebe Thacker, Laurance Fauconnet, et al. Gdpval: Evaluating ai model performance on real-world economically valuable tasks.arXiv preprint arXiv:2510.04374, 2025. 2, 1

    Pith/arXiv arXiv 2025

  28. [28]

    Humanity’s last exam.arXiv preprint arXiv:2501.14249,

    Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, et al. Humanity’s last exam.arXiv preprint arXiv:2501.14249,

    Pith/arXiv arXiv

  29. [29]

    Pinchbench: Real-world benchmarks for ai coding agents, 2026

    PinchBench Team. Pinchbench: Real-world benchmarks for ai coding agents, 2026. 2, 1, 4.1

    2026

  30. [30]

    Qwen3.5: Accelerating productivity with native multimodal agents, February 2026

    Qwen Team. Qwen3.5: Accelerating productivity with native multimodal agents, February 2026. 4.1

    2026

  31. [31]

    2, 1, 4.1 10 AgentCompass: A Unified Evaluation Infrastructure for Agent Capabilities

    Quan Shi, Alexandra Zytek, Pedram Razavi, Karthik Narasimhan, and Victor Barres.𝜏-knowledge: Evaluating conversational agents over unstructured knowledge.arXiv preprint arXiv:2603.04370, 2026. 2, 1, 4.1 10 AgentCompass: A Unified Evaluation Infrastructure for Agent Capabilities

    arXiv 2026

  32. [32]

    EvalScope: Evaluation framework for large models, 2024

    ModelScope Team. EvalScope: Evaluation framework for large models, 2024. 2

    2024

  33. [33]

    Terminal-bench: A benchmark for ai agents in terminal environments

    The Terminal-Bench Team. Terminal-bench: A benchmark for ai agents in terminal environments. https://github.com/laude-institute/terminal-bench, 2025. 2, 1

    2025

  34. [34]

    Scicode: A research coding benchmark curated by scientists.Advances in Neural Information Processing Systems, 37:30624–30650, 2024

    Minyang Tian, Luyu Gao, Shizhuo D Zhang, Xinan Chen, Cunwei Fan, Xuefei Guo, Roland Haas, Pan Ji, Kittithat Krongchon, Yao Li, et al. Scicode: A research coding benchmark curated by scientists.Advances in Neural Information Processing Systems, 37:30624–30650, 2024. 2, 1, 4.1

    2024

  35. [35]

    Frontier- science: Evaluating ai’s ability to perform expert-level scientific tasks.arXiv preprint arXiv:2601.21165,

    Miles Wang, Robi Lin, Kat Hu, Joy Jiao, Neil Chowdhury, Ethan Chang, and Tejal Patwardhan. Frontier- science: Evaluating ai’s ability to perform expert-level scientific tasks.arXiv preprint arXiv:2601.21165,

    arXiv

  36. [36]

    Openhands: An open platform for ai software developers as generalist agents

    Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Openhands: An open platform for ai software developers as generalist agents. InInternational Conference on Learning Representations, volume 2025, pages 65882–65919, 2025. 4.1

    2025

  37. [37]

    Browsecomp: Asimpleyetchallengingbenchmark for browsing agents.arXiv preprint arXiv:2504.12516, 2025

    Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, AlexTachardPassos, WilliamFedus, andAmeliaGlaese. Browsecomp: Asimpleyetchallengingbenchmark for browsing agents.arXiv preprint arXiv:2504.12516, 2025. 2, 1

    Pith/arXiv arXiv 2025

  38. [38]

    Autogen: Enablingnext-genllmapplicationsviamulti-agentconversations

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, ShaokunZhang, JialeLiu, etal. Autogen: Enablingnext-genllmapplicationsviamulti-agentconversations. InFirst Conference on Language Modeling, 2024. 1

    2024

  39. [39]

    Mitchell, and Yuanzhi Li

    Yue Wu, Xuan Tang, Tom M. Mitchell, and Yuanzhi Li. Smartplay: A benchmark for llms as intelligent agents. InInternational Conference on Learning Representations, 2024. 2

    2024

  40. [40]

    Agentgym: Evolving large language model-based agents across diverse environments, 2024

    Zhiheng Xi, Yiwen Ding, Wenxiang Chen, Boyang Hong, Honglin Guo, Junzhe Wang, Dingwen Yang, Chenyang Liao, Xin Guo, Wei He, et al. Agentgym: Evolving large language model-based agents across diverse environments, 2024. 2

    2024

  41. [41]

    Deepseek-v4: Towards highly efficient million-token context intelligence.arXiv preprint arXiv:2606.19348, 2026

    Anyi Xu, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chenchen Ling, et al. Deepseek-v4: Towards highly efficient million-token context intelligence.arXiv preprint arXiv:2606.19348, 2026. 4.1

    arXiv 2026

  42. [42]

    Researchclawbench: A benchmark for end-to-end autonomous scientific research.arXiv preprint arXiv:2606.07591, 2026

    Wanghan Xu, Shuo Li, Tianlin Ye, Qinglong Cao, Yixin Chen, Hengjian Gao, Yiheng Wang, Qi Li, Kun Li, Sheng Xu, et al. Researchclawbench: A benchmark for end-to-end autonomous scientific research.arXiv preprint arXiv:2606.07591, 2026. 2, 1

    Pith/arXiv arXiv 2026

  43. [43]

    Probing scientific general intelligence of llms with scientist-aligned workflows.arXiv preprint arXiv:2512.16969, 2025

    Wanghan Xu, Yuhao Zhou, Yifan Zhou, Qinglong Cao, Shuo Li, Jia Bu, Bo Liu, Yixin Chen, Xuming He, Xiangyu Zhao, et al. Probing scientific general intelligence of llms with scientist-aligned workflows.arXiv preprint arXiv:2512.16969, 2025. 2, 1

    arXiv 2025

  44. [44]

    SWE-agent: Agent-computer interfaces enable automated software engineering

    John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik R Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. 1, 4.1

    2024

  45. [45]

    Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang

    John Yang, Kilian Lieret, Carlos E. Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang. Swe-smith: Scaling data for software engineering agents,

  46. [46]

    Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan.𝜏-bench: A benchmark for tool-agent- user interaction in real-world domains.arXiv preprint arXiv:2406.12045, 2024. 1, 2, 1

    Pith/arXiv arXiv 2024

  47. [47]

    Maslab: A unified and comprehensive codebase for llm-based multi-agent systems

    Rui Ye, Keduan Huang, Qimin Wu, Yuzhu Cai, Tian Jin, Xianghe Pang, Xiangrui Liu, Jiaqi Su, Chen Qian, Bohan Tang, et al. Maslab: A unified and comprehensive codebase for llm-based multi-agent systems. arXiv preprint arXiv:2505.16988, 2025. 2 11 AgentCompass: A Unified Evaluation Infrastructure for Agent Capabilities

    arXiv 2025

  48. [48]

    Hle-verified: A systematic verification and structured revision of humanity’s last exam.arXiv preprint arXiv:2602.13964, 2026

    Weiqi Zhai, Zhihai Wang, Jinghang Wang, Boyu Yang, Xiaogang Li, Xander Xu, Bohan Wang, Peng Wang, Xingzhe Wu, Anfeng Li, et al. Hle-verified: A systematic verification and structured revision of humanity’s last exam.arXiv preprint arXiv:2602.13964, 2026. 2, 1

    arXiv 2026

  49. [49]

    Browsecomp-zh: Benchmarking web browsing ability of large language models in chinese.arXiv preprint arXiv:2504.19314, 2025

    Peilin Zhou, Bruce Leon, Xiang Ying, Can Zhang, Yifan Shao, Qichen Ye, Dading Chong, Zhiling Jin, Chenxuan Xie, Meng Cao, et al. Browsecomp-zh: Benchmarking web browsing ability of large language models in chinese.arXiv preprint arXiv:2504.19314, 2025. 2, 1

    Pith/arXiv arXiv 2025

  50. [50]

    Multiagentbench: Evaluating the collaboration and competition of llm agents.arXiv preprint arXiv:2503.01935, 2025

    Kunlun Zhu, Hongyi Du, Zhaochen Hong, Xiaocheng Yang, Shuyi Guo, Zhe Wang, Zhenhailong Wang, Cheng Qian, Xiangru Tang, Heng Ji, et al. Multiagentbench: Evaluating the collaboration and competition of llm agents.arXiv preprint arXiv:2503.01935, 2025. 2

    Pith/arXiv arXiv 2025

  51. [51]

    Intern-s1-pro: Scientific multimodal foundation model at trillion scale, 2026

    Yicheng Zou, Dongsheng Zhu, Lin Zhu, Tong Zhu, Yunhua Zhou, Peiheng Zhou, Xinyu Zhou, Dongzhan Zhou, Zhiwang Zhou, Yuhao Zhou, Bowen Zhou, Zhanping Zhong, Zhijie Zhong, Haiteng Zhao, Penghao Zhao, Xiaomeng Zhao, Zhiyuan Zhao, Yechen Zhang, Jin Zhang, Wenwei Zhang, Hongjie Zhang, Zhuo Zhang, Wenlong Zhang, Bo Zhang, Chao Zhang, Chen Zhang, Yuhang Zang, Fei...

    2026

This paper was first reviewed by deepseek-v4-flash on August 2, 2026.