Pith. sign in

REVIEW 3 major objections 8 minor 4 cited by

Frontier AI models can apply rules they are given but rarely discover them from experience, a four-environment benchmark shows.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 04:04 UTC pith:OOWKQSR2

load-bearing objection A genuinely useful benchmark for long-horizon inductive agent evaluation, with an overgeneralized 'inductive bottleneck' claim that the w/ rules control only partly supports. the 3 major comments →

arxiv 2602.05843 v2 pith:OOWKQSR2 submitted 2026-02-05 cs.CL

OdysseyArena: Benchmarking Large Language Models For Long-Horizon, Active and Inductive Interactions

Hang Yan , Fangzhi Xu , Qiushi Sun , Jinyang Wu , Zixian Huang , Muye Huang , Jingyang Gong , Zichen Ding
show 9 more authors
Kanzhi Cheng Yian Wang Xinyu Che Zeyi Sun Jian Zhang Zhangyue Yin Haoran Luo Ben Kao Qika Lin
This is my paper
classification cs.CL
keywords LLM agentsinductive reasoningbenchmarklong-horizon interactionworld model inductiontransition functionactive explorationrule discovery
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

OdysseyArena argues that evaluations of LLM agents have tested deduction—executing tasks from explicitly stated rules—and largely ignored induction, where an agent must infer the hidden transition laws that govern an environment from its own trial and error. The paper builds four interactive environments whose dynamics instantiate four structural primitives: Boolean logic, stochastic continuous dynamics, periodic temporal patterns, and relational dependency graphs. Across 120 standardized tasks with horizons up to 200 steps, every frontier model succeeds when the rules are handed to them but performs far worse when they must discover the same rules, and many never surpass random action. The authors conclude that the binding constraint on autonomous discovery is inductive capacity, not task difficulty or instruction-following. If right, evaluation practice and agent architecture should shift toward first-person exploration and world-model induction.

Core claim

The central claim is that LLM agents are strong deductive reasoners but weak inductive reasoners: given explicit transition rules they approach ceiling performance, but with the same rules hidden they plateau at a fraction of that, and additional interaction steps do not close the gap. The paper locates the bottleneck in the discovery step—the agent's ability to hypothesize and test latent environment mechanics—rather than in the complexity of applying them. This is supported by the w/ versus w/o rules contrast and by behavioral analysis of trajectories, which shows persistent action loops, error credit-assignment failures, and long-horizon dependence decay.

What carries the argument

The central object is the environment transition function T, defined by (s_{t+1}, r_t) = T(s_t, a_t), which encodes the latent regularities an agent must infer. OdysseyArena decomposes T into four orthogonal structural primitives—discrete Boolean rules, continuous stochastic dynamics, periodic temporal patterns, and relational graph structures—each realized as a lightweight interactive environment. The benchmark's diagnostic lever is the w/ versus w/o rules contrast: identical tasks run with the latent rules exposed or hidden, which is meant to isolate inductive capacity from task logic.

Load-bearing premise

The load-bearing premise is that revealing the latent rules leaves everything else unchanged, so the w/ versus w/o rules gap isolates inductive discovery; Energy Dispatch (Table 4) shows Gemini 3 Pro Preview and GPT-5 scoring lower with rules (16.67% and 13.33%) than without (30.00% and 23.33%), so that premise fails for at least one environment.

What would settle it

Run the Energy Dispatch environment with all latent efficiency periods and constraints spelled out in the prompt: if Gemini 3 Pro Preview and GPT-5 still score near 16.67% and 13.33% under full rule disclosure, then the bottleneck in that environment is not rule discovery but long-horizon constrained optimization, and the paper's general inductive-bottleneck inference would need to be scoped or revised.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Scaling model size and reasoning effort improves deductive compliance but does not remove the inductive bottleneck; the best proprietary model still fails most inductive tasks.
  • Giving agents explicit rules lifts success to near-ceiling in three environments, so future gains depend on closing the discovery gap rather than on better instruction-following.
  • Extending the interaction budget beyond an initial exploration phase yields negligible gains, meaning the limitation is rule synthesis, not interaction volume.
  • Failure analysis attributes most errors to action loops and credit-assignment breakdowns, pointing to memory and hypothesis-testing mechanisms as the targets for improvement.
  • The Challenge suite shows the gap widens with horizons beyond 200 steps, indicating error accumulation compounds the inductive deficit.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the w/ versus w/o contrast holds, the same tasks could double as a training signal: environments could reward explicit rule hypotheses, not just final success, and the benchmark could measure inductive efficiency—how quickly an agent identifies the rule—rather than only whether it succeeds.
  • The Energy Dispatch anomaly—models scoring lower with rules than without—suggests that in multi-objective, periodic settings the bottleneck may be long-horizon constrained optimization rather than rule discovery; a clean test would run that environment with full rule disclosure and perfect rule recall.
  • A direct extension would turn the four primitives into a compositional curriculum and measure whether agents that discover rule set A transfer more quickly to rule set B, testing whether the bottleneck is per-environment search or a general inductive capability.
  • The step-saturation curves imply a diagnostic metric separate from final success: success per interaction step, which would distinguish agents that solve tasks from those that stumble on solutions.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 8 minor

Summary. The paper introduces OdysseyArena, a benchmark suite of four interactive environments (Turn On Lights, AI Trading, Energy Dispatch, Repo System) designed to evaluate long-horizon, active, and inductive interaction with latent transition dynamics. The authors instantiate four structural primitives, provide OdysseyArena-Lite (120 standardized tasks) and OdysseyArena-Challenge (extreme horizons), and evaluate 15+ LLMs. The main empirical claim, stated in the abstract and sharpened in §5.2, is that frontier LLMs are proficient at deductive rule-following but deficient at inductively discovering environment rules, as evidenced by a w/ rules versus w/o rules comparison. The paper also reports human performance, complexity scaling, action-loop analyses, and token-efficiency results. Code and data are made publicly available.

Significance. If the central claim held, this would be a useful contribution: it targets an underexplored capability—autonomous discovery of latent environment dynamics—and provides deterministic, reproducible tasks plus a concrete control (rules provided vs. not provided) for separating discovery from execution. The paper's strengths include public code/data, deterministic task metadata, evaluation of 15+ models with multiple reasoning efforts, and a sensible decomposition of transition dynamics into four primitives. However, the central inference is currently overgeneralized: the w/ rules control fails in one of the four environments, and the human reference scores are undermined by low inter-annotator agreement. The benchmark itself is promising, but the headline conclusion needs to be either re-scoped or supported by additional controls.

major comments (3)
  1. [§5.2 and Table 4] The central claim that LLMs are 'strong deductive reasoners, weak inductive reasoners' rests on the w/ rules versus w/o rules contrast. Table 4 shows the intended pattern in Turn On Lights and Repo System, but not in Energy Dispatch: Gemini 3 Pro Preview drops from 30.00 Avg@4 without rules to 16.67 with rules, GPT-5 drops from 23.33 to 13.33, and most open models remain at 0.00 in both conditions. If explicit rules remove the discovery requirement, performance should rise substantially; instead it falls or stays flat. This indicates that in Energy Dispatch the bottleneck may be multi-objective optimization, long-horizon planning, or prompt complexity rather than rule discovery. Figure 4, cited in §5.2, plots only Turn On Lights, so the visual evidence is narrower than the general conclusion drawn in the abstract and §7. The authors should either provide a rule-informed optimal/oracle po
  2. [§5.2 / Appendix B.1] The w/ rules condition is not specified in sufficient detail. The main text and Appendix B.1 report results in Table 4 but never state exactly what information the model receives for each environment. For Energy Dispatch, is the full latent efficiency vector E_t provided for every day, or only the functional form (e.g., 'wind and solar have hidden periods')? For AI Trading, is the exact matrix W given? Without this specification, the control is not reproducible, and the surprising Energy Dispatch drop cannot be diagnosed (e.g., prompt-format confusion, overly verbose rule dumps, or rules that conflict with the model's learned heuristics). The paper should describe the w/ rules prompts fully and, if possible, analyze token/context effects.
  3. [Table 7 / §D.5] Human performance is used as a reference point in Table 2 and §6.1, but the reported inter-annotator reliability is low: Fleiss' κ = 0.42 for Turn On Lights, 0.40 for Energy Dispatch, 0.18 for Repo System, and ICC = 0.12 for AI Trading. These values are far below the conventional threshold for reliable labels, especially for Repo System and AI Trading. The manuscript should report agreement on the binary success metric used in Table 2, provide consensus-based human scores, or discuss why low agreement does not affect the aggregate human success rates. As written, the 'humans achieve perfect success (30/30)' claim in §6.1 is not backed by stable human annotations.
minor comments (8)
  1. [Table 6] The column header 'Lete' should be 'Lite'.
  2. [Appendix D.5] Typographical error: 'Repo Syetem' in the appendix heading should be 'Repo System'.
  3. [References] The reference 'V odrahalli' has a stray space; should be 'Vodrahalli'.
  4. [Author affiliations] 'Nanyang Techonological University' should be 'Nanyang Technological University'.
  5. [Figures 5 and 10] The grayscale task-success maps are hard to read; using distinct symbols or hatched patterns for human/LLM/unsolved cells would improve clarity.
  6. [Figure 6 / Appendix B.3] The 'random' baseline is mentioned but never defined. Specify the random policy (e.g., uniform over actions) and report its variance, since several models underperform it.
  7. [§3.1] The four primitives are described as 'orthogonal' and 'comprehensive', but no formal definition or coverage argument is provided. Consider softening this claim or adding a brief justification.
  8. [Table 4] For AI Trading, only profit rates are shown in the w/ rules comparison; including Pass@4 or a normalized metric would make the table consistent with Table 2.

Circularity Check

0 steps flagged

No significant circularity: the benchmark conclusions are empirical evaluation results, not derivations built from their own inputs.

full rationale

OdysseyArena's central claims are produced by running LLMs on the authors' own environments and comparing against human, random, and algorithmic baselines (e.g., Optimal/Conservative/Progressive strategies in Table 5). No free parameter is fitted to force the conclusion, and no equation in the paper equates the target result with an input by construction. The w/ rules versus w/o rules comparison in §5.2 is an experimental control, not a self-definitional identity: it directly tests what happens when latent rules are revealed, and the results in Turn On Lights (Table 4) support the deductive-versus-inductive asymmetry claimed there. The Energy Dispatch rows of Table 4 do contradict the general 'primary bottleneck is discovery' inference (Gemini 3 Pro falls from 30.00 to 16.67 with rules, GPT-5 from 23.33 to 13.33), but that is a validity/confound concern about whether the control isolates induction, not a circular reduction of the claim to its inputs. The paper's self-citations (φ-decoding, Genius, TIDE, OS-Genesis) appear only as related-work context or motivation and are not load-bearing evidence for the benchmark's validity or for the inductive-bottleneck conclusion. The benchmark is self-contained, externally checkable through released code and data, and its headline result is an evaluation outcome rather than a fitted prediction. Therefore no significant circularity is found.

Axiom & Free-Parameter Ledger

1 free parameters · 3 axioms · 0 invented entities

The benchmark introduces no new physical/mathematical entity and requires no fitted model parameters to reach its empirical conclusion. The load-bearing assumptions are methodological: the orthogonality of the four primitives, the validity of the w/ rules control, and the reliability of human annotation. The difficulty-tier thresholds are hand-chosen and feed the 'inductive ceiling' analysis.

free parameters (1)
  • Easy/Medium/Hard task split thresholds = Proxy-based (e.g., number of lights); exact cutoffs not specified
    Figure 5 segments tasks into difficulty tiers to demonstrate an 'inductive ceiling'; the hand-chosen thresholds affect the plateau conclusion.
axioms (3)
  • ad hoc to paper The four primitive transition structures (discrete Boolean, continuous stochastic, periodic, relational graph) are orthogonal and jointly cover latent world dynamics.
    Section 3.1 asserts 'irreducible to the others' with no proof or external grounding; the suite's coverage claim rests on this taxonomy.
  • ad hoc to paper The w/ rules vs w/o rules contrast isolates induction from deduction; any performance drop is attributed to missing rule knowledge rather than differences in prompt format, task complexity, or environment state.
    Section 5.2 / Figure 4 use this contrast to infer the 'inductive bottleneck'; Energy Dispatch w/ rules results (Table 4) contradict the isolation assumption.
  • domain assumption Human annotators provide a reliable upper-bound estimate of task solvability.
    Human results in Table 2 come from 4 annotators per example; Appendix D.5 reports low agreement (κ=0.18 for Repo System, ICC=0.12 for AI Trading), so the human baseline may itself be noisy.

pith-pipeline@v1.3.0-alltime-deepseek · 30868 in / 11853 out tokens · 125197 ms · 2026-08-03T04:04:26.806397+00:00 · methodology

0 comments
read the original abstract

The rapid advancement of Large Language Models (LLMs) has catalyzed the development of autonomous agents capable of navigating complex environments. However, existing evaluations primarily adopt a deductive paradigm, where agents execute tasks based on explicitly provided rules and static goals, often within limited planning horizons. Crucially, this neglects the inductive necessity for agents to discover latent transition laws from experience autonomously, which is the cornerstone for enabling agentic foresight and sustaining strategic coherence. To bridge this gap, we introduce OdysseyArena, which re-centers agent evaluation on long-horizon, active, and inductive interactions. We formalize and instantiate four primitives, translating abstract transition dynamics into concrete interactive environments. Building upon this, we establish OdysseyArena-Lite for standardized benchmarking, providing a set of 120 tasks to measure an agent's inductive efficiency and long-horizon discovery. Pushing further, we introduce OdysseyArena-Challenge to stress-test agent stability across extreme interaction horizons (e.g., > 200 steps). Extensive experiments on 15+ leading LLMs reveal that even frontier models exhibit a deficiency in inductive scenarios, identifying a critical bottleneck in the pursuit of autonomous discovery in complex environments. Our code and data are available at https://github.com/xufangzhi/Odyssey-Arena

Figures

Figures reproduced from arXiv: 2602.05843 by Ben Kao, Fangzhi Xu, Hang Yan, Haoran Luo, Jian Zhang, Jingyang Gong, Jinyang Wu, Kanzhi Cheng, Muye Huang, Qika Lin, Qiushi Sun, Xinyu Che, Yian Wang, Zeyi Sun, Zhangyue Yin, Zichen Ding, Zixian Huang.

Figure 1
Figure 1. Figure 1: Comparison between deductive and inductive settings in multi-turn agentic tasks. 1. Introduction The emergence of Large Language Models (LLMs; Gem￾ini Team, 2025; Anthropic AI, 2024) has sparked unprece￾dented interest in autonomous agents that can perceive envi￾ronments, make decisions, and take actions to accomplish complex tasks. These AI agents are increasingly deployed across diverse domains—from robo… view at source ↗
Figure 2
Figure 2. Figure 2: Demonstrations of four ODYSSEYARENA environments: Turn On Lights, AI Trading, Energy Dispatch, and Repo System. For clarity, we omit the task prompts here and present only the interaction trajectories. Full prompts are provided in Appendix C. intervention on a single light may trigger a deterministic cascade of state changes across the network. Consequently, success requires active exploration to infer the… view at source ↗
Figure 3
Figure 3. Figure 3: Overview of the benchmark architecture, illustrating the environment configuration initialization (left) and the interaction loop between the LLM agent and the environment step logic (right). 4. ODYSSEYARENA-LITE and ODYSSEYARENA-CHALLENGE Building upon the infinite task space provided by ODYSSE￾YARENA, we derive two distinct benchmarking protocols to serve as standardized instantiations designed to evalua… view at source ↗
Figure 4
Figure 4. Figure 4: Success rate comparison of w/ and w/o rules in Turn On Lights. We select Llama 3.3 70B Instruct, GLM-4-32B-0414, Qwen3-235B-A22B-Instruct, DeepSeek-V3.2, Grok 4 Fast, GPT￾5, Gemini 3 Pro Preview for illustration. main evaluations, but is provided as an optional benchmark for future research. 5. Experiments 5.1. Experimental Settings We evaluate over 15 trending LLMs on ODYSSEYARENA￾LITE, encompassing propr… view at source ↗
Figure 7
Figure 7. Figure 7: Model performance is significantly related to loop ratio. Infeasible region indicates that a high Loop Ratio results in an inability to solve long-horizon inductive reasoning tasks. 6.3. Action Loops and Inductive Stagnation Analysis of agent trajectories reveals a prevalent failure mode characterized by persistent “action loops,” where mod￾els repeat invalid operations despite receiving negative envi￾ronm… view at source ↗
Figure 8
Figure 8. Figure 8: All data for Success Rate against Step. We do not plot for Energy Dispatch environment due to its complex success conditions. Under this formulation, a smaller Loop Ratio indicates that the agent is more capable to find hidden rules through in￾ductive reasoning. B.6. Step Distribution Analysis We plot the distribution of the total steps number required to complete the task for each trajectory, including bo… view at source ↗
Figure 9
Figure 9. Figure 9: Step density distribution for LLM models: (a) Gemini 3 Pro Preview, (b) GPT-5, (c) Gemini 2.5 Pro, (d) gpt-oss-120b (high), (e) DeepSeek-V3.2, (f) Grok 4 Fast, (g) Qwen3-235B-A22B-Instruct, (h) gpt-oss-120b (medium), (i) Qwen3-30B-A3B-Instruct, (j) GLM-4- 32B-0414, (k) gpt-oss-120b (low), (l) Llama 3.3 70B Instruct, (m) Qwen3-4B-Instruct (n) Llama 3.1 8B Instruct, (o) GLM-4-9B-Chat. Task (a) (b) (c) (d) (e… view at source ↗
Figure 10
Figure 10. Figure 10: Task success status (based on pass@4). Each row represents: (a) Human, (b) Gemini3 Pro Preview, (c) GPT-5, (d) Gemini 2.5 Pro, (e) gpt-oss-120b (high), (f) DeepSeek-V3.2, (g) Grok 4 Fast, (h) Qwen3-235B-A22B-Instruct, (i) gpt-oss-120b (medium), (j) Qwen3-30B-A3B-Instruct, (k) GLM-4-32B-0414, (l) gpt-oss-120b (low), (m) Llama 3.3 70B Instruct, (n) Qwen3-4B-Instruct, (o) Llama 3.1 8B Instruct, (p) GLM-4-9B-… view at source ↗
Figure 11
Figure 11. Figure 11: The line chart represents token usage and the bar chart represents token efficiency. Token Usage is measured in units of 106 , while Token Efficiency is reported in units of 10−6 [PITH_FULL_IMAGE:figures/full_fig_p019_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Error types of Gemini 3 Pro Preview. (a) Behavior Stagnation, (b) Error Credit Assignment, (c) Long-Horizon Depen￾dence Decay, (d) Local Optima. the system. For example, in Repo System, the agent attempts to resolve local compatibility between a specific package pair while failing to account for other package constraints of the entire dependency graph. B.9. Different Strategies for AI Trading In this subs… view at source ↗
Figure 13
Figure 13. Figure 13: Screenshot of the user interface and instructions provided to the human annotators. 27 [PITH_FULL_IMAGE:figures/full_fig_p027_13.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models

    cs.AI 2026-07 accept novelty 6.0

    VLM judges of CUA trajectories are systematically lenient; OSReward measures this with human gold, and OS-Shepherd open models close most of the reliability gap cheaply.

  2. SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning

    cs.CL 2026-07 conditional novelty 6.0

    SEED improves agentic RL by converting a policy's own completed trajectories into natural-language skills and distilling the skill-induced probability shift back into the policy as dense token-level supervision.

  3. Data-Driven Boundary Control of Distributed Port-Hamiltonian Systems

    eess.SY 2026-04 unverdicted novelty 5.0

    GP-dPHS learning plus interconnection boundary control yields probabilistic boundedness conditions for closed-loop trajectories under Hamiltonian model mismatch.

  4. OPID: On-Policy Skill Distillation for Agentic Reinforcement Learning

    cs.CL 2026-06 unverdicted novelty 4.0

    OPID distills episode- and step-level skills from completed on-policy trajectories, routes them via critical-first mechanism, and combines the resulting log-probability shift advantage with outcome advantage for polic...

Reference graph

Works this paper leans on

41 extracted references · 1 canonical work pages · cited by 4 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    J., Bethge, M., and Schulz, E

    Akata, E., Schulz, L., Coda-Forno, J., Oh, S. J., Bethge, M., and Schulz, E. Playing repeated games with large language models. Nature Human Behaviour, 9 0 (7): 0 1380–1390, May 2025. ISSN 2397-3374. doi:10.1038/s41562-025-02172-y. URL http://dx.doi.org/10.1038/s41562-025-02172-y

    doi:10.1038/s41562-025-02172-y 2025

  3. [3]

    The claude 3 model family: Opus, sonnet, haiku

    Anthropic\;AI. The claude 3 model family: Opus, sonnet, haiku. Claude-3 Model Card, 1: 0 1, 2024. URL https://assets.anthropic.com/m/61e7d27f8c8f5919/original/Claude-3-Model-Card.pdf

    2024

  4. [4]

    H., and Bengio, Y

    Chevalier-Boisvert, M., Bahdanau, D., Lahlou, S., Willems, L., Saharia, C., Nguyen, T. H., and Bengio, Y. Babyai: A platform to study the sample efficiency of grounded language learning. In International Conference on Learning Representations, 2019. URL https://iclr.cc/virtual/2019/poster/733

    2019

  5. [5]

    On the measure of intelligence

    Chollet, F. On the measure of intelligence. arXiv preprint arXiv:1911.01547, 2019. URL https://arxiv.org/abs/1911.01547

    Pith/arXiv arXiv 1911

  6. [6]

    Evaluating long-context reasoning in llm-based webagents

    Chung, A., Zhang, Y., Lin, K., Rawal, A., Gao, Q., and Chai, J. Evaluating long-context reasoning in llm-based webagents. In NeurIPS 2025 Workshop on Bridging Language, Agent, and World Models for Reasoning and Planning, 2025. URL https://openreview.net/forum?id=oxj422wRvO

    2025

  7. [7]

    The dynamical challenge

    Clark, A. The dynamical challenge. Cognitive science, 21 0 (4): 0 461--481, 1997. URL https://www.sciencedirect.com/science/article/abs/pii/S0364021399800305

    1997

  8. [8]

    Mind2web: Towards a generalist agent for the web

    Deng, X., Gu, Y., Zheng, B., Chen, S., Stevens, S., Wang, B., Sun, H., and Su, Y. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36: 0 28091--28114, 2023. URL https://openreview.net/forum?id=kiYqbO3wqw

    2023

  9. [9]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities

    Gemini Team . Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025. URL https://arxiv.org/abs/2507.06261

    Pith/arXiv arXiv 2025

  10. [10]

    Chatglm: A family of large language models from glm-130b to glm-4 all tools

    GLM, T., Zeng, A., Xu, B., Wang, B., Zhang, C., Yin, D., Zhang, D., Rojas, D., Feng, G., Zhao, H., et al. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793, 2024. URL https://arxiv.org/abs/2406.12793

    Pith/arXiv arXiv 2024

  11. [11]

    The llama 3 herd of models

    Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. URL https://arxiv.org/abs/2407.21783

    Pith/arXiv arXiv 2024

  12. [12]

    and Schmidhuber, J

    Ha, D. and Schmidhuber, J. Recurrent world models facilitate policy evolution. Advances in neural information processing systems, 31, 2018. URL https://proceedings.neurips.cc/paper_files/paper/2018/file/2de5d16682c3c35007e4e92982f1a2ba-Paper.pdf

    2018

  13. [13]

    H., Gonzalez, J

    Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023. URL https://dl.acm.org/doi/abs/10.1145/3600006.3613165

    arXiv 2023

  14. [14]

    M., Ullman, T

    Lake, B. M., Ullman, T. D., Tenenbaum, J. B., and Gershman, S. J. Building machines that learn and think like people. Behavioral and brain sciences, 40: 0 e253, 2017. URL https://pubmed.ncbi.nlm.nih.gov/27881212/

    arXiv 2017

  15. [15]

    State space models on temporal graphs: A first-principles study

    Li, J., Wu, R., Jin, X., Ma, B., Chen, L., and Zheng, Z. State space models on temporal graphs: A first-principles study. Advances in Neural Information Processing Systems, 37: 0 127030--127058, 2024. URL https://openreview.net/forum?id=UaJErAOssN

    2024

  16. [16]

    Y., Le Bras, R., Richardson, K., Sabharwal, A., Poovendran, R., Clark, P., and Choi, Y

    Lin, B. Y., Le Bras, R., Richardson, K., Sabharwal, A., Poovendran, R., Clark, P., and Choi, Y. Zebralogic: On the scaling limits of llms for logical reasoning. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=sTAJ9QyA6l

    2025

  17. [17]

    Deepseek-v3

    Liu, A., Mei, A., Lin, B., Xue, B., Wang, B., Xu, B., Wu, B., Zhang, B., Lin, C., Dong, C., et al. Deepseek-v3. 2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556, 2025. URL https://arxiv.org/abs/2512.02556

    Pith/arXiv arXiv 2025

  18. [18]

    Agentbench: Evaluating llms as agents

    Liu, X., Yu, H., Zhang, H., Xu, Y., Lei, X., Lai, H., Gu, Y., Ding, H., Men, K., Yang, K., et al. Agentbench: Evaluating llms as agents. In ICLR, 2024. URL https://openreview.net/forum?id=zAdUB0aCTQ

    2024

  19. [19]

    Gaia: a benchmark for general ai assistants

    Mialon, G., Fourrier, C., Wolf, T., LeCun, Y., and Scialom, T. Gaia: a benchmark for general ai assistants. In The Twelfth International Conference on Learning Representations, 2023

    2023

  20. [20]

    gpt-oss-120b & gpt-oss-20b model card

    OpenAI. gpt-oss-120b & gpt-oss-20b model card. gpt-oss model card, 1: 0 1, 2025. URL https://arxiv.org/abs/2508.10925

    Pith/arXiv arXiv 2025

  21. [21]

    G., Mao, H., Yan, F., Ji, C

    Patil, S. G., Mao, H., Yan, F., Ji, C. C.-J., Suresh, V., Stoica, I., and Gonzalez, J. E. The berkeley function calling leaderboard (bfcl): From tool use to agentic evaluation of large language models. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/pdf?id=2GmDdhBdDk

    2025

  22. [22]

    E., Li, W., Campbell-Ajala, F., Toyama, D

    Rawles, C., Clinckemaillie, S., Chang, Y., Waltz, J., Lau, G., Fair, M., Li, A., Bishop, W. E., Li, W., Campbell-Ajala, F., Toyama, D. K., Berry, R. J., Tyamagundlu, D., Lillicrap, T. P., and Riva, O. Androidworld: A dynamic benchmarking environment for autonomous agents. In The Thirteenth International Conference on Learning Representations, 2025. URL ht...

    2025

  23. [23]

    Reflexion: Language agents with verbal reinforcement learning

    Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K., and Yao, S. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36: 0 8634--8652, 2023. URL https://openreview.net/pdf?id=vAElhFcKW6

    2023

  24. [24]

    Alfworld: Aligning text and embodied environments for interactive learning

    Shridhar, M., Yuan, X., Cote, M.-A., Bisk, Y., Trischler, A., and Hausknecht, M. Alfworld: Aligning text and embodied environments for interactive learning. In International Conference on Learning Representations, 2021. URL https://openreview.net/pdf?id=0IOX0YcCdTn

    2021

  25. [25]

    Corex: Pushing the boundaries of complex reasoning through multi-model collaboration

    Sun, Q., Yin, Z., Li, X., Wu, Z., Qiu, X., and Kong, L. Corex: Pushing the boundaries of complex reasoning through multi-model collaboration. arXiv preprint arXiv:2310.00280, 2023

    Pith/arXiv arXiv 2023

  26. [26]

    Os-genesis: Automating gui agent trajectory construction via reverse task synthesis

    Sun, Q., Cheng, K., Ding, Z., Jin, C., Wang, Y., Xu, F., Wu, Z., Jia, C., Chen, L., Liu, Z., et al. Os-genesis: Automating gui agent trajectory construction via reverse task synthesis. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 5555--5579, 2025 a

    2025

  27. [27]

    Scienceboard: Evaluating multimodal autonomous agents in realistic scientific workflows

    Sun, Q., Liu, Z., Ma, C., Ding, Z., Xu, F., Yin, Z., Zhao, H., Wu, Z., Cheng, K., Liu, Z., et al. Scienceboard: Evaluating multimodal autonomous agents in realistic scientific workflows. arXiv preprint arXiv:2505.19897, 2025 b

    Pith/arXiv arXiv 2025

  28. [28]

    Mars: Situated inductive reasoning in an open-world environment

    Tang, X., Li, J., Liang, Y., Zhu, S.-C., Zhang, M., and Zheng, Z. Mars: Situated inductive reasoning in an open-world environment. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL https://openreview.net/forum?id=3qoQ6AolAz

    2024

  29. [29]

    Michelangelo: Long context evaluations beyond haystacks via latent structure queries

    Vodrahalli, K., Ontanon, S., Tripuraneni, N., Xu, K., Jain, S., Shivanna, R., Hui, J., Dikkala, N., Kazemi, M., Fatemi, B., et al. Michelangelo: Long context evaluations beyond haystacks via latent structure queries. CoRR, 2024. URL https://openreview.net/forum?id=jdc57bqY3u

    2024

  30. [30]

    Large language models for robotics: Opportunities, challenges, and perspectives

    Wang, J., Shi, E., Hu, H., Ma, C., Liu, Y., Wang, X., Yao, Y., Liu, X., Ge, B., and Zhang, S. Large language models for robotics: Opportunities, challenges, and perspectives. Journal of Automation and Intelligence, 4 0 (1): 0 52--64, 2025 a . URL https://www.sciencedirect.com/science/article/pii/S2949855424000613

    2025

  31. [31]

    M., Xu, J., R \"u hle, V., and Rajmohan, S

    Wang, W., Han, D., Diaz, D. M., Xu, J., R \"u hle, V., and Rajmohan, S. Odysseybench: Evaluating llm agents on long-horizon complex office application workflows. arXiv preprint arXiv:2508.09124, 2025 b . URL https://arxiv.org/abs/2508.09124

    Pith/arXiv arXiv 2025

  32. [32]

    W., Passos, A

    Wei, J., Sun, Z., Papay, S., McKinney, S., Han, J., Fulford, I., Chung, H. W., Passos, A. T., Fedus, W., and Glaese, A. Browsecomp: A simple yet challenging benchmark for browsing agents. arXiv preprint arXiv:2504.12516, 2025. URL https://arxiv.org/pdf/2504.12516

    Pith/arXiv arXiv 2025

  33. [33]

    J., Cheng, Z., Shin, D., Lei, F., et al

    Xie, T., Zhang, D., Chen, J., Li, X., Zhao, S., Cao, R., Hua, T. J., Cheng, Z., Shin, D., Lei, F., et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processing Systems, 37: 0 52040--52094, 2024. URL https://openreview.net/forum?id=tN61DTr4Ed#discussion

    2024

  34. [34]

    -decoding: Adaptive foresight sampling for balanced inference-time exploration and exploitation

    Xu, F., Yan, H., Ma, C., Zhao, H., Liu, J., Lin, Q., and Wu, Z. -decoding: Adaptive foresight sampling for balanced inference-time exploration and exploitation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 13214--13227, Vienna, Austria, July 2025 a . Association for Computational ...

    doi:10.18653/v1/2025.acl-long.647 2025

  35. [35]

    Genius: A generalizable and purely unsupervised self-training framework for advanced reasoning

    Xu, F., Yan, H., Ma, C., Zhao, H., Sun, Q., Cheng, K., He, J., Liu, J., and Wu, Z. Genius: A generalizable and purely unsupervised self-training framework for advanced reasoning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 13153--13167, Vienna, Austria, July 2025 b . Association ...

    work page doi:10.18653/v1/2025.acl-long.644 2025

  36. [36]

    F., Song, Y., Li, B., Tang, Y., Jain, K., Bao, M., Wang, Z

    Xu, F. F., Song, Y., Li, B., Tang, Y., Jain, K., Bao, M., Wang, Z. Z., Zhou, X., Guo, Z., Cao, M., Yang, M., Lu, H. Y., Martin, A., Su, Z., Maben, L. M., Mehta, R., Chi, W., Jang, L. K., Xie, Y., Zhou, S., and Neubig, G. Theagentcompany: Benchmarking LLM agents on consequential real world tasks. In The Thirty-ninth Annual Conference on Neural Information ...

    2025

  37. [37]

    Tide: Trajectory-based diagnostic evaluation of test-time improvement in llm agents

    Yan, H., Che, X., Xu, F., Sun, Q., Ding, Z., Cheng, K., Zhang, J., Qin, T., Liu, J., and Lin, Q. Tide: Trajectory-based diagnostic evaluation of test-time improvement in llm agents. arXiv preprint arXiv:2602.02196, 2025. URL https://arxiv.org/abs/2602.02196

    arXiv 2025

  38. [38]

    Qwen3 technical report

    Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. URL https://arxiv.org/abs/2505.09388

    Pith/arXiv arXiv 2025

  39. [39]

    R., and Cao, Y

    Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K. R., and Cao, Y. React: Synergizing reasoning and acting in language models. In The eleventh international conference on learning representations, 2023. URL https://openreview.net/forum?id=WE_vluYUL-X

    2023

  40. [40]

    Yao, S., Shinn, N., Razavi, P., and Narasimhan, K. R. \ \ tau\ \ -bench: A benchmark for underline\ T\ ool- underline\ A\ gent- underline\ U\ ser interaction in real-world domains. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=roNSXZpUDN

    2025

  41. [41]

    F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Ou, T., Bisk, Y., Fried, D., et al

    Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Ou, T., Bisk, Y., Fried, D., et al. Webarena: A realistic web environment for building autonomous agents. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=oKn9c6ytLx

    2024