Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

OPT-BENCH: Evaluating LLM Agent on Large-Scale Search Spaces Optimization Problems

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Incorporating historical context improves LLM optimization across 20 ML tasks and 10 NP problems, according to a new benchmark and agent framework.

desk verdict A useful new agent benchmark whose headline 'history helps' claim is undermined by a flawed Improvement Rate formula that mixes lower- and higher-is-better metrics. read the letter →

arxiv 2506.10764 v1 pith:KDQNKFPE submitted 2025-06-12 cs.AI cs.LG

Xiaozhe Li , Jixuan Chen , Xinyu Fang , Shengyuan Ding , Haodong Duan , Qingwen Liu , Kai Chen This is my paper
classification cs.AIcs.LG
keywords LLMagentsiterativeoptimizationhistoricalfeedbackbenchmarkingmachinelearningtasksNP-hardproblemssolutionrefinementagentevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

OPT-BENCH asks whether an LLM can get better at a problem by seeing what it tried before and what score that attempt earned, instead of answering once from scratch. The paper introduces a 30-task benchmark, 20 machine-learning tasks drawn from Kaggle competitions and 10 classical NP-hard problems, and an agent loop, OPT-Agent, that drafts a solution, validates it, then improves or debugs it using the history of previous solutions, metrics, and error messages. Across nine LLMs from six model families and 5, 10, and 20 optimization steps, the paper reports that supplying this historical context raises win counts and improvement rates relative to a no-history baseline for most models, on both task families. The reason to care is that real optimization is iterative: a benchmark that measures feedback-driven refinement tests a different capability than single-pass question answering.

What carries the argument

The load-bearing mechanism is OPT-Agent's draft-improve-debug loop. At each step the LLM receives the problem description, submission format, and, when history is enabled, the previous solution, its metric, and any validation error; it then either drafts a fresh solution, refines the last valid one, or repairs the last buggy one. A rule-based or metric-based validator turns every attempt into a concrete score or a validity verdict, so the loop always has a checkable quality signal to feed back into the next prompt. The headline comparison metric is Win Count, the number of tasks where the history-augmented run beats the no-history run under otherwise identical settings, supplemented by Improvement Rate, Buggy Rate, and Rank.

What would settle it

Run both conditions on the same 30 tasks, but in the history condition replace the agent's own previous solutions and metrics with an equal-length history drawn from a different task of the same type. If the advantage over the no-history baseline persists, the gain is not specific to learning from the current problem's feedback and the central claim would be falsified; if the advantage disappears, the paper's interpretation is supported.

Watch

Extended reading notes

Core claim

The central claim is that historical context is a measurable and general driver of LLM optimization quality. Using identical task definitions, datasets, validators, and evaluation metrics, OPT-Agent with history outperforms OPT-Agent without history on a majority of the 20 ML tasks and 10 NP problems for most of the nine LLMs tested; the paper's improvement-rate numbers are above 1 in nearly all ML configurations, and win counts favor the history condition on NP problems more often than not. The paper also reports that longer optimization horizons generally improve final metrics, that moderate temperatures tend to balance exploration and solution validity, and that open-source models produce more invalid solutions and trail proprietary models on NP tasks. On the ML side, the benefit of history appears as better hyperparameter choices and feature engineering; on the NP side, the benefit is real but weaker, because models often discard the previous solution and start over instead of repairing the specific failing constraint. The intended conclusion is that evaluating LLMs on iterative, feedback-driven optimization is feasible and that this capability is partly independent of one-shot reasoning strength.

Load-bearing premise

The comparison only demonstrates that history helps if the no-history baseline is otherwise identical in task information, prompt formatting, and starting conditions; the paper states that the only difference is the absence of historical information but does not fully specify how the baseline prompt is constructed, leaving open the possibility that the gain comes from extra context rather than from learning.

Editorial extensions

If this is right

  • Benchmarks for LLM problem-solving should include iterative loops with feedback, since a single generation no longer measures the capability these results target.
  • Optimization budgets matter: the paper reports that improvement over the initial solution rises with the number of steps, from 5 to 20, for most models.
  • Temperature should be treated as a per-model hyperparameter for optimization agents, with low-to-moderate values usually but not always best.
  • Draft-based optimization should be evaluated separately from refinement, since it yields higher invalid-solution rates yet often higher improvement rates when solutions are valid.
  • Future LLM development can use the 30 validated tasks and public scripts as a reproducible yardstick for optimization ability.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's central result does not isolate why history helps; a natural follow-up is to give the no-history baseline a fixed amount of extra context that is task-irrelevant, so any remaining gain in the history condition is specifically about the current problem's feedback.
  • The NP half of the benchmark could be used to test whether models genuinely reuse valid substructures from earlier attempts or merely regenerate answers, by logging how often the final solution contains segments of previous valid solutions.
  • Because the paper names growing prompt size as a limitation, a testable extension is that compressing history, summaries of what changed and by how much rather than full previous code, preserves most of the benefit at much lower token cost.
  • A skeptical reading is that the ML win counts may partly reflect the no-history baseline losing access to the previous code it needs for debugging; isolating that would require a baseline that receives the previous error message and metric but not the previous solution.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces OPT-BENCH, a benchmark containing 20 Kaggle machine-learning tasks and 10 classical NP problems, together with OPT-Agent, an LLM agent framework that iteratively drafts, improves, and debugs solutions using historical feedback. The authors evaluate nine LLMs from six families and report that including historical context improves optimization performance, as measured by win counts and an Improvement Rate, and also study the effects of iteration count, temperature, and draft versus refine settings. The benchmark, code, and evaluation tools are open-sourced.

Significance. If the empirical claims are validated, OPT-BENCH fills a clear gap: existing LLM benchmarks largely test single-pass responses, whereas iterative optimization with feedback is an important and under-measured capability. The benchmark's assets—real-world ML tasks, human-verified initial solutions, automated NP validation scripts, and an open-source framework—are valuable resources for the community. The paper's main weakness is that its central quantitative claim rests on a metric that is defined and interpreted inconsistently, and on win counts without statistical support. The benchmark itself is sound as a contribution; the empirical analysis needs correction before the headline claim is acceptable.

major comments (4)
  1. [Section 3.2, Table 2] The Improvement Rate as defined, IR(α,β) = (1/20) Σ α_i/β_i, with α_i the metric under the improved setting and β_i the baseline metric, is not a valid measure of improvement when the 20 ML tasks mix lower-is-better metrics (12 tasks per Table 7, e.g., MSE, RMSE, RMSLE, log loss) with higher-is-better metrics (8 tasks, e.g., accuracy, AUC, F1, kappa). For a lower-is-better metric, an actual improvement gives α_i/β_i < 1, while for a higher-is-better metric it gives a ratio > 1; averaging these incompatible quantities makes IR > 1 uninterpretable. Table 2 reports IR(w,w.o) > 1 for every model and step count, which is only possible if lower-is-better ratios were inverted without documentation. Section 3.3 then reads IR > 1 as "underscoring the value of contextual information," and the abstract's claim that history "significantly enhances" optimization relies on this measure. Please re-analyze the data using direction-aware ratios, report per-task signed improvements, and correct the conclusions accordingly.
  2. [Section 3.2, Tables 2-3] The win-count comparisons are based on a single run per condition, with no repeated seeds, confidence intervals, or significance tests. With only 20 ML tasks and 10 NP problems, win counts such as 12/8, 11/9, or 10/10 are within the range of random variation, so the abstract's word "significantly" is not supported by the evidence. Please report the number of seeds or runs, and provide a statistical test (e.g., a binomial test against the null of equal performance, or a paired test on per-task metric changes) to substantiate the claim.
  3. [Section 3.2 and Figure 4] The baseline "without historical information" is described as differing only by the absence of historical information, but the prompt template in Figure 4 contains separate fields for "History Information," "Previous Solution," "Previous (buggy) Implementation," and "Previous (buggy) Output." If the baseline omits all of these, the comparison does not isolate "history" from "having a previous solution," and the improvement might simply reflect the ability to start from an existing draft rather than learning from feedback. Please specify exactly which prompt fields are kept and which are removed in the baseline, and describe what action the baseline agent performs at each optimization step.
  4. [Section 5, Limitations] The paper's own Limitations section acknowledges that "averaging performance metrics may introduce scale inconsistencies" for OPT-BENCH-ML. The Improvement Rate in Section 3.2 is exactly such an average of ratios across heterogeneous metrics (MSE, accuracy, etc.). Since the central claim depends on this rate, the acknowledgment is not sufficient: the analysis should be corrected, for example, by reporting per-task signed improvements or normalized changes, and the corrected metric should be used to re-evaluate all claims in Section 3.3 and the abstract.
minor comments (5)
  1. [Section 3.3] There is a typo: "which is a key sapect for future improvement" should read "a key aspect."
  2. [Figure 2] The NP example is labeled "Hamiltonian-cycle" but the task description mentions finding a subgraph and maximizing the number of visited vertices, while the validation script checks for a cycle. Please clarify whether the task is to find a Hamiltonian cycle or to find the longest cycle/subgraph in a graph, and align the description with the actual validation logic.
  3. [Table 1] The table refers to "NPHardBench [10]" but the reference list and Section 4.1 use "NPHardEval"; the names should be consistent.
  4. [Section 2.1, Tables 7-8] The text says each NP problem contains "five distinct instances," but it is not specified how win counts and ranks aggregate across those instances. Please state whether win counts are computed per instance, per problem, or per problem set, and report the aggregation rule.
  5. [Figures 5 and 6] The trace diagrams are dense and hard to read at the current resolution. Please ensure the published figures are legible, or provide a zoomed view of representative paths.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the empirical history-vs-baseline comparison is self-contained and not derived from its own inputs.

full rationale

OPT-BENCH's central claim is an experimental result comparing OPT-Agent with and without historical context. The evaluation metrics (Win Count, IR, Buggy Rate, Rank) are defined directly on observed task outcomes; IR is a ratio of measured metrics, not a fitted parameter or a quantity equivalent to its input. The initial solutions are seeded using AIDE, but AIDE is an external framework by other authors and the claim does not depend on AIDE's correctness. No uniqueness theorem or ansatz is imported from prior work, and no known result is renamed. Section 3.2's IR formula does mix lower- and higher-is-better metrics, so reading IR(w,w.o)>1 as 'improvement' is a validity concern (also partially acknowledged in the Limitations as scale inconsistencies when averaging metrics), but this is a measurement-construction issue, not a case where a prediction reduces by construction to its inputs. The comparison is controlled as stated: the only intended difference is the inclusion of historical information. Accordingly the circularity score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 2 invented entities

The core scientific content is an empirical benchmark. The only explicit free parameters are the experimental settings (step counts, temperatures, model choices), which are not fitted to data. The key assumptions are that history prompting helps, that the IR metric is meaningful across mixed metrics, and that the NP validation scripts are correct. No new physical entities are introduced.

assumptions (3)
  • domain assumption LLMs can improve their own optimization solutions when prompted with historical feedback.
    Central premise of the benchmark and OPT-Agent; without it the entire evaluation is meaningless.
  • domain assumption Averaging ratios across tasks with heterogeneous metrics yields a meaningful improvement rate.
    Section 3.2 defines IR as the average of metric ratios; this assumes MSE and accuracy ratios are comparable without explicit direction normalization.
  • domain assumption The validation scripts correctly judge NP solution validity.
    Section 2.1 relies on rule-based validation.py for NP tasks; a bug there would invalidate all NP results.
invented entities (2)
  • OPT-BENCH dataset
    purpose: A test suite of 30 optimization tasks for evaluating LLM agents.
    A new benchmark artifact; its validity is assessed only by this paper's experiments, with no external falsifiable handle.
  • OPT-Agent framework
    purpose: A prompting pipeline that iteratively drafts, improves, and debugs solutions using historical feedback.
    A new agent framework; no independent evidence outside the paper validates its effectiveness beyond the authors' own experiments.

how reviews work

0 comments
Cite this review

Pith. "Pith review of OPT-BENCH: Evaluating LLM Agent on Large-Scale Search Spaces Optimization Problems." pith.science (2026). https://pith.science/paper/KDQNKFPE

@misc{pith2026250610764,
  author       = {Pith},
  title        = {Pith review of: OPT-BENCH: Evaluating LLM Agent on Large-Scale Search Spaces Optimization Problems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KDQNKFPE}},
  note         = {Machine review of arXiv:2506.10764}
}
read the original abstract

Large Language Models (LLMs) have shown remarkable capabilities in solving diverse tasks. However, their proficiency in iteratively optimizing complex solutions through learning from previous feedback remains insufficiently explored. To bridge this gap, we present OPT-BENCH, a comprehensive benchmark designed to evaluate LLM agents on large-scale search space optimization problems. OPT-BENCH includes 20 real-world machine learning tasks sourced from Kaggle and 10 classical NP problems, offering a diverse and challenging environment for assessing LLM agents on iterative reasoning and solution refinement. To enable rigorous evaluation, we introduce OPT-Agent, an end-to-end optimization framework that emulates human reasoning when tackling complex problems by generating, validating, and iteratively improving solutions through leveraging historical feedback. Through extensive experiments on 9 state-of-the-art LLMs from 6 model families, we analyze the effects of optimization iterations, temperature settings, and model architectures on solution quality and convergence. Our results demonstrate that incorporating historical context significantly enhances optimization performance across both ML and NP tasks. All datasets, code, and evaluation tools are open-sourced to promote further research in advancing LLM-driven optimization and iterative reasoning. Project page: \href{https://github.com/OliverLeeXZ/OPT-BENCH}{https://github.com/OliverLeeXZ/OPT-BENCH}.

Figures

Figures reproduced from arXiv: 2506.10764 by the authors.

Figure 1
Figure 1. Overview of the OPT-BENCH dataset and OPT-Agent Framework. The left panel illustrates the data structure of OPT-Bench, encompassing ML and NP problems. Each module includes problem definitions, dataset files, validation script (NP), evaluation metrics, and submission formats, integrating human-verified initial solutions and LLM-assisted refinement. The right panel details the evaluation workflow, where solutions are… view at source ↗
Figure 2
Figure 2. Specific cases from OPT-BENCH. Take the spaceship titanic classification task and the Hamiltonian cycle optimization problem as representative examples. Benchmark Task Num Iterative Optimization Metrics ML NP NPHardBench [10] ✗ 9 ✗ Weighted Accuracy, Failure Rate MLE-Bench [5] 75 ✗ ✓ Medal Rate MLAgentBench [17] 13 ✗ ✓ Success Rate, Avg. Improvement OPT-BENCH 20 10 ✓ Win Count, Buggy Rate, Rank, IR [PITH_FULL_IMAGE… view at source ↗
Figure 3
Figure 3. OPT-Agent Optimization Trace on ML and NP Tasks. Nodes are color-coded by task status: yellow for Draft, blue for Debug, and red for Improve. Each node displays performance metrics and descriptive details, reflecting iterative improvements during optimization. OPT-Agent leverages historical information to enhance solution quality across ML and NP tasks, exemplified by the bike sharing demand prediction and Hamiltoni… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Prompt Template of OPT-Agent. Orange denotes draft action. Green denotes improve action. Purple denotes debug action. Blue denotes shared prompts. C OPT-Agent Results Analysis C.1 ML Task As shown in [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: Detailed OPT-Agent-ML Trace on the Bike Sharing Demand Task, utilizing gemini-2.0-flash as LLM base model. The red, and blue nodes represent the improve, and debug action, respectively. C.2 NP Problem As shown in [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: Detailed OPT-Agent-NP Trace on the Hamiltonian Cycle Task, utilizing gemini-2.0-flash as the LLM base model. The yellow, red, and blue nodes represent the draft, improve, and debug action, respectively. 12 [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Fixed prompts in OPT-Agent. This encompasses the response format, implementation guidelines, solution draft sketch guidelines, solution improvement sketch guidelines, and solution debug sketch guidelines for ML tasks, as well as example inputs and outputs, instructions…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. PEARL: Solver-in-the-Loop Interactive Optimization Modeling from Natural Language

    cs.AI 2026-05 reject novelty 7.0 of 10

    Training an LLM as a multi-turn agent that runs and repairs solver code raises verified optimization solve rates, with the 4B PEARL model outperforming DeepSeek-V3.2-685B in aggregate.

Reference graph

Works this paper leans on

45 extracted references · 8 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Mathqa: Towards interpretable math word problem solving with operation-based formalisms.arXiv preprint arXiv:1905.13319, 2019

    Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation-based formalisms.arXiv preprint arXiv:1905.13319, 2019

    arXiv 1905

  3. [3]

    Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021

    arXiv 2021

  4. [4]

    Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  5. [6]

    Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021

    arXiv 2021

  6. [7]

    Palm: Scaling language modeling with pathways.Journal of Machine Learning Research, 24(240):1– 113, 2023

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways.Journal of Machine Learning Research, 24(240):1– 113, 2023

    2023

  7. [8]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    arXiv

  8. [9]

    Selection-inference: Exploiting large language models for interpretable logical reasoning.arXiv preprint arXiv:2205.09712, 2022

    Antonia Creswell, Murray Shanahan, and Irina Higgins. Selection-inference: Exploiting large language models for interpretable logical reasoning.arXiv preprint arXiv:2205.09712, 2022

    arXiv 2022

Show all 45 references
  1. [10]

    Nphardeval: Dynamic benchmark on reasoning ability of large language models via complexity classes.arXiv preprint arXiv:2312.14890, 2023

    Yao Fan, Xinyu Hua, et al. Nphardeval: Dynamic benchmark on reasoning ability of large language models via complexity classes.arXiv preprint arXiv:2312.14890, 2023

    2023 arXiv

  2. [11]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    Google. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024

    2024 arXiv

  3. [12]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  4. [13]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  5. [14]

    R-bench: Graduate-level multi-disciplinary benchmarks for llm & mllm complex reasoning evaluation.arXiv preprint arXiv:2505.02018, 2025

    Meng-Hao Guo, Jiajun Xu, Yi Zhang, Jiaxi Song, Haoyang Peng, Yi-Xuan Deng, Xinzhi Dong, Kiyohiro Nakayama, Zhengyang Geng, Chen Wang, et al. R-bench: Graduate-level multi-disciplinary benchmarks for llm & mllm complex reasoning evaluation.arXiv preprint arXiv:2505.02018, 2025

    2025 arXiv

  6. [15]

    Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020

    2009 arXiv

  7. [16]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021. 14

    2021 arXiv

  8. [17]

    Mlagentbench: Evaluating language agents on machine learning experimentation

    Qian Huang, Jian V ora, Percy Liang, and Jure Leskovec. Mlagentbench: Evaluating language agents on machine learning experimentation. InForty-first International Conference on Machine Learning, 2024

    2024

  9. [18]

    Aide: Ai-driven exploration in the space of code, 2025

    Zhengyao Jiang, Dominik Schmidt, Dhruv Srikanth, Dixing Xu, Ian Kaplan, Deniss Jacenko, and Yuxiang Wu. Aide: Ai-driven exploration in the space of code, 2025

    2025

  10. [19]

    Let’s verify step by step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. InThe Twelfth International Conference on Learning Representations, 2023

    2023

  11. [20]

    Truthfulqa: Measuring how models mimic human falsehoods

    Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. InProceedings of the 60th Annual Meeting of the Association for Computa- tional Linguistics, pages 3214–3252, 2022

    2022

  12. [21]

    Criticbench: Benchmarking llms for critique-correct reasoning, 2024

    Zicheng Lin, Zhibin Gou, Tian Liang, Ruilin Luo, Haowei Liu, and Yujiu Yang. Criticbench: Benchmarking llms for critique-correct reasoning, 2024

    2024

  13. [22]

    Agentbench: Evaluating llms as agents.arXiv preprint arXiv:2308.03688, 2023

    Xiao Liu et al. Agentbench: Evaluating llms as agents.arXiv preprint arXiv:2308.03688, 2023

    2023 arXiv

  14. [23]

    Self-refine: Iterative refinement with self-feedback.arXiv preprint arXiv:2303.17651, 2023

    Aman Madaan et al. Self-refine: Iterative refinement with self-feedback.arXiv preprint arXiv:2303.17651, 2023

    2023 arXiv

  15. [24]

    Mle-bench: Evaluating machine learning agents on real-world machine learning engineering tasks.arXiv preprint arXiv:2410.07095, 2024

    OpenAI. Mle-bench: Evaluating machine learning agents on real-world machine learning engineering tasks.arXiv preprint arXiv:2410.07095, 2024

    2024 arXiv

  16. [25]

    Openai o1 system card

    OpenAI. Openai o1 system card. Technical report, OpenAI, 2025

    2025

  17. [26]

    Toolbench: An open platform for training, serving, and evaluating large language models as tool agents.https://github.com/OpenBMB/ToolBench, 2023

    OpenBMB. Toolbench: An open platform for training, serving, and evaluating large language models as tool agents.https://github.com/OpenBMB/ToolBench, 2023

    2023

  18. [27]

    Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730–27744, 2022

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730...

    2022

  19. [28]

    Art: Automatic multi-step reasoning and tool-use for language models.arXiv preprint arXiv:2303.09014, 2023

    Laura Parisi et al. Art: Automatic multi-step reasoning and tool-use for language models.arXiv preprint arXiv:2303.09014, 2023

    2023 arXiv

  20. [29]

    Toolformer: Language models can teach themselves to use tools.arXiv preprint arXiv:2302.04761, 2023

    Timo Schick, Jane Dwivedi-Yu, et al. Toolformer: Language models can teach themselves to use tools.arXiv preprint arXiv:2302.04761, 2023

    2023 arXiv

  21. [30]

    Reflexion: Language agents with verbal reinforcement learning.arXiv preprint arXiv:2303.11366, 14, 2023

    Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning.arXiv preprint arXiv:2303.11366, 14, 2023

    2023 arXiv

  22. [31]

    Alfworld: Aligning text and embodied environments for interactive learning.arXiv preprint arXiv:2010.03768, 2020

    Mohit Shridhar et al. Alfworld: Aligning text and embodied environments for interactive learning.arXiv preprint arXiv:2010.03768, 2020

    2010 arXiv

  23. [32]

    Beyond the imitation game: Quantifying and extrapolating the capabilities of language models

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint a...

    2022 arXiv

  24. [33]

    Commonsenseqa: A question answering challenge targeting commonsense knowledge

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Languag...

    2019

  25. [34]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023. 15

    2023 arXiv

  26. [35]

    Karthik Valmeekam, Matthew Marquez, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change.Advances in Neural Information Processing Systems, 36:38975–38987, 2023

    2023

  27. [36]

    Superglue: A stickier benchmark for general-purpose language understanding systems

    Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. InAdvances in Neural Information Processing Systems, volume 32, 2019

    2019

  28. [37]

    Glue: A multi-task benchmark and analysis platform for natural language understanding

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP, pages 353–355, 2018

    2018

  29. [38]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H Chi, Quoc V Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. InAdvances in Neural Information Processing Systems, volume 35, pages 24824–24837, 2022

    2022

  30. [39]

    Are large language models really good logical reasoners? a comprehensive evaluation and beyond.IEEE Transactions on Knowledge and Data Engineering, 2025

    Fangzhi Xu, Qika Lin, Jiawei Han, Tianzhe Zhao, Jun Liu, and Erik Cambria. Are large language models really good logical reasoners? a comprehensive evaluation and beyond.IEEE Transactions on Knowledge and Data Engineering, 2025

    2025

  31. [40]

    Intercode: Standardizing and benchmarking interactive coding with execution feedback.arXiv preprint arXiv:2306.14898, 2023

    John Yang et al. Intercode: Standardizing and benchmarking interactive coding with execution feedback.arXiv preprint arXiv:2306.14898, 2023

    2023 arXiv

  32. [41]

    Webshop: Towards scalable real-world web interaction with grounded language agents.arXiv preprint arXiv:2207.01206, 2022

    Shunyu Yao et al. Webshop: Towards scalable real-world web interaction with grounded language agents.arXiv preprint arXiv:2207.01206, 2022

    2022 arXiv

  33. [42]

    Tree of thoughts: Deliberate problem solving with large language models.arXiv preprint arXiv:2305.10601, 2023

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models.arXiv preprint arXiv:2305.10601, 2023

    2023 arXiv

  34. [43]

    React: Synergizing reasoning and acting in language models.arXiv preprint arXiv:2210.03629, 2022

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models.arXiv preprint arXiv:2210.03629, 2022

    2022 arXiv

  35. [44]

    Hellaswag: Can a machine really finish your sentence? InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, 2019

    2019

  36. [45]

    Iolbench: Benchmarking llms on linguistic reasoning.arXiv preprint arXiv:2501.04249, 2024

    Yicheng Zhang et al. Iolbench: Benchmarking llms on linguistic reasoning.arXiv preprint arXiv:2501.04249, 2024

    2024

  37. [46]

    Webarena: A realistic web environment for building autonomous agents

    Shuyan Zhou et al. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854, 2023. 16

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.