Pith. sign in

REVIEW 2 major objections 6 minor 7 cited by

DABstep: Data Agent Benchmark for Multi-step Reasoning

T0 review · 2 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read DABstep is a new benchmark of 450+ real financial-analysis tasks, and it shows that even the best LLM agent solves only 14.55% of the hard ones.

desk verdict DABstep is a solid, usable benchmark for data-analysis agents, but the central claim that Hard tasks are impossible single-shot is asserted rather than demonstrated, and the baselines would be much stronger with a single-shot control and error bars. read the letter →

arxiv 2506.23719 v1 pith:BCR6YZTA submitted 2025-06-30 cs.LG cs.AI

Alex Egg , Martin Iglesias Goyanes , Friso Kingma , Andreu Mora , Leandro von Werra , Thomas Wolf This is my paper
classification cs.LGcs.AI
keywords DABstepmulti-stepreasoningLLMagentsdataanalysisbenchmarkfinancialanalyticsfactoidevaluationcodeexecutionheterogeneous
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

DABstep is a benchmark of more than 450 data-analysis tasks drawn from real financial-analytics workloads. Each task forces an agent to combine code execution, heterogeneous data files, and domain documentation, and to answer in a machine-checkable factoid format. The paper's central claim is that these tasks require genuine multi-step reasoning and that current LLM agents are far from doing that well: the best baseline solves only 14.55% of the hard split, while easy single-step tasks are largely mastered. If this holds, the benchmark identifies multi-step planning and documentation-grounded reasoning, rather than basic code or SQL generation, as the current bottleneck in autonomous data analysis.

What carries the argument

The load-bearing object is the benchmark task structure itself: each item pairs a natural-language question with heterogeneous context files, formatting guidance, and a difficulty tag. Its main design mechanism is the parameterization of 95 core workflows into over 450 instances, so success cannot come from memorized answers. The factoid answer format plus a hybrid scoring algorithm that tolerates numeric, list, and string variations provides objective, LLM-free evaluation, and a hidden test set keeps the measure focused on zero-shot generalization.

What would settle it

Run an expert analyst or a single-pass code-generation system on the 378 Hard tasks with no iterative feedback; if a substantial fraction are solved correctly in one execution, the benchmark's defining multi-step requirement would not hold as stated.

Watch

Extended reading notes

Core claim

The paper introduces DABstep and argues that state-of-the-art LLM agents reliably handle single-shot analyses but largely fail when analysis requires iterative decomposition. Of 450+ tasks, 378 are classified Hard and designed so no question can be answered through a single direct execution; agents must filter, aggregate, consult manuals, compute intermediate results, and refine. On a hidden test set with objective factoid scoring, the best agent reaches 76.39% on Easy but only 14.55% on Hard, and every model tested drops sharply on the Hard split. The paper treats this gap as evidence that multi-step reasoning over heterogeneous data sources is an unsolved capability, and it releases the tasks, scoring code, baselines, and a leaderboard to make that gap measurable.

Load-bearing premise

The central premise is that every Hard task truly demands multiple sequential reasoning steps, and the paper does not demonstrate, formally or empirically, that a single well-crafted script cannot answer one of them.

Editorial extensions

If this is right

  • If DABstep's Hard split is taken as representative, current agents cannot be trusted for unsupervised real-world financial data analysis.
  • The score gap between Easy and Hard splits indicates that single-step code or SQL proficiency is no longer the limiting factor; decomposition and planning are.
  • The release of a public leaderboard and developer set makes agent performance on these tasks repeatable and comparable without heavy scaffolding.
  • Low-cost open models score close to zero on Hard, so cost-performance trade-offs and model capability are strongly coupled for iterative analysis.
  • Failure analysis implies that improvements in instruction following and documentation-aware planning may matter more than raw coding ability.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension would be to check whether increasing the number of allowed execution steps or giving agents an explicit planning scratchpad raises Hard accuracy substantially; the paper's standardized loop caps each task at ten steps, which may understate model capability.
  • If the single-execution premise is wrong for many Hard tasks, part of the measured gap may be an artifact of scoring or prompting rather than of multi-step reasoning itself.
  • The parameterization design suggests a natural stress test: permuting the same 95 core tasks further could expose whether models generalize the reasoning procedure or overfit to surface patterns.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. DABstep introduces a benchmark of over 450 financial data analysis tasks derived from Adyen's operational workloads, combining structured datasets (CSV, JSON) with unstructured documentation (Markdown). Tasks have factoid answers and are scored with a deterministic hybrid algorithm that was validated against human judgments on 75 answers (100% agreement). The paper evaluates 16 LLM-based agents under a ReAct-style harness with up to 10 steps, reporting that the best model (o4-mini) achieves only 14.55% accuracy on the Hard split (378 tasks) versus 76.39% on Easy tasks. The benchmark is released with a public leaderboard, open-source baselines, and a developer set. The central claim is that Hard tasks require genuine multi-step reasoning that distinguishes DABstep from Text-to-SQL and code-generation benchmarks.

Significance. If validated, DABstep would be a valuable resource: it provides real-world, heterogeneous data analysis tasks with objective, scalable evaluation, and the low hard-split accuracy suggests a meaningful gap in current agent capabilities. The deterministic scoring protocol, the human validation of that protocol (100% agreement on 75 examples), the public leaderboard, and the release of code and data are concrete strengths that support reproducibility and community use. The paper's novelty, however, rests on the assertion that Hard tasks cannot be solved by one-shot code generation; this is not yet demonstrated, and the lack of a single-shot control weakens the interpretation of the headline 14.55% result as a multi-step reasoning deficit. A human expert baseline would also help calibrate task solvability. These issues are fixable and do not undermine the overall enterprise, but they need to be addressed before the benchmark's distinctive contribution is fully established.

major comments (2)
  1. [Section 2.1, 3.2, and 4.1] The load-bearing claim that Hard tasks 'cannot be answered through a single direct execution' is not supported by the evidence. All baselines in Section 4.1 use a ReAct loop with a maximum of 10 steps and execution feedback; no single-shot code generation condition is reported. The term 'single direct execution' is ambiguous: if it means one Python script, then a script could in principle load all context files (payments.csv, fees.json, manual.md) and perform the required joins, filters, and rule-based computations in a single run, so what forces iteration is the agent's need to discover schemas and rules rather than the computational structure of the task. If it means one LLM call without execution feedback, that is a harness choice that the paper does not measure. The Appendix A.4 trace shows a failed 7-step trajectory but does not demonstrate that a single comprehensive script would fail. Without a one-shot control, the interpretation of Table 1's 14.55% Hard accuracy as a multi-step reasoning deficit is not established; it could reflect one-shot code generation difficulty alone. Please add a single-shot baseline (e.g., prompt the model to write a complete script and execute it without iterative feedback) and, if the claim is retained, provide a formal definition of what constitutes a required 'step' and verify per task that the stated minimum step count is necessary.
  2. [Section 5 and A.2] No human expert baseline is reported. The paper states that tasks are derived from real internal queries, which suggests solvability, but it does not demonstrate that the provided context files are sufficient for a human to answer each question as posed. Without a human performance estimate on a sample of tasks, the low agent accuracies could in principle reflect task ambiguity or underspecification rather than limitations in agent reasoning. The validation in Appendix A.2 (75 answers, 100% scoring agreement) checks only the scoring function, not the solvability or clarity of the tasks themselves. I recommend reporting human expert accuracy on a representative sample (for instance, from the developer set) and, if any tasks are found to be ambiguous or unsolvable from the given context, either revising those tasks or documenting the judgment criteria.
minor comments (6)
  1. [Section 2.2] The phrase 'gains in: high evaluation reliability' is awkwardly constructed; consider rewording to 'gains in evaluation reliability, scalability...'.
  2. [Table 1 caption] The caption states that all baselines run for a maximum of 10 steps with a ReAct-style prompt 'except for the reasoning models'; please clarify whether reasoning models also have a 10-step limit or a different limit, since this affects the comparability of results.
  3. [Appendix A.2, Algorithm 1] The text specifies a numeric tolerance of 10^-4, but Algorithm 1 uses tolerance=10^-2; please reconcile this discrepancy.
  4. [Section 3.2] The sentence 'From our baselines in Section 4, there is a 49% correlation with performance on the easy set to performance on the hard set' is vague; please specify which correlation coefficient is used and whether it is computed across models or across tasks.
  5. [Figure 1] The figure caption refers to 'at least 3 execution steps' and 'at least 6 steps' for Easy and Hard tasks; please define what constitutes an execution step and whether this is the number of agent actions or code executions.
  6. [Section 4.3] The discussion of self-attention and abstract conceptual similarity is speculative; consider softening the claim or providing citations to empirical evidence that directly supports this explanation.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the 14.55% Hard accuracy is a measured baseline result, and the benchmark's multi-step design claim is an asserted premise, not an output of a fitted or self-referential chain.

full rationale

DABstep's central claim is empirical: 16 LLM agents were run under a fixed ReAct-style harness on a held-out hidden test set, and the best agent scored 14.55% on the Hard split (Table 1, Section 4). This is a measurement, not a derivation; no parameter is fitted to a subset of the data and then renamed a prediction, and no equation in the paper reduces to its own input. The paper's main interpretative premise—that Hard tasks cannot be answered by a single direct execution (Section 2.1)—is asserted rather than demonstrated: no single-shot baseline is reported and 'single direct execution' is left undefined. That is an evidentiary gap affecting benchmark validity, but it is not circularity: the paper does not define Hard tasks in terms of the 14.55% result or derive the result from the premise. The only overlapping-author citations are GAIA [28] (Thomas Wolf) and the smolagents wrapper [36] (von Werra, Wolf, Roucher); both are contextual and implementation references, and neither supplies the load-bearing argument for the benchmark's difficulty or the baseline scores. Because the benchmark is evaluated against external models with hidden ground truths and an automated scoring rule validated against 75 human-judged answers, the reported numbers are independently checkable rather than forced by construction. Accordingly, no circular step is present.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The benchmark claim rests on assumptions about ground-truth correctness, documentation sufficiency, representativeness, and hidden-set integrity. No theoretical entities are introduced. Several hand-set thresholds (fuzzy threshold, numeric tolerance, step cap) affect reported accuracy, though none is fitted to a scientific target.

free parameters (3)
  • Fuzzy string match threshold = 0.95
    Chosen in A.2 for accepting near-miss strings; leniency directly affects which answers count as correct.
  • Numeric tolerance for float answers = 0.01 (Algorithm 1) vs 1e-4 (prose)
    The paper is internally inconsistent about the tolerance; this changes which close numeric answers are accepted.
  • Maximum ReAct steps per baseline run = 10
    Baselines stop after 10 steps, a hand-set cap that may depress Hard accuracy and makes the reported scores a lower bound.
assumptions (4)
  • domain assumption Each task has exactly one correct ground-truth answer and the curated answers are correct.
    Stated in Section 3.3: 'There is only one correct ground truth answer for each task.' Wrong or ambiguous ground truths would invalidate accuracy scores.
  • domain assumption The released context files and distilled manuals contain all information needed to solve each task.
    Section 3.1 states the context provides the necessary domain knowledge; if documentation is incomplete, failures may reflect missing information rather than agent limitations.
  • domain assumption Tasks derived from Adyen operational workloads are representative of real-world data analysis.
    Section 1 grounds the benchmark in operational workloads at Adyen; if these are idiosyncratic, generalization to other domains is limited.
  • domain assumption The hidden test set remains contamination-free and cannot be reconstructed from public materials.
    Section 3.3 argues hiding ground truths preserves integrity; this is a forward-looking assumption about leakage.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DABstep: Data Agent Benchmark for Multi-step Reasoning." pith.science (2026). https://pith.science/paper/BCR6YZTA

@misc{pith2026250623719,
  author       = {Pith},
  title        = {Pith review of: DABstep: Data Agent Benchmark for Multi-step Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BCR6YZTA}},
  note         = {Machine review of arXiv:2506.23719}
}
read the original abstract

We introduce DABstep, a novel benchmark for evaluating AI agents on realistic multi-step data analysis tasks. DABstep comprises over 450 real-world challenges derived from a financial analytics platform, requiring models to combine code-based data processing with contextual reasoning over heterogeneous documentation. Each task demands an iterative, multi-step problem-solving approach, testing capabilities in data manipulation, cross-referencing multiple sources, and precise result reporting. The benchmark provides a factoid-style answer format with automatic correctness checks for objective scoring at scale. We evaluate leading LLM-based agents, revealing a substantial performance gap: even the best agent achieves only 14.55% accuracy on the hardest tasks. We detail our benchmark's design, dataset composition, task formulation, evaluation protocol, report baseline results and analyze failure modes. DABstep is released with a public leaderboard and toolkit to accelerate research in autonomous data analysis.

Figures

Figures reproduced from arXiv: 2506.23719 by the authors.

Figure 1
Figure 1. System overview of DABstep’s agent-task interaction. The figure illustrates the core [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Agent step 0 where environment exploration happens. [PITH_FULL_IMAGE:figures/full_fig_p017_2.png] view at source ↗
Figure 3
Figure 3. Agent step 1 where the agent loads domain-specific documentation into its working mem [PITH_FULL_IMAGE:figures/full_fig_p018_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Agent step 2 where the agent tries to find more about the merchant but it fails to produce [PITH_FULL_IMAGE:figures/full_fig_p019_4.png]
Figure 5
Figure 5. Figure 5: Agent step 3 where the agent successfully finds merchant business related data. [PITH_FULL_IMAGE:figures/full_fig_p020_5.png]
Figure 6
Figure 6. Figure 6: Agent step 4 where the agent refines its understanding about fee calculations. [PITH_FULL_IMAGE:figures/full_fig_p021_6.png]
Figure 7
Figure 7. Figure 7: Agent step 5 where the agent is looking for relevant information in the merchant’s payment [PITH_FULL_IMAGE:figures/full_fig_p022_7.png]
Figure 8
Figure 8. Figure 8: Agent step 6, part 0, where agent is retrieving payment traffic from the merchant. The full [PITH_FULL_IMAGE:figures/full_fig_p023_8.png]
Figure 9
Figure 9. Figure 9: Agent step 6, part 1, where the agent joins the payment traffic information with the rules [PITH_FULL_IMAGE:figures/full_fig_p024_9.png]
Figure 10
Figure 10. Figure 10: Agent step 6, part 2, where the execution logs show the found fees by the agent. [PITH_FULL_IMAGE:figures/full_fig_p025_10.png]
Figure 11
Figure 11. Figure 11: Agent step 7 where agent produces a final answer compliant with the task guidelines. [PITH_FULL_IMAGE:figures/full_fig_p026_11.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. DataClawEval: A Benchmark for Data Engineering Agents in Real Industrial Harness

    cs.AI 2026-07 accept novelty 7.0 of 10

    A production-grounded, sandbox-graded benchmark of 100 multi-engine data-engineering tasks shows frontier agents top out at 74.9 with no cross-engine winner.

  2. UrbanDS: A Graph-Guided LLM Multi-Agent System for Data-Intensive Urban Tasks

    cs.AI 2026-07 conditional novelty 6.0 of 10

    A graph of reusable dataset skills and relations lets LLM agents discover and join the right urban datasets, outperforming prior data-science agents on data-intensive tasks.

  3. Messier: A High-Resolution Corpus for Cross-Benchmark Agent Evaluation

    cs.AI 2026-07 conditional novelty 6.0 of 10

    A unified corpus of 957k trial outcomes shows frontier progress is uneven and strict all-pass aggregation obscures capability and can reorder agents.

  4. Before the Action: Benchmarking LLMs on Prospective Hypothesis Discovery

    cs.CL 2026-07 conditional novelty 6.0 of 10

    HypoArena is a 988-case benchmark that asks LLMs to generate hypothesis sets from conclusion-free reconstructed contexts and ranks 15 models via pairwise arena judgments.

  5. FDABench: A Benchmark for Data Agents on Analytical Queries over Heterogeneous Data

    cs.DB 2025-09 conditional novelty 6.0 of 10

    FDABench is a 2,007-task benchmark for AI agents that must combine structured databases with documents, web, image, video, and audio sources to answer analytical queries.

  6. CIPHER: A Decoupled Exploration-Selection Framework for Test-Time Scaling of Data Science Agents

    cs.AI 2026-07 conditional novelty 4.0 of 10

    A decoupled exploration-selection framework for test-time scaling lifts data-science agent accuracy by up to roughly 12 percentage points over matched baselines.

  7. DSBC : Data Science task Benchmarking with Context engineering

    cs.AI 2025-07 conditional novelty 4.0 of 10

    A new 303-question benchmark of realistic data science tasks finds that Claude-4-Sonnet with multi-step code generation outperforms other LLMs and agent styles, but the accuracy numbers rest on a thinly validated auto...

Reference graph

Works this paper leans on

51 extracted references · 38 canonical work pages · cited by 7 Pith papers

  1. [1]

    Llama 3.2: Revolutionizing edge ai and vision with open, customizable models

    Meta AI. Llama 3.2: Revolutionizing edge ai and vision with open, customizable models. https://ai.meta.com/blog/ llama-3-2-connect-2024-vision-edge-mobile-devices/ , 2024

    work page 2024

  2. [2]

    The llama 4 herd: The beginning of a new era of natively multimodal ai innovation

    Meta AI. The llama 4 herd: The beginning of a new era of natively multimodal ai innovation. https://ai.meta.com/blog/llama-4-multimodal-intelligence/ , 2025

    work page 2025

  3. [3]

    Claude 3.5: Next-generation language models

    Anthropic. Claude 3.5: Next-generation language models. https://www.anthropic.com/ news/claude-3-5-sonnet , 2024

    work page 2024

  4. [4]

    Claude 3.7: Advancements in language understanding

    Anthropic. Claude 3.7: Advancements in language understanding. https://www. anthropic.com/news/claude-3-7-sonnet , 2025

    work page 2025

  5. [5]

    Ruisheng Cao, Fangyu Lei, Haoyuan Wu, Jixuan Chen, Yeqiao Fu, Hongcheng Gao, Xinzhuang Xiong, Hanchong Zhang, Wenjing Hu, Yuchen Mao, et al. Spider2-v: How far are multimodal agents from automating data science and engineering workflows? In Advances in Neural Information Processing Systems 37 (NeurIPS) , volume 37, pages 107703–107744, 2024

    work page 2024

  6. [6]

    Mle-bench: Evaluating machine learning agents on machine learning engineering

    Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, et al. Mle-bench: Evaluating machine learning agents on machine learning engineering. arXiv preprint arXiv:2410.07095, 2024

    arXiv

  7. [7]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

    arXiv 2021

  8. [8]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

Show all 51 references
  1. [9]

    Laradji, Manuel Del Verme, Tom Marty, David Vazquez, Nicolas Chapados, and Alexandre Lacoste

    Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H. Laradji, Manuel Del Verme, Tom Marty, David Vazquez, Nicolas Chapados, and Alexandre Lacoste. Workarena: how capable are web agents at solving common knowledge work tasks? ICML’24, 2024

    2024

  2. [10]

    Gemini 2.5: Our most intelligent ai model, 2025

    Google. Gemini 2.5: Our most intelligent ai model, 2025. URL https://blog.google/ technology/google-deepmind/gemini-model-thinking-updates-march-2025/ . Google Blog

    2025

  3. [11]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. 9

    2024 arXiv

  4. [12]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  5. [13]

    Text-to-SQL in the wild: A naturally-occurring dataset based on stack exchange data

    Moshe Hazoom, Vibhor Malik, and Ben Bogin. Text-to-SQL in the wild: A naturally-occurring dataset based on stack exchange data. InProceedings of the 1st Workshop on Natural Language Processing for Programming (NLP4Prog 2021), pages 77–87, 2021

    2021

  6. [14]

    Measuring mathematical problem solving with the MATH dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Bench- marks Trac...

    2021

  7. [15]

    InfiAgent-DABench: Evaluating agents on data analysis tasks

    Xueyu Hu, Ziyu Zhao, Shuang Wei, Ziwei Chai, Qianli Ma, Guoyin Wang, Xuwu Wang, Jing Su, Jingjing Xu, Ming Zhu, Yao Cheng, Jianbo Yuan, Jiwei Li, Kun Kuang, Yang Yang, Hongxia Yang, and Fei Wu. InfiAgent-DABench: Evaluating agents on data analysis tasks. In Proceedings of the ...

    2024

  8. [16]

    Mlagentbench: evaluating language agents on machine learning experimentation

    Qian Huang, Jian V ora, Percy Liang, and Jure Leskovec. Mlagentbench: evaluating language agents on machine learning experimentation. In Proceedings of the 41st International Confer- ence on Machine Learning, ICML’24, 2024

    2024

  9. [17]

    DA-code: Agent data science code generation benchmark for large language models

    Yiming Huang, Jianwen Luo, Yan Yu, Yitong Zhang, Fangyu Lei, Yifan Wei, Shizhu He, Lifu Huang, Xiao Liu, Jun Zhao, and Kang Liu. DA-code: Agent data science code generation benchmark for large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natur...

    2024

  10. [18]

    Financebench: A new benchmark for financial question answering, 2023

    Pranab Islam, Anand Kannappan, Douwe Kiela, Rebecca Qian, Nino Scherrer, and Bertie Vidgen. Financebench: A new benchmark for financial question answering, 2023

    2023

  11. [19]

    SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024

    2024

  12. [20]

    DSBench: How far are data science agents from becoming data science experts? In The Thirteenth International Conference on Learning Representa- tions, 2025

    Liqiang Jing, Zhehui Huang, Xiaoyang Wang, Wenlin Yao, Wenhao Yu, Kaixin Ma, Hongming Zhang, Xinya Du, and Dong Yu. DSBench: How far are data science agents from becoming data science experts? In The Thirteenth International Conference on Learning Representa- tions, 2025

    2025

  13. [21]

    Ds-1000: A natural and reliable benchmark for data science code generation

    Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen- tau Yih, Daniel Fried, Sida Wang, and Tao Yu. Ds-1000: A natural and reliable benchmark for data science code generation. In International Conference on Machine Learning , pages 18319–18345....

    2023

  14. [22]

    KaggleDBQA: Realistic eval- uation of text-to-SQL parsers

    Chia-Hsuan Lee, Oleksandr Polozov, and Matthew Richardson. KaggleDBQA: Realistic eval- uation of text-to-SQL parsers. InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Proce...

    2021

  15. [23]

    Spider 2.0: Evaluating language models on real-world enterprise text-to-SQL workflows

    Fangyu Lei, Jixuan Chen, Yuxiao Ye, Ruisheng Cao, Dongchan Shin, Hongjin SU, ZHAO- QING SUO, Hongcheng Gao, Wenjing Hu, Pengcheng Yin, Victor Zhong, Caiming Xiong, Ruoxi Sun, Qian Liu, Sida Wang, and Tao Yu. Spider 2.0: Evaluating language models on real-world enterprise text-...

    2025

  16. [24]

    Can llm already serve as a database interface? a big benchmark for large-scale database-grounded text-to-sqls

    Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, et al. Can llm already serve as a database interface? a big benchmark for large-scale database-grounded text-to-sqls. In Advances in Neural Information Processing Sy...

    2023

  17. [25]

    Xi Victoria Lin, Chenglong Wang, Luke Zettlemoyer, and Michael D. Ernst. NL2Bash: A corpus and semantic parser for natural language interface to the linux operating system. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)...

    2018

  18. [26]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  19. [27]

    Agentbench: Evaluating LLMs as agents

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. Agentbench:...

    2024

  20. [28]

    Gaia: a benchmark for general ai assistants

    Gr ´egoire Mialon, Cl ´ementine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants. InThe Twelfth International Conference on Learn- ing Representations, 2023

    2023

  21. [29]

    GSM-symbolic: Understanding the limitations of mathematical reasoning in large language models

    Seyed Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. GSM-symbolic: Understanding the limitations of mathematical reasoning in large language models. In The Thirteenth International Conference on Learning Represen- tations, 2025

    2025

  22. [30]

    Introducing gpt-4o: Multimodal capabilities and efficiency

    OpenAI. Introducing gpt-4o: Multimodal capabilities and efficiency. https://openai.com/ index/hello-gpt-4o, 2024

    2024

  23. [31]

    Gpt-4o mini: advancing cost-efficient intelligence

    OpenAI. Gpt-4o mini: advancing cost-efficient intelligence. https://openai.com/index/ gpt-4o-mini-advancing-cost-efficient-intelligence/ , 2024

    2024

  24. [32]

    Openai o1 and new tools for developers

    OpenAI. Openai o1 and new tools for developers. https://openai.com/index/ o1-and-new-tools-for-developers , 2024

    2024

  25. [33]

    Gpt-4.1 technical overview

    OpenAI. Gpt-4.1 technical overview. https://openai.com/index/gpt-4-1, 2025

    2025

  26. [34]

    Introducing openai o3 and o4-mini

    OpenAI. Introducing openai o3 and o4-mini. https://openai.com/index/ introducing-o3-and-o4-mini , 2025

    2025

  27. [35]

    On the difficulty of evaluating baselines: A study on recommender systems

    Steffen Rendle, Li Zhang, and Yehuda Koren. On the difficulty of evaluating baselines: A study on recommender systems. arXiv preprint arXiv:1905.01395, 2019

    1905 arXiv

  28. [36]

    smolagents: A smol library to build great agentic systems, 2025

    Aymeric Roucher, Albert Villanova del Moral, Thomas Wolf, Leandro von Werra, and Erik Kaunism¨aki. smolagents: A smol library to build great agentic systems, 2025. URL https: //github.com/huggingface/smolagents. GitHub repository

    2025

  29. [37]

    Let me speak freely? a study on the impact of format restrictions on large language model performance

    Zhi Rui Tam, Cheng-Kuang Wu, Yi-Lin Tsai, Chieh-Yen Lin, Hung-yi Lee, and Yun-Nung Chen. Let me speak freely? a study on the impact of format restrictions on large language model performance. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Proces...

    2024

  30. [38]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems, volume 30, 2017

    2017

  31. [39]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations, 2019

    2019

  32. [40]

    Text-to-sql generation for question answering on electronic medical records

    Ping Wang, Tian Shi, and Chandan K Reddy. Text-to-sql generation for question answering on electronic medical records. In Proceedings of The Web Conference 2020, pages 350–361, 2020. 11

    2020

  33. [41]

    Won- derbread: A benchmark for evaluating multimodal foundation models on business process management tasks

    Michael Wornow, Avanika Narayan, Ben Viggiano, Ishan Khare, Tathagat Verma, Tibor Thompson, Miguel Hernandez, Sudharsan Sundar, Chloe Trujillo, Krrish Chawla, et al. Won- derbread: A benchmark for evaluating multimodal foundation models on business process management tasks. In...

    2024

  34. [42]

    Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments

    Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. InAdvances in Neural Information Processing...

    2024

  35. [43]

    Sorry-bench: Systematically evaluating large language model safety refusal behaviors

    Tinghao Xie, Xiangyu Qi, Yi Zeng, Yangsibo Huang, Udari Madhushani Sehwag, Kaixuan Huang, Luxi He, Boyi Wei, Dacheng Li, Ying Sheng, et al. Sorry-bench: Systematically evaluating large language model safety refusal behaviors. arXiv preprint arXiv:2406.14598, 2024

    2024 arXiv

  36. [44]

    Intercode: Standard- izing and benchmarking interactive coding with execution feedback

    John Yang, Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao. Intercode: Standard- izing and benchmarking interactive coding with execution feedback. In Advances in Neural Information Processing Systems, volume 36, pages 23826–23854, 2023

    2023

  37. [45]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. InInternational Conference on Learning Representations (ICLR), 2023

    2023

  38. [46]

    Natural language to code generation in interactive data science notebooks

    Pengcheng Yin, Wen-Ding Li, Kefan Xiao, Abhishek Rao, Yeming Wen, Kensen Shi, Joshua Howland, Paige Bailey, Michele Catasta, Henryk Michalewski, Oleksandr Polozov, and Charles Sutton. Natural language to code generation in interactive data science notebooks. In Proceedings of ...

    2023

  39. [47]

    Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task

    Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Proceed...

    2018

  40. [48]

    Benchmarking data science agents

    Yuge Zhang, Qiyang Jiang, XingyuHan XingyuHan, Nan Chen, Yuqing Yang, and Kan Ren. Benchmarking data science agents. In Proceedings of the 62nd Annual Meeting of the Associ- ation for Computational Linguistics (Volume 1: Long Papers), pages 5677–5700, 2024

    2024

  41. [49]

    Judging llm-as-a-judge with mt- bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt- bench and chatbot arena. In Advances in Neural Information Processing Systems, volume 36, pages 46595–46623, 2023

    2023

  42. [50]

    Seq2sql: Generating structured queries from natural language using reinforcement learning

    Victor Zhong, Caiming Xiong, and Richard Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning. arXiv preprint arXiv:1709.00103, 2017

    2017 arXiv

  43. [51]

    Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig

    Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web environment for building autonomous agents. In The Twelfth International Conference on Learni...

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.