Pith. sign in

REVIEW 4 major objections 6 minor 49 references

DSBC : Data Science task Benchmarking with Context engineering

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper introduces DSBC, a 303-question benchmark built from real user logs, and reports that Claude-4-Sonnet generating multi-step code from engineered context clearly outperforms the other models and approaches.

desk verdict Useful usage-grounded data-science benchmark with a thoughtful context-engineering schema, but the headline rankings rest on a barely validated VLM judge and the dataset isn't released yet, so treat the numbers as provisional. read the letter →

arxiv 2507.23336 v2 pith:YBIEJNV4 submitted 2025-07-31 cs.AI cs.CLcs.MA

Ram Mohan Rao Kadiyala , Siddhant Gupta , Jebish Purbey , Giulio Martini , Ali Shafique , Suman Debnath , Hamza Farooq This is my paper
classification cs.AIcs.CLcs.MA
keywords datascienceagentsbenchmarkcontextengineeringlargelanguagemodelsLLMevaluationtemperaturesensitivityleakageVLM-as-a-judge
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that data-science agents should be judged on tasks real users actually ask, and it builds DSBC, a 303-question benchmark derived from logged usage of a commercial data-science agent. Every query is paired with a 'clean' rewrite that strips data leakage, and instead of hand-written dataset descriptions the model receives a standardized structured context listing column names, types, null counts, summaries, and the first five rows. Testing Claude-4-Sonnet, Gemini-2.5-Flash, and OpenAI-o4-Mini under three execution modes and eleven temperature settings, the paper reports that Claude-4-Sonnet with multi-step code generation clearly outperforms the field, that multi-step modes beat single-shot code generation, and that accuracy falls sharply as a query spans more task categories. If these results hold, model and approach choice matters far more than temperature when deploying a data-science agent.

What carries the argument

The load-bearing mechanism is context engineering: a standardized nested-JSON description of each data file — row and column counts, column names, data types, null counts, numeric min/max/mean/median/percentiles, categorical unique counts, datetime ranges and frequency, and the first five rows — that is inserted into the prompt in place of raw files or hand-written descriptions. Its job is to give the model enough information to solve the query while keeping sensitive data out of the context and removing the randomness that manually written descriptions introduce. A second mechanism is the raw/clean query pairing: each question exists in a leaked 'raw' form and a leakage-free 'clean' rewrite, letting the benchmark measure how much a prompt's hidden assumptions inflate apparent performance. The accuracy numbers themselves are produced by a VLM-as-a-judge setup in which Gemini-2.5-Flash returns a single-word Yes/No verdict in a JSON schema.

What would settle it

Grade a random sample of 30-50 responses across all fifteen model-approach-query-type setups with two independent human annotators against the ground-truth code and answer, and compare their verdicts with the VLM judge's Yes/No labels. If human-judge agreement falls below roughly 95%, or if the human-graded ranking does not place Claude-4-Sonnet multi-code-cell first, the paper's central performance claims are not established.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is a stable set of performance disparities on realistic queries: Claude-4-Sonnet using a multi-code-cell approach (two to three code snippets generated in one pass, then executed in sequence) reaches the highest overall accuracy of any setup (roughly 58-60% averaged across temperatures, peaking just above 62%), ahead of the same model steering the SmolAgent agent framework (about 55%) and far ahead of single-cell direct generation, where OpenAI-o4-Mini is the best of the three (about 31%). Multi-step code generation and SmolAgent both clearly beat single-cell generation, temperature shows no systematic effect for any model-approach combination, and Gemini-2.5-Flash is the most temperature-sensitive model. The paper also reports that accuracy drops from 55.9% on queries covering one task category to 43.1% on two-category queries and 25.5% on three-category queries, and that roughly 70% of failures are incorrect final responses rather than code, data-access, or formatting errors.

Load-bearing premise

Every accuracy number and ranking in the paper comes from a vision-language model judge (Gemini-2.5-Flash) that was checked against human judgment on only 3 of the 165 attempts, with no inter-annotator agreement reported, so even a small rate of judge error could change the rankings and the temperature findings.

Editorial extensions

If this is right

  • A standardized structured context can replace both raw data files and hand-written descriptions in benchmarks and deployments, addressing privacy concerns at the cost of a fixed metadata format.
  • The multi-code-cell advantage over SmolAgent — at 1.8x versus 3.2x the single-cell cost — implies that structured multi-step generation can match or beat a full agent framework for less money.
  • Because accuracy falls from 55.9% on one-category queries to 25.5% on three-category queries, multi-skill queries are the bottleneck that future agents must target, not single-skill ones.
  • The raw-versus-clean gap shows that prompt-level data leakage can distort measured performance, so benchmarks should ship leakage-free rewrites alongside original queries.
  • Task-category and model-approach variance suggest a query-routing policy — matching each incoming query to the cheapest model and approach that handles its category well — could preserve most accuracy while cutting inference cost.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The rankings and temperature findings rest on a judge that was manually validated on 3 of 165 attempts with no reported inter-annotator agreement; an obvious next step is to re-grade a larger random sample with independent human annotators and re-rank the setups on human verdicts.
  • The benchmark's design makes a comparison the authors did not run: holding queries fixed and swapping the structured JSON context for a hand-written description would isolate how much of the measured performance is attributable to the context format itself.
  • The accuracy staircase across one, two, and three task categories hints at compounding compositional failure — errors in one sub-task cascade into later ones — which a progress measure could track by scoring sub-task plans separately from final answers.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces DSBC, a data science agent benchmark built from 303 queries derived from observed usage of the authors' commercial data science applications. The benchmark spans eight task categories, uses 11 Kaggle datasets, and applies a structured context-engineering format rather than free-form data descriptions. The authors evaluate three LLMs (Claude-4-Sonnet, Gemini-2.5-Flash, OpenAI-o4-Mini) across three approaches (single-step code, multi-step code, SmolAgent), two query types (raw and clean), and eleven temperature values, yielding 165 attempts per sample. Accuracy is determined by a Gemini-2.5-Flash VLM-as-a-judge with a JSON schema and regex parsing. The central finding is that Claude-4-Sonnet with multi-step code generation outperforms the other configurations, that multi-step and SmolAgent beat single-step approaches overall, and that no significant temperature pattern is observed. The benchmark and evaluation framework are released under Apache 2.0.

Significance. If the evaluation methodology is sound, the benchmark is a useful contribution: it is derived from real user interactions, includes multi-label task categories, pairs raw and clean versions of queries to study data leakage, and uses context engineering to avoid manual dataset descriptions. The release of the dataset and framework under an open license is a concrete asset for future data science agent research. The paper also reports cost ratios across approaches, which is practically relevant. However, the headline accuracy comparisons and temperature-sensitivity conclusions rest entirely on an unvalidated VLM judge, and the paper reports no statistical significance testing; these issues currently limit the strength of the conclusions.

major comments (4)
  1. [Section 7, Appendix A (Prompt 5)] The VLM-as-a-judge is insufficiently validated for the load-bearing accuracy claims. The paper states that only three of the 165 attempts were manually checked; since each of the 303 samples is evaluated over 165 attempts, this amounts to at most three of 49,995 judge decisions, and no inter-annotator agreement is reported. A judge error rate of even a few percent, if correlated with model, approach, or temperature, could change the rankings in Table 7 and the 'no significant temperature pattern' conclusion in Section 8. The fact that Gemini-2.5-Flash serves as both a judged model and the judge raises a self-preference concern that is not addressed. I ask the authors to validate the judge on a stratified random sample of outputs covering all models, approaches, query types, and temperatures, report agreement against human labels, and ideally use a second independent judge or a rubric-based human evaluation for at least a representative subset.
  2. [Section 8, Table 7] The central claim that Claude-4-Sonnet with multi-code 'clearly outperformed' the rest is not supported by any significance testing. Several adjacent configurations differ by only 2–5 percentage points (e.g., Claude Multi-Code Clean at 58.36% vs. Claude SmolAgent Clean at 55.24%; Gemini Multi-Code Clean at 49.06% vs. Gemini Multi-Code Raw at 44.16%). With 303 samples and 11 temperature values, paired per-sample comparisons or bootstrap confidence intervals are needed to determine whether these gaps are distinguishable from noise. The same applies to the statement that no significant temperature pattern exists; the paper currently reports only descriptive statistics and plots.
  3. [Section 7.3 vs. Appendix B] The SmolAgent step limit is inconsistent: Section 7.3 states a limit of 8 steps, while Appendix B states a step limit of 5 for each query during inference. This is a reproducibility issue that affects the validity of all SmolAgent results, since the number of allowed steps directly determines how much context the agent can gather. The authors should state the correct value and, if both values were used in different runs, explain which runs used which limit.
  4. [Appendix A, Prompt 5] The judge prompt specifies that numerical values 'rounded .2f values should match' and that errors beyond the second decimal after rounding can be ignored, but the paper does not analyze how sensitive the results are to this tolerance. Since many queries have numeric answers, a loose or inconsistent rounding rule could systematically favor certain output formats. Please report the distribution of judge decisions by answer type and, if possible, assess robustness to the tolerance parameter.
minor comments (6)
  1. [Section 8] The reference 'as seen in Figure 46' appears in the main-text discussion of temperature effects, but Figure 46 is placed in Appendix I and appears to be a general results plot; the intended cross-reference is likely Figure 4 or a dedicated temperature figure. Please correct the reference.
  2. [Section 8] The heading 'Variations in results with Model and approach' is repeated twice in the same section; please rename the second occurrence to reflect its content, which concerns task-domain differences.
  3. [Table 4] The column header 'A VG.ACC' should read 'AVG.ACC' and should be defined in the caption as the average score over all attempts and temperature values for queries using that dataset.
  4. [Appendix C] There is a typo in the annotation guidelines: 'categorizied' should be 'categorized'.
  5. [Appendix H] The captions for Figures 31–45 are empty or incomplete ('Incorrect response cause distribution -'); please provide descriptive captions that identify the model, approach, and query type for each figure.
  6. [Section 7, footnote 1] The footnote explaining that SmolAgent was initially tested with one randomly chosen temperature value is unclear: it is not specified which temperature was chosen, and the sentence 'no change was observed in results between the Raw and Clean queries' would benefit from reporting the actual values.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the benchmark's claims are empirical measurements, not derived from their inputs.

full rationale

This paper is an empirical benchmark study rather than a derivation. The central claim—that Claude-4-Sonnet with multi-step code generation outperforms the other configurations—rests on measured success rates over 303 samples, with correctness judged against manually coded ground truth by a VLM judge. No parameter is fitted to the data and then renamed as a prediction; no equation is derived from itself; and no load-bearing result is imported from the authors' prior work via self-citation. The use of Gemini-2.5-Flash as the VLM judge while also evaluating Gemini-2.5-Flash is a measurement-validity concern (possible self-preference or correlated judge errors), not circularity, because the correctness criterion is defined by the manually coded ground truth and the judge's verdict is not used to define the benchmark's inputs. Likewise, the limited manual validation (three of 165 attempts) is a reliability limitation, not a circular-reasoning flaw. The benchmark construction is based on observed usage of the authors' commercial applications, which is an empirical source rather than a restatement of the conclusions. Therefore no specific circular step can be identified, and the paper is self-contained with respect to circularity concerns.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim depends on the VLM judge accuracy, the representativeness of the observed queries, and the correctness of manual solutions; these are unvalidated domain assumptions. No theoretical free parameters are fitted, but several evaluation design choices are hand-set.

free parameters (3)
  • VLM judge numeric tolerance = 0.01 (2 decimal places)
    The judge prompt states that numeric values should match after rounding to 2 decimal places; this hand-chosen tolerance determines whether responses are marked correct.
  • Context engineering sample rows = 5 rows
    The context includes the first 5 rows of each data file (Table 3), chosen by hand; this affects model performance and is not an input from prior benchmarks.
  • SmolAgent step limit = 8 or 5 (reported inconsistently)
    Section 7.3 states a limit of 8 steps, while Appendix B states 5; this inconsistency affects reproducibility and confounds results.
assumptions (3)
  • domain assumption VLM-as-a-judge using Gemini-2.5-Flash produces accurate correctness labels
    All accuracy numbers depend on this; only 3 of 165 attempts were manually verified (Section 7).
  • domain assumption Queries observed from the authors' commercial applications are representative of real-world data science agent usage
    The benchmark's realism claim rests on this; no external validation or sampling methodology is provided (Section 3).
  • domain assumption Manual solutions coded by annotators are correct ground truth
    The benchmark assumes the annotator-written solutions are correct; no independent check is described.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DSBC : Data Science task Benchmarking with Context engineering." pith.science (2026). https://pith.science/paper/YBIEJNV4

@misc{pith2026250723336,
  author       = {Pith},
  title        = {Pith review of: DSBC : Data Science task Benchmarking with Context engineering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YBIEJNV4}},
  note         = {Machine review of arXiv:2507.23336}
}
read the original abstract

Recent advances in large language models (LLMs) have significantly impacted data science workflows, giving rise to specialized data science agents designed to automate analytical tasks. Despite rapid adoption, systematic benchmarks evaluating the efficacy and limitations of these agents remain scarce. In this paper, we introduce a comprehensive benchmark specifically crafted to reflect real-world user interactions with data science agents by observing usage of our commercial applications. We evaluate three LLMs: Claude-4.0-Sonnet, Gemini-2.5-Flash, and OpenAI-o4-Mini across three approaches: zero-shot with context engineering, multi-step with context engineering, and with SmolAgent. Our benchmark assesses performance across a diverse set of eight data science task categories, additionally exploring the sensitivity of models to common prompting issues, such as data leakage and slightly ambiguous instructions. We further investigate the influence of temperature parameters on overall and task-specific outcomes for each model and approach. Our findings reveal distinct performance disparities among the evaluated models and methodologies, highlighting critical factors that affect practical deployment. The benchmark dataset and evaluation framework introduced herein aim to provide a foundation for future research of more robust and effective data science agents.

Figures

Figures reproduced from arXiv: 2507.23336 by the authors.

Figure 1
Figure 1. An example each from our benchmark along with some works cited above which were published less than [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. An overview of the annotation process used [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Success rates for each sample (303) of our benchmark over several (165) attempts [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (43 more)
Figure 4
Figure 4. Figure 4: Variation in Accuracies through each set of (LLM, query type and approach) [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 6
Figure 6. Figure 6: Average accuracy with each attempt VS num [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 5
Figure 5. Figure 5: Error cause distribution overall : temperature [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 7
Figure 7. Figure 7: Success rates for each sample over several (165) attempts - Correlation Analysis [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Success rates for each sample over several (165) attempts - Statistics [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Success rates for each sample over several (165) attempts - Data Parsing [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Success rates for each sample over several (165) attempts - Data Pre-processing [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: Success rates for each sample over several (165) attempts - Feature Engineering [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]
Figure 12
Figure 12. Figure 12: Success rates for each sample over several (165) attempts - Feature Transformation [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]
Figure 13
Figure 13. Figure 13: Success rates for each sample over several (165) attempts - Distribution Analysis [PITH_FULL_IMAGE:figures/full_fig_p018_13.png]
Figure 14
Figure 14. Figure 14: Success rates for each sample over several (165) attempts - Data Visualization [PITH_FULL_IMAGE:figures/full_fig_p018_14.png]
Figure 15
Figure 15. Figure 15: Results from each attempt over each temperature value used - Correlation Analysis [PITH_FULL_IMAGE:figures/full_fig_p019_15.png]
Figure 16
Figure 16. Figure 16: Results from each attempt over each temperature value used - Statistics [PITH_FULL_IMAGE:figures/full_fig_p019_16.png]
Figure 17
Figure 17. Figure 17: Results from each attempt over each temperature value used - Data Parsing [PITH_FULL_IMAGE:figures/full_fig_p020_17.png]
Figure 18
Figure 18. Figure 18: Results from each attempt over each temperature value used - Data Pre-processing [PITH_FULL_IMAGE:figures/full_fig_p020_18.png]
Figure 19
Figure 19. Figure 19: Results from each attempt over each temperature value used - Feature Engineering [PITH_FULL_IMAGE:figures/full_fig_p020_19.png]
Figure 20
Figure 20. Figure 20: Results from each attempt over each temperature value used - Feature Transformation [PITH_FULL_IMAGE:figures/full_fig_p021_20.png]
Figure 21
Figure 21. Figure 21: Results from each attempt over each temperature value used - Distribution Analysis [PITH_FULL_IMAGE:figures/full_fig_p021_21.png]
Figure 22
Figure 22. Figure 22: Results from each attempt over each temperature value used - Data Visualization [PITH_FULL_IMAGE:figures/full_fig_p021_22.png]
Figure 23
Figure 23. Figure 23: Variation in Accuracies through - Correlation Analysis [PITH_FULL_IMAGE:figures/full_fig_p022_23.png]
Figure 24
Figure 24. Figure 24: Variation in Accuracies through - Statistics [PITH_FULL_IMAGE:figures/full_fig_p022_24.png]
Figure 25
Figure 25. Figure 25: Variation in Accuracies through - Data Parsing [PITH_FULL_IMAGE:figures/full_fig_p023_25.png]
Figure 26
Figure 26. Figure 26: Variation in Accuracies through - Data Pre-processing [PITH_FULL_IMAGE:figures/full_fig_p023_26.png]
Figure 27
Figure 27. Figure 27: Variation in Accuracies through - Feature Engineering [PITH_FULL_IMAGE:figures/full_fig_p023_27.png]
Figure 28
Figure 28. Figure 28: Variation in Accuracies through - Feature Transformation [PITH_FULL_IMAGE:figures/full_fig_p024_28.png]
Figure 29
Figure 29. Figure 29: Variation in Accuracies through - Distribution Analysis [PITH_FULL_IMAGE:figures/full_fig_p024_29.png]
Figure 30
Figure 30. Figure 30: Variation in Accuracies through - Data Visualization [PITH_FULL_IMAGE:figures/full_fig_p024_30.png]
Figure 31
Figure 31. Figure 31: Incorrect response cause distribution - [PITH_FULL_IMAGE:figures/full_fig_p025_31.png]
Figure 32
Figure 32. Figure 32: Incorrect response cause distribution - [PITH_FULL_IMAGE:figures/full_fig_p025_32.png]
Figure 33
Figure 33. Figure 33: Incorrect response cause distribution - [PITH_FULL_IMAGE:figures/full_fig_p026_33.png]
Figure 34
Figure 34. Figure 34: Incorrect response cause distribution - [PITH_FULL_IMAGE:figures/full_fig_p026_34.png]
Figure 35
Figure 35. Figure 35: Incorrect response cause distribution - [PITH_FULL_IMAGE:figures/full_fig_p027_35.png]
Figure 36
Figure 36. Figure 36: Incorrect response cause distribution - [PITH_FULL_IMAGE:figures/full_fig_p027_36.png]
Figure 37
Figure 37. Figure 37: Incorrect response cause distribution - [PITH_FULL_IMAGE:figures/full_fig_p028_37.png]
Figure 38
Figure 38. Figure 38: Incorrect response cause distribution - [PITH_FULL_IMAGE:figures/full_fig_p028_38.png]
Figure 39
Figure 39. Figure 39: Incorrect response cause distribution - [PITH_FULL_IMAGE:figures/full_fig_p029_39.png]
Figure 40
Figure 40. Figure 40: Incorrect response cause distribution - [PITH_FULL_IMAGE:figures/full_fig_p029_40.png]
Figure 41
Figure 41. Figure 41: Incorrect response cause distribution - [PITH_FULL_IMAGE:figures/full_fig_p030_41.png]
Figure 42
Figure 42. Figure 42: Incorrect response cause distribution - [PITH_FULL_IMAGE:figures/full_fig_p030_42.png]
Figure 43
Figure 43. Figure 43: Incorrect response cause distribution - [PITH_FULL_IMAGE:figures/full_fig_p031_43.png]
Figure 44
Figure 44. Figure 44: Incorrect response cause distribution - [PITH_FULL_IMAGE:figures/full_fig_p031_44.png]
Figure 45
Figure 45. Figure 45: Incorrect response cause distribution - I Other Plots [PITH_FULL_IMAGE:figures/full_fig_p032_45.png]
Figure 46
Figure 46. Figure 46: Results from each attempt over each temperature value used [PITH_FULL_IMAGE:figures/full_fig_p032_46.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

49 extracted references · 36 canonical work pages

  1. [1]

    Anthropic. 2025. https://www.anthropic.com/news/claude-4 Introducing the next generation of claude

    work page 2025

  2. [2]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732

    arXiv 2021

  3. [3]

    Bhanu Pratap Biswas. 2024. https://www.kaggle.com/datasets/bhanupratapbiswas/weather-data Weather data

    work page 2024

  4. [4]

    Ruisheng Cao, Fangyu Lei, Haoyuan Wu, Jixuan Chen, Yeqiao Fu, Hongcheng Gao, Xinzhuang Xiong, Hanchong Zhang, Wenjing Hu, Yuchen Mao, et al. 2024. Spider2-v: How far are multimodal agents from automating data science and engineering workflows? Advances in Neural Information Processing Systems, 37:107703--107744

    work page 2024

  5. [5]

    Saikat Chakraborty, Yujian Li, Matt Irvine, Ripon Saha, and Baishakhi Ray. 2018. Entropy guided spectrum based bug localization using statistical language model. arXiv preprint arXiv:1802.06947

    arXiv 2018

  6. [6]

    Saikat Chakraborty and Baishakhi Ray. 2021. On multi-modal learning of editing source code. In 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 443--455. IEEE

    work page 2021

  7. [7]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374

    arXiv 2021

  8. [8]

    Miri Choi. 2021. https://www.kaggle.com/datasets/mirichoi0218/insurance Insurance

    work page 2021

Show all 49 references
  1. [9]

    Jackson Crow. 2020. https://www.kaggle.com/datasets/jacksoncrow/stock-market-dataset Stock market dataset

    2020

  2. [10]

    Anxo DS. 2024. https://www.kaggle.com/datasets/anxods/world-population-and-forecast-dataset World population and forecast dataset

    2024

  3. [11]

    Alex Egg, Martin Iglesias Goyanes, Friso Kingma, Andreu Mora, Leandro von Werra, and Thomas Wolf. 2025. Dabstep: Data agent benchmark for multi-step reasoning. arXiv preprint arXiv:2506.23719

    2025 arXiv

  4. [12]

    Sarah Fakhoury, Saikat Chakraborty, Madan Musuvathi, and Shuvendu K Lahiri. 2023. Towards generating functionally correct code edits from natural language issue descriptions. arXiv preprint arXiv:2304.03816

    2023 arXiv

  5. [13]

    Zhiyu Fan, Xiang Gao, Martin Mirchev, Abhik Roychoudhury, and Shin Hwei Tan. 2023. Automated repair of programs from large language models. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pages 1469--1481. IEEE

    2023

  6. [14]

    Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. 2025. Measuring coding challenge competence with apps. In Thirty-fifth Conference on Neural Information Processing Systems Datasets a...

    2025

  7. [15]

    Aradhana Hirapara. 2023. https://www.kaggle.com/datasets/aradhanahirapara/farm-produce-data-80-years Farm produce data 80 years

    2023

  8. [16]

    Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. 2024 a . Mlagentbench: Evaluating language agents on machine learning experimentation. In International Conference on Machine Learning, pages 20271--20309. PMLR

    2024

  9. [17]

    Yiming Huang, Jianwen Luo, Yan Yu, Yitong Zhang, Fangyu Lei, Yifan Wei, Shizhu He, Lifu Huang, Xiao Liu, Jun Zhao, et al. 2024 b . Da-code: Agent data science code generation benchmark for large language models. In Proceedings of the 2024 Conference on Empirical Methods in Nat...

    2024

  10. [18]

    Abhishek Jha. 2024. https://www.kaggle.com/datasets/abhisheksjha/time-series-air-quality-data-of-india-2010-2023 Time series air quality data of india (2010-2023)

    2024

  11. [19]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2024. http://arxiv.org/abs/2310.06770 Swe-bench: Can language models resolve real-world github issues?

    2024 arXiv

  12. [20]

    Liqiang Jing, Zhehui Huang, Xiaoyang Wang, Wenlin Yao, Wenhao Yu, Kaixin Ma, Hongming Zhang, Xinya Du, and Dong Yu. 2025. Dsbench: How far are data science agents from becoming data science experts? In The Thirteenth International Conference on Learning Representations

    2025

  13. [21]

    Sungmin Kang, Juyeon Yoon, and Shin Yoo. 2023. Large language models are few-shot testers: Exploring llm-based general bug reproduction. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pages 2312--2323. IEEE

    2023

  14. [22]

    Yunho Kim, Seokhyeon Mun, Shin Yoo, and Moonzoo Kim. 2019. Precise learn-to-rank fault localization using dynamic and static features of target programs. ACM Transactions on Software Engineering and Methodology (TOSEM), 28(4):1--34

    2019

  15. [23]

    Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu. 2023. Ds-1000: A natural and reliable benchmark for data science code generation. In International Conference on Machine Learning, pages 18319--1...

    2023

  16. [24]

    Zhiyu Li, Shuai Lu, Daya Guo, Nan Duan, Shailesh Jannu, Grant Jenks, Deep Majumder, Jared Green, Alexey Svyatkovskiy, Shengyu Fu, et al. 2022. Automating code review activities by large-scale pre-training. In Proceedings of the 30th ACM Joint European Software Engineering Conf...

    2022

  17. [25]

    Xiao Liu, Zirui Wu, Xueqing Wu, Pan Lu, Kai-Wei Chang, and Yansong Feng. 2024. https://aclanthology.org/2024.findings-acl.548 Are LLM s capable of data-based statistical and causal reasoning? benchmarking advanced quantitative reasoning with data . In Findings of the Associati...

    2024

  18. [26]

    Lingrui Mei, Jiayu Yao, Yuyao Ge, Yiwei Wang, Baolong Bi, Yujun Cai, Jiazhi Liu, Mingyu Li, Zhong-Zhi Li, Duzhen Zhang, Chenlin Zhou, Jiayi Mao, Tianze Xia, Jiafeng Guo, and Shenghua Liu. 2025. http://arxiv.org/abs/2507.13334 A survey of context engineering for large language models

    2025 arXiv

  19. [27]

    Mikhail. 2024. https://www.kaggle.com/datasets/mikhail1681/walmart-sales Walmart sales

    2024

  20. [28]

    Meir Nizri. 2022. https://www.kaggle.com/datasets/meirnizri/covid19-dataset Covid-19 dataset

    2022

  21. [29]

    OpenAI. 2025. https://cdn.openai.com/pdf/4375e605-f9a6-438d-bcc8-190599c183a6/o3_cua_system_card.pdf o4-mini system card

    2025

  22. [30]

    Ulrik Thyge Pedersen. 2023. https://www.kaggle.com/datasets/ulrikthygepedersen/life-expectancy Life expectancy

    2023

  23. [31]

    Aymeric Roucher, Albert Villanova del Moral, Thomas Wolf, Leandro von Werra, and Erik Kaunismäki. 2025. `smolagents`: a smol library to build great agentic systems

    2025

  24. [32]

    Saurabh Shahane. 2023. https://www.kaggle.com/datasets/saurabhshahane/electricity-load-forecasting Electricity load forecasting

    2023

  25. [33]

    Dominik Sobania, Martin Briesch, Carol Hanna, and Justyna Petke. 2023. An analysis of the automatic bug fixing performance of chatgpt. In 2023 IEEE/ACM International Workshop on Automated Program Repair (APR), pages 23--30. IEEE

    2023

  26. [34]

    Ansh Tanwar. 2023. https://www.kaggle.com/datasets/anshtanwar/monthly-food-price-estimates Monthly food price estimates

    2023

  27. [35]

    Gemini Team. 2025. http://arxiv.org/abs/2507.06261 Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities

    2025 arXiv

  28. [36]

    Rosalia Tufano. 2023. Automating code review. In 2023 IEEE/ACM 45th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion), pages 192--196. IEEE

    2023

  29. [37]

    Junjie Wang, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang, and Qing Wang. 2024. Software testing with large language models: Survey, landscape, and vision. IEEE Transactions on Software Engineering, 50(4):911--936

    2024

  30. [38]

    Chunqiu Steven Xia, Matteo Paltenghi, Jia Le Tian, Michael Pradel, and Lingming Zhang. 2024. Fuzz4all: Universal fuzzing with large language models. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pages 1--13

    2024

  31. [39]

    Chunqiu Steven Xia and Lingming Zhang. 2022. Less training, more repairing please: revisiting automated program repair via zero-shot learning. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering...

    2022

  32. [40]

    Xin Yang, Raula Gaikovina Kula, Norihiro Yoshida, and Hajimu Iida. 2016. Mining the modern code review repositories: A dataset of people, process and product. In 2016 IEEE/ACM 13th Working Conference on Mining Software Repositories (MSR), pages 460--463

    2016

  33. [41]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR)

    2023

  34. [42]

    Pengcheng Yin, Wen-Ding Li, Kefan Xiao, Abhishek Rao, Yeming Wen, Kensen Shi, Joshua Howland, Paige Bailey, Michele Catasta, Henryk Michalewski, et al. 2023. Natural language to code generation in interactive data science notebooks. In Proceedings of the 61st Annual Meeting of...

    2023

  35. [43]

    Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018 a . https://doi.org/10.18653/v1/D18-1425 S pider: A large-scale human-labeled dataset for complex and cross-domain se...

    2018 doi

  36. [44]

    Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al. 2018 b . Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Proceedings of the 2018 ...

    2018

  37. [45]

    Dan Zhang, Sining Zhoubian, Min Cai, Fengzu Li, Lekang Yang, Wei Wang, Tianjiao Dong, Ziniu Hu, Jie Tang, and Yisong Yue. 2025. Datascibench: An llm agent benchmark for data science

    2025

  38. [46]

    Jiyang Zhang, Sheena Panthaplackel, Pengyu Nie, Junyi Jessy Li, and Milos Gligoric. 2022. Coditt5: Pretraining for source code and natural language editing. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, pages 1--12

    2022

  39. [47]

    Yuge Zhang, Qiyang Jiang, XingyuHan XingyuHan, Nan Chen, Yuqing Yang, and Kan Ren. 2024. Benchmarking data science agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5677--5700

    2024

  40. [48]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...

  41. [49]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.