Pith. sign in

REVIEW 2 major objections 7 minor 109 references

DSAgentBench: Can Agents Automate End-to-End Data-Science Workflows in Real Computer Environments?

T0 review · 2 major / 7 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read DSAgentBench tests agents on 275 full data-science workflows inside a real OS; the strongest agent succeeds on 56.70% of tasks and all open-source agents stay below 1%.

desk verdict Solid benchmark contribution with a real capability gap, but the 'deterministic evaluator' claim is overstated and the open-source comparison is unevenly controlled. read the letter →

arxiv 2608.10366 v1 pith:6FH6U7EN submitted 2026-08-11 cs.AI cs.CL

Mizanur Rahman , Mohammed Saidul Islam , Ridwan Mahbub , Md Tahmid Rahman Laskar , Shafiq Joty , Enamul Hoque Prince This is my paper · ORCID
classification cs.AIcs.CL
keywords datascienceagentsbenchmarkcomputeruseGUIgroundinglong-horizonworkflowsvision-languagemodelsdeterministicevaluationagent
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces DSAgentBench, a benchmark that claims to be the first to test whether AI agents can run complete data-science workflows—loading data, exploring it, building models, and producing visualizations—inside a real operating system rather than in a sandboxed code environment. It contains 275 human-authored tasks spanning the data-science life cycle, each graded by a deterministic evaluator that checks the final artifacts (numerical results, files, plots, model performance) rather than stopping at successful code execution. The paper evaluates 15 closed- and open-source agents and reports that the strongest agent, Claude-4.6-Sonnet, succeeds on 56.70% of tasks, while all open-source agents remain below 1%. The intended consequence is that prior code-only benchmarks overstate agent readiness, and that real progress on autonomous data science must be measured inside full computer environments.

What carries the argument

The load-bearing object is the benchmark environment itself: a real Ubuntu desktop in which each of 275 tasks is defined by a configuration (files, datasets, installed tools), a natural-language instruction, and a deterministic Python evaluator that grades the final artifacts. Agents perceive the desktop either as a screenshot alone or as a screenshot augmented with an accessibility tree (structured UI metadata describing element roles, names, and positions), and act through a unified mouse-and-keyboard action space. A task succeeds only if the evaluator verifies the analytical output—numerical values within tolerance, correct output files, labeled plots, model performance thresholds—rather than merely that code executed. The paper's ablations, including step budgets of 15, 30, and 50 and a terminal-first prompting variant, are designed to show that failures come from reasoning and grounding limits rather than from the interaction budget.

What would settle it

Re-run the 275 tasks with open-source agents given the same accessibility-tree observations or a native terminal/API action space while keeping the deterministic evaluators unchanged; if any open-source agent's success rate rises above 1%, the paper's blanket claim about open-source agents would fail under that condition.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that end-to-end data-science automation on a real desktop remains largely unsolved. Claude-4.6-Sonnet, the strongest of 15 evaluated agents, completes 56.70% of the 275 tasks when given a screenshot plus an accessibility tree, against 85.09% for human participants; the next-best agent reaches 29.81%. Every open-source agent scores below 1% in the screenshot-only setting, and the paper's failure analysis attributes 97–98% of their errors to grounding—failing to map screen content to correct clicks and typed commands—rather than to data-science reasoning. The authors present this gap as evidence that current agentic systems lack the combination of UI grounding, tool orchestration, and long-horizon reasoning that real data-science work demands.

Load-bearing premise

The headline comparison assumes that a single screenshot-driven point-and-click interface is a fair and representative way to deploy every agent; in the main results the strongest closed-source agents also receive an accessibility tree while open-source agents do not, so the reported gap may partly reflect the interface rather than pure capability.

Editorial extensions

If this is right

  • If the benchmark is accepted, current agents are not yet reliable enough to automate real data-science work end to end; human oversight remains necessary.
  • The near-zero open-source results imply that open models, at least under this interface, are not yet competitive options for desktop computer-use data science.
  • Raising the interaction budget from 15 to 50 steps improves GPT-4o's success only marginally (24.54% to 25.81%), so progress depends on better grounding and reasoning, not more chances to act.
  • The evaluator design—deterministic artifact checks with an LLM judge applied only after those gates pass—offers a model for judging agent work by outcomes rather than by code execution.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Our inference: the headline 'open-source below 1%' is conditional on the screenshot-only setting; if open-source agents were given the same accessibility-tree input or a native terminal API, the gap might narrow, since the paper's own tables show A11y gives modest gains to some closed-source models.
  • Our inference: without controlling the observation modality across model families, the benchmark measures the interaction interface as much as the agents; a fairer comparison would hold the interface constant for every model.
  • Our inference: the task distribution, weighted toward exploratory analysis and hard multi-stage workflows, will reward agents that recover from early errors, so trajectory-level metrics (first-failure step, recovery rate) may become as informative as final success rate.
  • Our inference: the same environment-plus-deterministic-evaluator template could extend to other artifact-producing fields, such as software engineering or automated science, where success should be judged by verifiable outputs rather than by code execution.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 7 minor

Summary. DSAgentBench introduces a benchmark of 275 human-authored data-science tasks in a real Ubuntu desktop environment, extending OSWorld with data-science tools and external data sources. Agents perceive screenshots or screenshots plus accessibility trees and act through a unified GUI action space; task success is determined by per-task evaluation functions. The paper evaluates 15 closed-source, hybrid, and open-source agents and reports that Claude-4.6-Sonnet reaches 56.70% success under the Screenshot + A11y setting, while all open-source agents remain below 1%. The benchmark, evaluators, and prompts are planned for public release.

Significance. If the results hold, DSAgentBench addresses a genuine gap: most data-science benchmarks do not require real operating-system interaction, and most GUI benchmarks do not require end-to-end analytical workflows. The human-annotation pipeline with dual verification, the fixed environment configuration, and the planned release of tasks and evaluators are strengths. However, the central claim of a 'deterministic evaluator' is not fully supported because 33 of 275 tasks use an LLM visual judge, and the open-source comparison is confounded by different observation and action settings. These issues directly affect the two headline claims of the paper, so the benchmark's value as a reproducible measurement instrument is currently in question.

major comments (2)
  1. [§5.2, Table 16, Appendix B.6] The claim that DSAgentBench uses a 'deterministic evaluator' (abstract; §1 contribution iii; §3.1; §3.2.2) is contradicted by the evaluation design described in §5.2 and Table 16: for 33 of 275 tasks (the visualization/reporting category), scores are partially produced by GPT-4o or Gemini-2.5-Pro as an LLM judge. Because task success is defined as a score ≥0.95, variation in the judge's assessment of 'visual quality and semantic alignment' can change whether a task counts as successful. The deterministic gates reduce but do not eliminate this issue, since the LLM judgement is part of scoring for those tasks. The paper reports no stability analysis for judge model choice, prompt wording, temperature, or API version, and the Limitations section does not acknowledge this non-determinism. This is load-bearing because the abstract and contribution (iii) present the deterministic evaluator as the benchmark's core reproducibility guarantee. I request either a quantitative stability analysis (e.g., multiple judge models and temperatures, with the resulting success-rate interval) or a restatement of the evaluator as 'hybrid deterministic-plus-LLM' with the corresponding uncertainty reported on all headline numbers.
  2. [§5.3, Table 13, Limitations] The headline comparison 'all open-source agents remain below 1%' is not apples-to-apples. Open-source models are evaluated only in the screenshot-only setting and do not receive the A11y tree, whereas the headline closed-source result (Claude-4.6-Sonnet, 56.70%) uses Screenshot + A11y. Table 3 shows that A11y helps closed-source models (e.g., Claude-4.6-Sonnet rises from 50.55% to 56.70%; GPT-4o from 19.34% to 24.54%), so the open-source models are denied a modality that improves the closed-source numbers. In addition, open-source models are given different action spaces (e.g., UI-TARS native actions vs. the pyautogui code interface used by the closed-source models) and different prompting, and they are self-hosted with vLLM (App. B.1, B.4, B.5). The Limitations section acknowledges the A11y asymmetry but not the action-space and prompting confounds. The claim that open-source agents are 'below 1%' should be restricted to the exact configuration tested, and a matched screenshot-only comparison with open-source-native action spaces should be reported before drawing the general capability-gap conclusion.
minor comments (7)
  1. [Limitations] The final sentence of the Limitations section ('The evaluators verify required outputs, data mappings, labels, and task-specific correctness, overall visual clarity and semantic alignment.') is grammatically incomplete; it appears to intend a contrast such as 'but not overall visual clarity and semantic alignment,' which is exactly the limitation that should be stated explicitly.
  2. [Appendix B.5 and Table 5] The acronym DSworld appears in Appendix B.5 ('DSworld framework') and in Table 5 ('DS-World task category hierarchy') but is not defined or cited in the references; either provide a citation or explain the relationship between DSworld and DSAgentBench.
  3. [Tables 3, 9-12] Model naming is inconsistent between Table 3 (e.g., GPT-4o, Claude-4.6-Sonnet, Gemini-2.5-Pro) and Tables 9-12 (e.g., GPT-4.0, Claude-Sonnet-4.6, Gemini-2.5-Pro), which makes cross-referencing results unnecessarily difficult.
  4. [Table 15] Table 15 labels average scores as percentages, but Section 5.2 defines evaluator scores in [0,1]; the table should use one consistent convention.
  5. [Figures and captions] Figure 2 caption says 'deterministic evaluation scripts,' which conflicts with the LLM-judge component described in Section 5.2; align the terminology throughout.
  6. [References] The reference list contains two entries with the same title 'Scaling computer-use grounding via user interface decomposition and synthesis' (Xie et al., 2025a and 2025b), which appears to be a duplicate; if they are distinct works, the entries need distinguishing details.
  7. [§5.3] The human baseline of 85.09% is reported as a point estimate from three participants; since it is used as a reference for the deterministic evaluation protocol, the number of tasks attempted by each participant and the inter-participant variance should be reported.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark's headline results are empirical measurements produced by a fixed evaluation protocol, not derived from fitted inputs, self-citation chains, or definitionally tied outputs.

full rationale

Walking the paper's claimed derivation chain, I find no step in which a prediction or first-principles result reduces to its own inputs by construction. The central claims are empirical: task success rates for 15 agents are produced by running those agents in an OSWorld-based environment and scoring final artifacts with task-specific evaluators; no parameter is fitted to the reported outcomes and then renamed as a prediction. The benchmark construction is human-authored with LLM assistance only for wording and edge-case identification, and the evaluators were defined and validated by humans, so the tasks are not definitionally constructed from the evaluated models' outputs. The visual-judge component for roughly 10% of tasks uses GPT-4o or Gemini-2.5-Pro to assess visualization quality, with a cross-judge design to avoid self-evaluation; this is a reproducibility and determinism concern rather than a circularity concern, because the judge is not fitted to the headline success rates and the deterministic gates precede it. The paper's self-citations (e.g., the authors' survey, Text2Vis, DashboardQA) appear in related-work and motivation contexts and are not load-bearing for the benchmark's validity. The open-source/screenshot-only versus closed-source/A11y comparison is a fairness or representativeness issue, not a definitional reduction, especially since Table 3 also reports screenshot-only results for closed-source models. No uniqueness theorem, ansatz, or self-citation is invoked to forbid alternatives or force the reported gap. Accordingly, under the specific-reduction standard required here, the honest finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The central claim rests on benchmark design choices (success threshold, step budget, tolerance) and domain assumptions about representativeness and evaluator validity. No new physical entities are introduced. The undefined 'DSworld' framework in B.5 is an unflagged assumption.

free parameters (4)
  • task_success_threshold = 0.95
    Defines a task as successful when the evaluator score is at least 0.95. This hand-chosen threshold directly determines all reported success rates; changing it would change the headline numbers.
  • max_interaction_steps = 15
    The default maximum number of agent actions per task. The ablation in Table 11 shows only marginal gains when increased to 50, but the low ceiling may compress open-source performance.
  • numerical_tolerance_epsilon = 0.01
    Tolerance for matching numeric outputs in evaluators. Affects pass or fail for correlation-style tasks.
  • visual_judge_threshold_gating
    An LLM judge is invoked only after deterministic validation gates pass, and only for about 10% of tasks. The choice of judge model and rubric introduces non-determinism into the 'deterministic evaluator' claim.
assumptions (6)
  • domain assumption The OSWorld environment provides a realistic proxy for real computer environments.
    The paper extends OSWorld (Xie et al., 2024) and treats Ubuntu desktop interaction as representative of how data science is performed. If the environment is unrealistic, the benchmark's external validity is reduced.
  • domain assumption The 275 human-authored tasks are representative of real-world data-science workflows.
    Task taxonomy was derived from 100 Kaggle notebooks and expert annotators, but this is a sampling assumption not validated against an external benchmark.
  • domain assumption Deterministic evaluators plus LLM visual judges correctly capture analytical correctness and visualization quality.
    Numerical evaluators check values within tolerance; visual judges use LLMs. The validity of using an LLM as a quality judge is assumed, not proven.
  • domain assumption The human reference performance (85.09%) from three participants is a reliable baseline.
    Only three participants, no confidence interval; small sample may not represent the population of data scientists.
  • domain assumption The 15-step action budget is sufficient for measuring end-to-end workflows.
    The paper abates 15/30/50 steps and finds marginal gains, but this is tested on GPT-4o only; the budget may still bias against exploration-heavy agents.
  • ad hoc to paper The DSworld framework referenced in B.5 exists and is appropriate.
    Section B.5 mentions 'running the DSworld framework' without defining it or citing it. This is an undefined dependency in the experimental setup.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DSAgentBench: Can Agents Automate End-to-End Data-Science Workflows in Real Computer Environments?." pith.science (2026). https://pith.science/paper/6FH6U7EN

@misc{pith2026260810366,
  author       = {Pith},
  title        = {Pith review of: DSAgentBench: Can Agents Automate End-to-End Data-Science Workflows in Real Computer Environments?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6FH6U7EN}},
  note         = {Machine review of arXiv:2608.10366}
}
read the original abstract

Real-world data science involves long-horizon workflows that span data wrangling, exploration, modeling, visualization, and validation, and require coordinated use of tools such as notebooks, IDEs, terminals, browsers, and databases within real operating environments. Yet existing benchmarks lack real-computer interaction and do not evaluate whether agents can execute complete end-to-end data-science workflows in realistic computing environments, failing to capture the multi-stage, multi-tool nature of data-science practice. We introduce DSAgentBench, the first benchmark to evaluate whether agents can automate full data-science workflows inside real computer environments. DSAgentBench contains 275 diverse tasks covering the entire data-science life-cycle, reflecting the complexity and tool coordination required in practice. Each task requires grounding decisions in intermediate outputs and coordinated tool use, and includes a deterministic evaluator that verifies analytical correctness, visual outputs, and model performance rather than code-only execution. Our extensive experiments with 15 closed- and open-source models show that even the strongest agent, Claude-4.6-Sonnet, achieves only 56.70% task success, while all open-source agents remain below 1%, frequently failing at tool orchestration, OS grounding, and multi-step reasoning. These results reveal a substantial capability gap between current agentic systems and real data-science workflows, positioning DSAgentBench as a foundation for developing grounded, verifiable, autonomous data-science agents. We release DSAgentBench at https://github.com/vis-nlp/DSAgentBench.

Figures

Figures reproduced from arXiv: 2608.10366 by the authors.

Figure 1
Figure 1. Example agent workflow task in DSAGENTBENCH, showing selected stages from a multi-step workflow: the agent retrieves data, executes code, and produces the final outputs. The illustrated trajectory is generated by GPT-4o. climate modeling (Sarker, 2021; Adeniran et al., 2024; Donoho, 2017). Achieving such outcomes requires a broad skill set that includes program￾ming, statistical reasoning, visualization literacy, do… view at source ↗
Figure 2
Figure 2. Benchmark Construction Pipeline: We source heterogeneous real-world datasets (1); design tasks and [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Task category distribution in our benchmark, [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: DSAGENTBENCH execution workflow: the agent perceives screenshots and the A11y tree, executes GUI-based actions, and is evaluated by a custom script. The environment runs Ubuntu OS with Python and common data-science libraries pre-installed, and captures screenshots at …
Figure 5
Figure 5. Figure 5: Example environments in DSAGENTBENCH: agent interaction across data-science environments, demonstrating autonomous coordination of file systems, databases, notebooks, IDEs, visualization, execution, debugging, and web-based data retrieval. install the Jupyter Notebook …
Figure 6
Figure 6. Figure 6: Example of error that occured with the model OpenAI CUA. [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]
Figure 7
Figure 7. Figure 7: Example of error that occured with the model Jedi-3B. [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: Example of error that occured with the model Jedi-8B. [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: Example of error that occured with the model UI-Tars-1.5-7B. [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: The system prompts used for the code-generation agent. The [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]
Figure 11
Figure 11. Figure 11: The system prompts used for the UITARS agent. The top box shows the standard prompting strategy, [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]
Figure 12
Figure 12. Figure 12: Example system prompt for the JEDI architecture. The Grounder prompt provides the precise tool [PITH_FULL_IMAGE:figures/full_fig_p022_12.png]
Figure 13
Figure 13. Figure 13: Example system prompt for the JEDI architecture. The Planner prompt enforces a strict structure for [PITH_FULL_IMAGE:figures/full_fig_p023_13.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

109 extracted references · 27 canonical work pages

  1. [1]

    2016 , publisher=

    Learning LaTeX , author=. 2016 , publisher=

    2016

  2. [2]

    Proceedings International Conference on Coordinated and Multiple Views in Exploratory Visualization-CMV 2003- , pages=

    Exploring context switching and cognition in dual-view coordinated visualizations , author=. Proceedings International Conference on Coordinated and Multiple Views in Exploratory Visualization-CMV 2003- , pages=. 2003 , organization=

    2003

  3. [3]

    Proceedings of the ACM on Human-Computer Interaction , volume=

    How do data science workers collaborate? roles, workflows, and tools , author=. Proceedings of the ACM on Human-Computer Interaction , volume=. 2020 , publisher=

    2020

  4. [4]

    Hugginggpt: Solving

    Shen, Yongliang and Song, Kaitao and Tan, Xu and Li, Dongsheng and Lu, Weiming and Zhuang, Yueting , journal=. Hugginggpt: Solving

  5. [5]

    arXiv preprint arXiv:2410.20424 , year=

    Autokaggle: A multi-agent framework for autonomous data science competitions , author=. arXiv preprint arXiv:2410.20424 , year=

    arXiv

  6. [7]

    Proceedings of the 41st International Conference on Machine Learning , articleno =

    Guo, Siyuan and Deng, Cheng and Wen, Ying and Chen, Hechang and Chang, Yi and Wang, Jun , title =. Proceedings of the 41st International Conference on Machine Learning , articleno =. 2024 , publisher =

    2024

  7. [8]

    arXiv preprint arXiv:2311.17541 , year=

    TaskWeaver: A Code-First Agent Framework , author=. arXiv preprint arXiv:2311.17541 , year=

    arXiv

  8. [9]

    Health Management & Information Science , volume=

    Business Intelligence Dashboard in Healthcare: Lesson Learned , author=. Health Management & Information Science , volume=. 2022 , publisher=

    2022

Show all 109 references
  1. [10]

    Journal of information systems research and innovation , volume=

    A Review on the Dashboard Characteristics for manufacturing Organizations , author=. Journal of information systems research and innovation , volume=

  2. [11]

    Reviews of Contemporary Business Analytics , volume=

    Drivers and barriers of adopting interactive dashboard reporting in the finance sector: an empirical investigation , author=. Reviews of Contemporary Business Analytics , volume=

  3. [12]

    2019 , school =

    Dashboards and Information Overload: Challenges and Design Guidelines for Health Information Management Systems in Developing Countries , author =. 2019 , school =

    2019

  4. [13]

    arXiv preprint arXiv:1710.07300 , year=

    Figureqa: An annotated figure dataset for visual reasoning , author=. arXiv preprint arXiv:1710.07300 , year=

    arXiv

  5. [14]

    Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages=

    Plotqa: Reasoning over scientific plots , author=. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages=

  6. [15]

    Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages=

    Leaf-qa: Locate, encode & attend for figure question answering , author=. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages=

  7. [16]

    Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages=

    STL-CQA: Structure-based transformers with localization and encoding for chart question answering , author=. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages=

    2020

  8. [17]

    Proceedings of the IEEE conference on computer vision and pattern recognition , pages=

    Dvqa: Understanding data visualizations via question answering , author=. Proceedings of the IEEE conference on computer vision and pattern recognition , pages=

  9. [18]

    Tableau Public , year =

  10. [19]

    Microsoft Power BI , year =

  11. [20]

    arXiv preprint arXiv:2203.10244 , year=

    Chartqa: A benchmark for question answering about charts with visual and logical reasoning , author=. arXiv preprint arXiv:2203.10244 , year=

    arXiv

  12. [21]

    2025 , eprint=

    ChartQAPro: A More Diverse and Challenging Benchmark for Chart Question Answering , author=. 2025 , eprint=

    2025

  13. [22]

    2016 , eprint=

    SQuAD: 100,000+ Questions for Machine Comprehension of Text , author=. 2016 , eprint=

    2016

  14. [23]

    2025 , eprint=

    ChartCitor: Multi-Agent Framework for Fine-Grained Chart Visual Attribution , author=. 2025 , eprint=

    2025

  15. [24]

    2022 , eprint=

    Chart Question Answering: State of the Art and Future Directions , author=. 2022 , eprint=

    2022

  16. [25]

    Advances in Neural Information Processing Systems , volume=

    Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments , author=. Advances in Neural Information Processing Systems , volume=

  17. [26]

    2015 , publisher=

    Decision Support Dashboards: Their impact on the ICT sector and how they can improve decision making , author=. 2015 , publisher=

    2015

  18. [27]

    2021 25th International Conference Information Visualisation (IV) , pages=

    A brief review of dashboard visualizations employed to support management or business decisions , author=. 2021 25th International Conference Information Visualisation (IV) , pages=. 2021 , organization=

    2021

  19. [28]

    2025 , eprint=

    ChartLens: Fine-grained Visual Attribution in Charts , author=. 2025 , eprint=

    2025

  20. [29]

    2023 , eprint=

    Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V , author=. 2023 , eprint=

    2023

  21. [30]

    2024 , eprint=

    AutoScraper: A Progressive Understanding Web Agent for Web Scraper Generation , author=. 2024 , eprint=

    2024

  22. [31]

    2023 , eprint=

    Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer , author=. 2023 , eprint=

    2023

  23. [32]

    2025 , eprint=

    MultiChartQA: Benchmarking Vision-Language Models on Multi-Chart Problems , author=. 2025 , eprint=

    2025

  24. [33]

    2020 , eprint=

    PlotQA: Reasoning over Scientific Plots , author=. 2020 , eprint=

    2020

  25. [34]

    2024 , eprint=

    CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs , author=. 2024 , eprint=

    2024

  26. [35]

    2024 , eprint=

    OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments , author=. 2024 , eprint=

    2024

  27. [36]

    2024 , eprint=

    WebArena: A Realistic Web Environment for Building Autonomous Agents , author=. 2024 , eprint=

    2024

  28. [37]

    2024 , eprint=

    OfficeBench: Benchmarking Language Agents across Multiple Applications for Office Automation , author=. 2024 , eprint=

    2024

  29. [38]

    2025 , eprint=

    ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use , author=. 2025 , eprint=

    2025

  30. [39]

    2024 , eprint=

    SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents , author=. 2024 , eprint=

    2024

  31. [40]

    2025 , eprint=

    Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis , author=. 2025 , eprint=

    2025

  32. [41]

    2025 , eprint=

    Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agents , author=. 2025 , eprint=

    2025

  33. [42]

    2025 , eprint=

    UI-TARS: Pioneering Automated GUI Interaction with Native Agents , author=. 2025 , eprint=

    2025

  34. [43]

    2024 , eprint=

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context , author=. 2024 , eprint=

    2024

  35. [44]

    2024 , eprint=

    GPT-4 Technical Report , author=. 2024 , eprint=

    2024

  36. [45]

    Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles , year=

    Efficient Memory Management for Large Language Model Serving with PagedAttention , author=. Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles , year=

  37. [46]

    2025 , eprint=

    GTA1: GUI Test-time Scaling Agent , author=. 2025 , eprint=

    2025

  38. [47]

    2025 , eprint=

    Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities , author=. 2025 , eprint=

    2025

  39. [48]

    2019 , eprint=

    Scene Text Visual Question Answering , author=. 2019 , eprint=

    2019

  40. [49]

    and Kumar, Pratyush , title =

    Methani, Nitesh and Ganguly, Pritha and Khapra, Mitesh M. and Kumar, Pratyush , title =. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) , month =

  41. [50]

    Price and Christopher Kanan , title =

    Kushal Kafle and Scott Cohen and Brian L. Price and Christopher Kanan , title =. CoRR , volume =. 2018 , url =

    2018

  42. [51]

    2018 , eprint=

    FigureQA: An Annotated Figure Dataset for Visual Reasoning , author=. 2018 , eprint=

    2018

  43. [52]

    2021 , eprint=

    InfographicVQA , author=. 2021 , eprint=

    2021

  44. [53]

    Applying Pragmatics Principles for Interaction with Visual Analytics , year=

    Hoque, Enamul and Setlur, Vidya and Tory, Melanie and Dykeman, Isaac , journal=. Applying Pragmatics Principles for Interaction with Visual Analytics , year=

  45. [54]

    Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems , pages =

    Kim, Dae Hyun and Hoque, Enamul and Agrawala, Maneesh , title =. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems , pages =. 2020 , isbn =. doi:10.1145/3313831.3376467 , abstract =

    2020

  46. [55]

    C hart A ssistant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuning

    Meng, Fanqing and Shao, Wenqi and Lu, Quanfeng and Gao, Peng and Zhang, Kaipeng and Qiao, Yu and Luo, Ping. C hart A ssistant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuning. Findings of the Association for Computat...

    2024 doi

  47. [56]

    2023 , eprint=

    ChartLlama: A Multimodal LLM for Chart Understanding and Generation , author=. 2023 , eprint=

    2023

  48. [57]

    2024 , eprint=

    ChartInstruct: Instruction Tuning for Chart Comprehension and Reasoning , author=. 2024 , eprint=

    2024

  49. [58]

    2024 , eprint=

    ChartGemma: Visual Instruction-tuning for Chart Reasoning in the Wild , author=. 2024 , eprint=

    2024

  50. [59]

    2023 , eprint=

    UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning , author=. 2023 , eprint=

    2023

  51. [60]

    arXiv preprint arXiv:2409.12191 , year=

    Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution , author=. arXiv preprint arXiv:2409.12191 , year=

    arXiv

  52. [61]

    2025 , eprint=

    Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling , author=. 2025 , eprint=

    2025

  53. [62]

    Discover Tableau , year =

  54. [63]

    2025 , url =

    Introducing GPT-5 , author =. 2025 , url =

    2025

  55. [64]

    2025 , eprint=

    Mobile-Agent-v3: Fundamental Agents for GUI Automation , author=. 2025 , eprint=

    2025

  56. [65]

    arXiv preprint arXiv:2107.03374 , year=

    Evaluating large language models trained on code , author=. arXiv preprint arXiv:2107.03374 , year=

    arXiv

  57. [66]

    arXiv preprint arXiv:2410.07331 , year=

    Da-code: Agent data science code generation benchmark for large language models , author=. arXiv preprint arXiv:2410.07331 , year=

    arXiv

  58. [67]

    arXiv preprint arXiv:2506.23719 , year=

    DABstep: Data Agent Benchmark for Multi-step Reasoning , author=. arXiv preprint arXiv:2506.23719 , year=

    arXiv

  59. [68]

    arXiv preprint arXiv:2506.06541 , year=

    Kramabench: A benchmark for ai systems on data-to-insight pipelines over data lakes , author=. arXiv preprint arXiv:2506.06541 , year=

  60. [69]

    International Conference on Machine Learning , pages=

    DS-1000: A natural and reliable benchmark for data science code generation , author=. International Conference on Machine Learning , pages=. 2023 , organization=

    2023

  61. [70]

    arXiv preprint arXiv:2310.03302 , year=

    Mlagentbench: Evaluating language agents on machine learning experimentation , author=. arXiv preprint arXiv:2310.03302 , year=

    arXiv

  62. [71]

    arXiv preprint arXiv:2409.07703 , year=

    DSBench: How Far Are Data Science Agents from Becoming Data Science Experts? , author=. arXiv preprint arXiv:2409.07703 , year=

    arXiv

  63. [72]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Benchmarking data science agents , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  64. [73]

    Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Natural language to code generation in interactive data science notebooks , author=. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  65. [74]

    arXiv preprint arXiv:2510.04023 , year=

    Llm-based data science agents: A survey of capabilities, challenges, and future directions , author=. arXiv preprint arXiv:2510.04023 , year=

  66. [75]

    arXiv preprint arXiv:2406.00515 , year=

    A survey on large language models for code generation , author=. arXiv preprint arXiv:2406.00515 , year=

    arXiv

  67. [76]

    arXiv preprint arXiv:2411.04890 , year=

    Gui agents with foundation models: A comprehensive survey , author=. arXiv preprint arXiv:2411.04890 , year=

    arXiv

  68. [77]

    Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    Text2vis: A challenging and diverse benchmark for generating multimodal visualizations from text , author=. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    2025

  69. [78]

    Advances in Neural Information Processing Systems , volume=

    Webshop: Towards scalable real-world web interaction with grounded language agents , author=. Advances in Neural Information Processing Systems , volume=

  70. [79]

    Advances in Neural Information Processing Systems , volume=

    Mind2web: Towards a generalist agent for the web , author=. Advances in Neural Information Processing Systems , volume=

  71. [80]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Visualwebarena: Evaluating multimodal agents on realistic visual web tasks , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  72. [81]

    arXiv preprint arXiv:2407.19056 , year=

    Officebench: Benchmarking language agents across multiple applications for office automation , author=. arXiv preprint arXiv:2407.19056 , year=

    arXiv

  73. [82]

    Proceedings of the 33rd ACM International Conference on Multimedia , pages=

    Screenspot-pro: Gui grounding for professional high-resolution computer use , author=. Proceedings of the 33rd ACM International Conference on Multimedia , pages=

  74. [83]

    arXiv preprint arXiv:2307.13854 , year=

    Webarena: A realistic web environment for building autonomous agents , author=. arXiv preprint arXiv:2307.13854 , year=

    arXiv

  75. [84]

    arXiv preprint arXiv:2405.14573 , year=

    Androidworld: A dynamic benchmarking environment for autonomous agents , author=. arXiv preprint arXiv:2405.14573 , year=

    arXiv

  76. [85]

    SN Computer Science , volume=

    Data science and analytics: an overview from data-driven smart computing, decision-making and applications perspective , author=. SN Computer Science , volume=. 2021 , publisher=

    2021

  77. [86]

    Computer Science & IT Research Journal , volume=

    The role of data science in transforming business operations: Case studies from enterprises , author=. Computer Science & IT Research Journal , volume=

  78. [87]

    ACM Computing Surveys (CSUR) , volume=

    Data science: a comprehensive overview , author=. ACM Computing Surveys (CSUR) , volume=. 2017 , publisher=

    2017

  79. [88]

    2022 , publisher=

    Data Visualization with Python and JavaScript: Scrape, Clean, Explore, and Transform Your Data , author=. 2022 , publisher=

    2022

  80. [89]

    IEEE transactions on visualization and computer graphics , volume=

    Towards natural language interfaces for data visualization: A survey , author=. IEEE transactions on visualization and computer graphics , volume=. 2022 , publisher=

    2022

  81. [90]

    arXiv preprint arXiv:2411.07763 , year=

    Spider 2.0: Evaluating language models on real-world enterprise text-to-sql workflows , author=. arXiv preprint arXiv:2411.07763 , year=

    arXiv

  82. [91]

    IEEE Transactions on Visualization and Computer Graphics , year=

    Viseval: A benchmark for data visualization in the era of large language models , author=. IEEE Transactions on Visualization and Computer Graphics , year=

  83. [92]

    Advances in Neural Information Processing Systems , volume=

    Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation , author=. Advances in Neural Information Processing Systems , volume=

  84. [93]

    Data interpreter: An

    Hong, Sirui and Lin, Yizhang and Liu, Bang and Liu, Bangbang and Wu, Binhao and Zhang, Ceyao and Li, Danyang and Chen, Jiaqi and Zhang, Jiayi and Wang, Jinlin and others , booktitle=. Data interpreter: An

  85. [94]

    arXiv preprint arXiv:2504.13865 , year=

    A survey on (m) llm-based gui agents , author=. arXiv preprint arXiv:2504.13865 , year=

    arXiv

  86. [95]

    2023 , publisher=

    R for data science: import, tidy, transform, visualize, and model data , author=. 2023 , publisher=

    2023

  87. [96]

    Journal of Computational and Graphical Statistics , volume=

    50 years of data science , author=. Journal of Computational and Graphical Statistics , volume=. 2017 , publisher=

    2017

  88. [97]

    arXiv preprint arXiv:2508.17398 , year=

    DashboardQA: Benchmarking Multimodal Agents for Question Answering on Interactive Dashboards , author=. arXiv preprint arXiv:2508.17398 , year=

    arXiv

  89. [98]

    arXiv preprint arXiv:2511.07332 , year=

    Grounding Computer Use Agents on Human Demonstrations , author=. arXiv preprint arXiv:2511.07332 , year=

    arXiv

  90. [99]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Cogagent: A visual language model for gui agents , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  91. [100]

    Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

    Showui: One vision-language-action model for gui visual agent , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

  92. [101]

    European Conference on Computer Vision , pages=

    Ferret-ui: Grounded mobile ui understanding with multimodal llms , author=. European Conference on Computer Vision , pages=. 2024 , organization=

    2024

  93. [102]

    arXiv preprint arXiv:2410.23218 , year=

    Os-atlas: A foundation action model for generalist gui agents , author=. arXiv preprint arXiv:2410.23218 , year=

    arXiv

  94. [103]

    arXiv preprint arXiv:2505.13227 , year=

    Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis , author=. arXiv preprint arXiv:2505.13227 , year=

  95. [104]

    arXiv preprint arXiv:2504.10458 , year=

    Gui-r1: A generalist r1-style vision-language action model for gui agents , author=. arXiv preprint arXiv:2504.10458 , year=

    arXiv

  96. [105]

    arXiv preprint arXiv:2507.15846 , year=

    GUI-G2: Gaussian Reward Modeling for GUI Grounding , author=. arXiv preprint arXiv:2507.15846 , year=

    arXiv

  97. [106]

    arXiv preprint arXiv:2508.05731 , year=

    Infigui-g1: Advancing gui grounding with adaptive exploration policy optimization , author=. arXiv preprint arXiv:2508.05731 , year=

  98. [107]

    2024 , url=

    Computer-Using Agent , author=. 2024 , url=

    2024

  99. [108]

    2025 , institution=

    2025 State of Data Science: Moving From Hype Toward Maturity , author=. 2025 , institution=

    2025

  100. [109]

    arXiv preprint arXiv:2508.09123 , year=

    Opencua: Open foundations for computer-use agents , author=. arXiv preprint arXiv:2508.09123 , year=

  101. [110]

    Advances in Neural Information Processing Systems , volume=

    Spider2-v: How far are multimodal agents from automating data science and engineering workflows? , author=. Advances in Neural Information Processing Systems , volume=

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.