{"id":"399063d8-e623-43c4-a937-1af01f9bdba4","arxiv_id":"2608.10366","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"DSAgentBench is a new 275-task benchmark for end-to-end data science in real computer environments, where current AI agents, especially open-source ones, mostly fail.","lead":"This paper introduces DSAgentBench, a benchmark that tests AI agents on complete data-science workflows inside a real Ubuntu desktop environment. It finds that even the best agent, Claude-4.6-Sonnet, succeeds on only 56.7% of tasks, while open-source agents succeed on under 1%.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'deterministic evaluator' is not fully deterministic: ~10% of tasks use an LLM visual judge, so the headline success rates are not independently reproducible.","rationale":"I read the paper in good faith. The benchmark is large (275 tasks), human-authored, dual-annotated, and the environment extends OSWorld in a plausible direction. The human baseline and the step-budget ablation add credibility, and the paper honestly lists the open-source A11y limitation. The reader's weakest assumption, that the open-source comparison is not apples-to-apples because of the A11y tree, is real but partially mitigated by Table 3, which reports screenshot-only results for closed-source models; the large gap persists even without A11y. The more load-bearing concern is the internal contradiction in the evaluator: the abstract and contributions repeatedly call it 'deterministic,' yet Section 5.2 and Table 16 show that a nontrivial subset of tasks relies on an LLM visual judge. Because the headline success rate and the human baseline both depend on the 0.95 threshold, and because no judge-stability analysis is provided, the central numbers are not independently reproducible as claimed. This does not call for rejection, since the issue is addressable by replacing the LLM judge with a fixed deterministic rubric or by publishing a judge-stability study; it does reinforce the CONDITIONAL verdict already assigned. If the concrete test shows high judge agreement and negligible score shifts, the concern would be resolved.","tokens_in":24984,"tokens_out":12291,"duration_ms":112174,"concrete_test":"Recompute scores for all 33 visualization/reporting tasks using two independent LLM judges (e.g., GPT-4o and Claude-4.6-Sonnet) with the published rubric at temperature 0, and compute task-level agreement (Cohen's kappa) as well as the resulting overall success rates for Claude-4.6-Sonnet and GPT-4o. Also recompute overall success using only the deterministic gates and treating visualization tasks as failed, then report the change in headline accuracy. If kappa is below 0.8 or either model's overall success rate shifts by more than 2 percentage points, the 'deterministic evaluator' claim and the headline numbers are not stable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The benchmark's central value proposition is a 'deterministic evaluator' (abstract; Section 1 contribution iii; Section 3.2.2), but Section 5.2 and Table 16 reveal that about 10% of tasks (the visualization/reporting category, 33 of 275) use GPT-4o or Gemini-2.5-Pro as a visual LLM judge after deterministic gates. This is an internal inconsistency: the reported headline of Claude-4.6-Sonnet at 56.70% task success, and the human baseline of 85.09%, are not produced by a fully deterministic scoring function. Success is defined by crossing a 0.95 threshold, which can hinge on the judge's subjective assessment of 'visual quality and semantic alignment.' No stability analysis is reported for judge model choice, prompt wording, temperature, or API version, and the paper's Limitations section does not acknowledge this non-determinism. A related and self-acknowledged confound is that open-source agents are evaluated only in screenshot-only mode while the headline closed-source result uses the A11y tree; however, Table 3 also reports screenshot-only results for closed-source models, and the qualitative gap persists there, so the comparison issue is less damaging than the evaluator non-determinism. The most load-bearing concern is therefore that the benchmark's core measurement instrument is not reproducible as claimed.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"DSAgentBench introduces a benchmark of 275 human-authored data-science tasks in a real Ubuntu desktop environment, extending OSWorld with data-science tools and external data sources. Agents perceive screenshots or screenshots plus accessibility trees and act through a unified GUI action space; task success is determined by per-task evaluation functions. The paper evaluates 15 closed-source, hybrid, and open-source agents and reports that Claude-4.6-Sonnet reaches 56.70% success under the Screenshot + A11y setting, while all open-source agents remain below 1%. The benchmark, evaluators, and prompts are planned for public release.","tokens_in":25267,"tokens_out":5545,"duration_ms":48781,"significance":"If the results hold, DSAgentBench addresses a genuine gap: most data-science benchmarks do not require real operating-system interaction, and most GUI benchmarks do not require end-to-end analytical workflows. The human-annotation pipeline with dual verification, the fixed environment configuration, and the planned release of tasks and evaluators are strengths. However, the central claim of a 'deterministic evaluator' is not fully supported because 33 of 275 tasks use an LLM visual judge, and the open-source comparison is confounded by different observation and action settings. These issues directly affect the two headline claims of the paper, so the benchmark's value as a reproducible measurement instrument is currently in question.","major_comments":[{"comment":"The claim that DSAgentBench uses a 'deterministic evaluator' (abstract; §1 contribution iii; §3.1; §3.2.2) is contradicted by the evaluation design described in §5.2 and Table 16: for 33 of 275 tasks (the visualization/reporting category), scores are partially produced by GPT-4o or Gemini-2.5-Pro as an LLM judge. Because task success is defined as a score ≥0.95, variation in the judge's assessment of 'visual quality and semantic alignment' can change whether a task counts as successful. The deterministic gates reduce but do not eliminate this issue, since the LLM judgement is part of scoring for those tasks. The paper reports no stability analysis for judge model choice, prompt wording, temperature, or API version, and the Limitations section does not acknowledge this non-determinism. This is load-bearing because the abstract and contribution (iii) present the deterministic evaluator as the benchmark's core reproducibility guarantee. I request either a quantitative stability analysis (e.g., multiple judge models and temperatures, with the resulting success-rate interval) or a restatement of the evaluator as 'hybrid deterministic-plus-LLM' with the corresponding uncertainty reported on all headline numbers.","section":"§5.2, Table 16, Appendix B.6"},{"comment":"The headline comparison 'all open-source agents remain below 1%' is not apples-to-apples. Open-source models are evaluated only in the screenshot-only setting and do not receive the A11y tree, whereas the headline closed-source result (Claude-4.6-Sonnet, 56.70%) uses Screenshot + A11y. Table 3 shows that A11y helps closed-source models (e.g., Claude-4.6-Sonnet rises from 50.55% to 56.70%; GPT-4o from 19.34% to 24.54%), so the open-source models are denied a modality that improves the closed-source numbers. In addition, open-source models are given different action spaces (e.g., UI-TARS native actions vs. the pyautogui code interface used by the closed-source models) and different prompting, and they are self-hosted with vLLM (App. B.1, B.4, B.5). The Limitations section acknowledges the A11y asymmetry but not the action-space and prompting confounds. The claim that open-source agents are 'below 1%' should be restricted to the exact configuration tested, and a matched screenshot-only comparison with open-source-native action spaces should be reported before drawing the general capability-gap conclusion.","section":"§5.3, Table 13, Limitations"}],"minor_comments":[{"comment":"The final sentence of the Limitations section ('The evaluators verify required outputs, data mappings, labels, and task-specific correctness, overall visual clarity and semantic alignment.') is grammatically incomplete; it appears to intend a contrast such as 'but not overall visual clarity and semantic alignment,' which is exactly the limitation that should be stated explicitly.","section":"Limitations"},{"comment":"The acronym DSworld appears in Appendix B.5 ('DSworld framework') and in Table 5 ('DS-World task category hierarchy') but is not defined or cited in the references; either provide a citation or explain the relationship between DSworld and DSAgentBench.","section":"Appendix B.5 and Table 5"},{"comment":"Model naming is inconsistent between Table 3 (e.g., GPT-4o, Claude-4.6-Sonnet, Gemini-2.5-Pro) and Tables 9-12 (e.g., GPT-4.0, Claude-Sonnet-4.6, Gemini-2.5-Pro), which makes cross-referencing results unnecessarily difficult.","section":"Tables 3, 9-12"},{"comment":"Table 15 labels average scores as percentages, but Section 5.2 defines evaluator scores in [0,1]; the table should use one consistent convention.","section":"Table 15"},{"comment":"Figure 2 caption says 'deterministic evaluation scripts,' which conflicts with the LLM-judge component described in Section 5.2; align the terminology throughout.","section":"Figures and captions"},{"comment":"The reference list contains two entries with the same title 'Scaling computer-use grounding via user interface decomposition and synthesis' (Xie et al., 2025a and 2025b), which appears to be a duplicate; if they are distinct works, the entries need distinguishing details.","section":"References"},{"comment":"The human baseline of 85.09% is reported as a point estimate from three participants; since it is used as a reference for the deterministic evaluation protocol, the number of tasks attempted by each participant and the inter-participant variance should be reported.","section":"§5.3"}],"recommendation":"major_revision","confidential_remarks":"The paper has a solid infrastructure contribution, but the two headline claims need to be either substantiated or qualified. The deterministic-evaluator issue is especially important: if the LLM-judge component cannot be shown to be stable, the authors should change the claims rather than the experiments. I do not see grounds for rejection because the benchmark itself, with these caveats fixed, would be useful to the community."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper introduces DSAgentBench, 275 tasks for end-to-end data science in real Linux environments, extending OSWorld with data-science tools. The core finding — that even Claude-4.6-Sonnet only reaches 56.7% while open-source agents stay below 1% — is plausible and important. The benchmark fills a genuine gap between OSWorld and code-only data-science benchmarks.\n\nWhat it does well: the task construction is careful. Dual-annotator verification, a human baseline (85.09%), and detailed task statistics show real effort. The evaluation is extensive: 15 models, two observation settings, ablations on step budget and terminal-first prompting, and error analysis with first-failure steps. Using different LLM judges to avoid self-evaluation is a thoughtful touch.\n\nThe main soft spot is the 'deterministic evaluator' claim. Section 5.2 admits that ~10% of tasks (33 visualization/reporting) use an LLM visual judge after deterministic gates. That is not deterministic, and the paper does not report stability across judge models, prompts, or temperatures. The human baseline and the 56.70% headline both depend on this subjective component. The Limitations section does not acknowledge it, which is a mismatch with the paper's central value proposition.\n\nThe open-source comparison is uneven: open-source models only get screenshot-only input while closed-source models get the A11y tree in the main results. But Table 3 also reports closed-source screenshot-only numbers, and Claude-4.6 still reaches 50.55% there versus 0.73% for the best open-source model. So the gap is real, though the 'below 1%' headline undersells the confound. The open-source models also use different prompts and are smaller, which is fair but should be stated more carefully.\n\nMinor points: Appendix A.1 says the taxonomy is based on DS-World, but DS-World is not in the references. The human baseline is only three people. No code or data is released yet, which limits independent verification.\n\nOverall, this is a worthwhile benchmark and a serious evaluation. The deterministic-evaluator claim needs revision, and the comparison should be reframed, but these are addressable. I would send it to a serious referee.","headline":"Solid benchmark contribution with a real capability gap, but the 'deterministic evaluator' claim is overstated and the open-source comparison is unevenly controlled.","tokens_in":25788,"tokens_out":2643,"would_cite":true,"duration_ms":25139,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DSAgentBench tests agents on 275 full data-science workflows inside a real OS; the strongest agent succeeds on 56.70% of tasks and all open-source agents stay below 1%.","keywords":["data science agents","benchmark","computer use","GUI grounding","long-horizon workflows","vision-language models","deterministic evaluation","agent evaluation"],"falsifier":"Re-run the 275 tasks with open-source agents given the same accessibility-tree observations or a native terminal/API action space while keeping the deterministic evaluators unchanged; if any open-source agent's success rate rises above 1%, the paper's blanket claim about open-source agents would fail under that condition.","tokens_in":24796,"feed_emoji":"📊","tokens_out":9657,"duration_ms":74710,"temperature":0.7,"pith_summary":"This paper introduces DSAgentBench, a benchmark that claims to be the first to test whether AI agents can run complete data-science workflows—loading data, exploring it, building models, and producing visualizations—inside a real operating system rather than in a sandboxed code environment. It contains 275 human-authored tasks spanning the data-science life cycle, each graded by a deterministic evaluator that checks the final artifacts (numerical results, files, plots, model performance) rather than stopping at successful code execution. The paper evaluates 15 closed- and open-source agents and reports that the strongest agent, Claude-4.6-Sonnet, succeeds on 56.70% of tasks, while all open-source agents remain below 1%. The intended consequence is that prior code-only benchmarks overstate agent readiness, and that real progress on autonomous data science must be measured inside full computer environments.","feed_headline":"Best AI agent finishes 57% of real data-science tasks","feed_subtitle":"New benchmark runs agents inside a real OS on 275 workflow tasks; open-source models all finish below 1%.","key_machinery":"The load-bearing object is the benchmark environment itself: a real Ubuntu desktop in which each of 275 tasks is defined by a configuration (files, datasets, installed tools), a natural-language instruction, and a deterministic Python evaluator that grades the final artifacts. Agents perceive the desktop either as a screenshot alone or as a screenshot augmented with an accessibility tree (structured UI metadata describing element roles, names, and positions), and act through a unified mouse-and-keyboard action space. A task succeeds only if the evaluator verifies the analytical output—numerical values within tolerance, correct output files, labeled plots, model performance thresholds—rather than merely that code executed. The paper's ablations, including step budgets of 15, 30, and 50 and a terminal-first prompting variant, are designed to show that failures come from reasoning and grounding limits rather than from the interaction budget.","core_discovery":"On the paper's own terms, the central discovery is that end-to-end data-science automation on a real desktop remains largely unsolved. Claude-4.6-Sonnet, the strongest of 15 evaluated agents, completes 56.70% of the 275 tasks when given a screenshot plus an accessibility tree, against 85.09% for human participants; the next-best agent reaches 29.81%. Every open-source agent scores below 1% in the screenshot-only setting, and the paper's failure analysis attributes 97–98% of their errors to grounding—failing to map screen content to correct clicks and typed commands—rather than to data-science reasoning. The authors present this gap as evidence that current agentic systems lack the combination of UI grounding, tool orchestration, and long-horizon reasoning that real data-science work demands.","pith_inferences":["Our inference: the headline 'open-source below 1%' is conditional on the screenshot-only setting; if open-source agents were given the same accessibility-tree input or a native terminal API, the gap might narrow, since the paper's own tables show A11y gives modest gains to some closed-source models.","Our inference: without controlling the observation modality across model families, the benchmark measures the interaction interface as much as the agents; a fairer comparison would hold the interface constant for every model.","Our inference: the task distribution, weighted toward exploratory analysis and hard multi-stage workflows, will reward agents that recover from early errors, so trajectory-level metrics (first-failure step, recovery rate) may become as informative as final success rate.","Our inference: the same environment-plus-deterministic-evaluator template could extend to other artifact-producing fields, such as software engineering or automated science, where success should be judged by verifiable outputs rather than by code execution."],"forward_implications":["If the benchmark is accepted, current agents are not yet reliable enough to automate real data-science work end to end; human oversight remains necessary.","The near-zero open-source results imply that open models, at least under this interface, are not yet competitive options for desktop computer-use data science.","Raising the interaction budget from 15 to 50 steps improves GPT-4o's success only marginally (24.54% to 25.81%), so progress depends on better grounding and reasoning, not more chances to act.","The evaluator design—deterministic artifact checks with an LLM judge applied only after those gates pass—offers a model for judging agent work by outcomes rather than by code execution."],"supporting_citations":[{"why":"Supplies the real-computer environment framework that DSAgentBench extends with data-science tools, task configurations, and evaluators.","marker":"Xie et al., 2024"},{"why":"Empirical study of data-science workers' tools and workflows, used to ground the benchmark's choice of applications and task categories.","marker":"Zhang et al., 2020"},{"why":"DA-CODE is the closest prior data-science agent benchmark, limited to a sandboxed notebook; used to contrast with OS-level interaction.","marker":"Huang et al., 2024"},{"why":"DS-1000 represents the static code-generation benchmark lineage that DSAgentBench argues does not capture real tool interaction.","marker":"Lai et al., 2023"},{"why":"DSBench is a contemporary data-science agent benchmark used as a comparison point for full-workflow coverage.","marker":"Jing et al., 2024"},{"why":"DABStep provides a multi-step data-agent benchmark against which DSAgentBench positions its harder, OS-grounded setting.","marker":"Egg et al., 2025"}],"fun_headline_variants":["Best AI agent scores 57% on real data-science tasks","Open-source agents crash below 1% in real data-science","Grounding errors doom open-source data-science agents","Top agent hits 57% on real OS data-science workflow","AI data-science agents fail on real desktops: best scores 57%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline comparison assumes that a single screenshot-driven point-and-click interface is a fair and representative way to deploy every agent; in the main results the strongest closed-source agents also receive an accessibility tree while open-source agents do not, so the reported gap may partly reflect the interface rather than pure capability.","fun_headline_variants_meta":{"raw":{"variants":["Best AI agent scores 57% on real data-science tasks","Open-source agents crash below 1% in real data-science","Grounding errors doom open-source data-science agents","Top agent hits 57% on real OS data-science workflow","AI data-science agents fail on real desktops: best scores 57%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000627,"raw_usage":{"total_tokens":2929,"prompt_tokens":1006,"completion_tokens":1923,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":622,"completion_tokens_details":{"reasoning_tokens":1831}},"tokens_in":622,"tokens_out":1923,"duration_ms":12420,"temperature":1.0,"reasoning_tokens":1831,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:20:45.963055+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the 275 tasks with open-source agents given the same accessibility-tree observations or a native terminal/API action space while keeping the deterministic evaluators unchanged; if any open-source agent's success rate rises above 1%, the paper's blanket claim about open-source agents would fail under that condition.","supporting_citations":[],"review_version":1}