{"id":"e14d7ddf-83a1-4075-83a7-750848858484","arxiv_id":"2608.03451","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A new heterogeneous-workspace benchmark for data agents, built from transformed Text-to-SQL tasks, with complete-table deterministic evaluation and a best measured accuracy of 66.34%.","lead":"DataSpace is a 410-task benchmark that drops AI data agents into messy workspaces where the evidence needed to answer a question is scattered across databases, spreadsheets, PDFs, and videos. Its deterministic scorer checks the final table, and the best current model still solves only about two thirds of tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reference-answer correctness is the load-bearing assumption: without inter-reviewer reliability data or release of the 350 withheld golds, the 66.34% ceiling and the 76-task hard core cannot be independently verified.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the gold answers are the foundation of every reported number, and the paper does not provide enough external evidence to certify them. The construction pipeline does include real safeguards: deterministic execution checks, round-trip validation in Equation 16, blind dual review, an append-only audit trail, and a deterministic evaluator. These make the gold chain more credible than a purely LLM-generated benchmark, and I do not see an internal inconsistency in the published protocol. However, the residual gold-error rate is unquantified, and withholding 350 of 410 references prevents independent re-derivation. A secondary concern is that the 66.34% figure comes from single runs with no confidence intervals, so the 1.71-point gap between Grok 4.5 and GPT-5.6 Sol could be within run-to-run variance; that affects the precision of the empirical claims but not the benchmark's construct validity. Because the central risk is the same one the reader identified, the conditional verdict stands unchanged.","tokens_in":33949,"tokens_out":8398,"duration_ms":87902,"concrete_test":"Take a stratified random sample of 40 tasks from the 350 withheld-reference set. Have two independent expert teams, neither involved in construction, re-derive the gold answer from the public question and workspace using only the published Section 4.5 protocol and no access to the stored reference. Compare each team's answer to the stored gold with the official evaluator and compute inter-team agreement. Require at least 38/40 exact evaluator matches and perfect team agreement on all 40; any mismatch should trigger an audit of the construction ledger and a re-baseline of the six backbones on corrected golds. A cheaper first step is for third parties to verify the 60 released gold tables by deterministic re-execution from the public workspaces.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Finding 1's validity depends on every reference answer being exactly right. The construction chain in Sections 4.1–4.4 has four links: (i) source EHRSQL/BULL gold SQL is assumed correct; (ii) Cross-Language Transformation is validated by execution equivalence plus an LLM judge (Fig. 12) whose agreement rate is not reported; (iii) LLM-rendered documents and videos pass automated cell checks but can still drift semantically, e.g., a predicate in a video abstraction could flip >= to >, which would change which rows belong in the gold; (iv) two expert reviewers must agree on gold and configuration, but no inter-reviewer reliability statistic, repair count, or removal count is given. Since only 60 of 410 gold tables are released, the reported 66.34% accuracy and the set of 76 tasks solved by none of the six backbones cannot be checked by third parties. If even a few reference answers are wrong, tasks move in and out of the hard-core set, the headline accuracy is biased, and the unsaturation conclusion is weakened. This is not a peripheral data-quality nit: the benchmark's central quantity, Task Accuracy, is defined by exact equivalence to these golds.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"DataSpace is a new benchmark for data agents that must answer a natural-language question by discovering and combining evidence across task-local heterogeneous workspaces (CSV, JSON, SQLite, Markdown, PDF, and video) and returning a complete tabular result. The paper contributes DataSpace-Builder, a four-stage pipeline that transforms EHRSQL and BULL text-to-SQL instances into 410 cross-language multimodal tasks with reference answers, a deterministic, header-invariant tabular evaluator, and an empirical study of six frontier multimodal backbones and five agent harnesses. The headline findings are that the benchmark remains unsaturated (best controlled accuracy 66.34%, with 76 tasks missed by all backbones), that harness choice causes a 15.36-point spread, and that multimodal integration and joins are the most consistent sources of degradation.","tokens_in":34223,"tokens_out":5140,"duration_ms":51810,"significance":"If the benchmark and its reference answers are sound, DataSpace fills a real gap: it combines heterogeneous evidence discovery, complete tabular output contracts, and deterministic evaluation in a way that most prior benchmarks do not. The construction pipeline is unusually careful: execution-grounded validation, round-trip artifact checks, blind two-reviewer task review, and a deterministic evaluator with explicit type, precision, and order semantics are all credible design choices. The public release of the evaluator, code, and 60 fully gold-annotated tasks is a useful contribution, and the trace-level failure analysis is more informative than typical aggregate error rates. However, the central validity claim rests on the correctness of the reference answers, and that premise is currently not fully auditable: the source SQL is assumed correct, the LLM judge agreement rate is unreported, the human review process is described without inter-reviewer reliability statistics, and 350 of 410 gold tables are withheld. In addition, all headline numbers come from single runs of stochastic API models with no confidence intervals.","major_comments":[{"comment":"The correctness of every reference answer inherits from the source EHRSQL/BULL gold SQL and from the cross-language transformation, but the validation chain is not auditable from the paper. The execution-equivalence check verifies only that the translated SQL and translated database agree; the only question-level semantic check is the LLM judge in Figure 12, whose agreement, repair, and reject rates are not reported. Section 4.4 and Appendix A.5 describe two-reviewer gold verification, yet no inter-reviewer reliability statistics, repair counts, removal counts, or configuration-disagreement rates are given. Because Task Accuracy in Eq. (9) is exact equivalence to these golds, even a small number of wrong reference answers would bias the 66.34% ceiling and change the membership of the 76-task hard core. Please report the LLM-judge validation statistics and the human-review agreement/repair/removal ledger, or provide an independent audit of a random sample of golds.","section":"§4.4, Appendix B"},{"comment":"Only 60 of 410 gold tables are released, and the selection criteria for these 60 'representative' tasks are not described. A third party cannot therefore check the withheld 350 golds, including the 76-task hard core that Finding 1 depends on, nor can they reproduce the official 66.34% accuracy from the released artifacts. If the 60 tasks are not a random or stratified sample, local end-to-end evaluation on them gives no valid estimate of full-benchmark behavior. Please release all gold tables and configurations, or justify the withholding with a concrete protocol (e.g., a held-out test set with a fixed public leaderboard) and show that the 60 released tasks are representative along the dimensions used in Section 5.","section":"Appendix B"},{"comment":"All headline results are from a single run per backbone–harness pair over stochastic API models. The top two backbones differ by only seven tasks (66.34% vs. 64.63%), and the efficiency comparisons in §6.3 (e.g., 74.2% fewer tokens, 50.3% fewer actions) are computed from the same single runs. Without repeated runs, seeds, temperature-0 determinism checks, or bootstrap confidence intervals, the model ranking and the quantitative efficiency gaps are not statistically grounded. Please report variance across at least 3–5 repeated runs for the main comparisons, or demonstrate that the endpoints are deterministic enough that single runs are adequate.","section":"§6.1"}],"minor_comments":[{"comment":"The illustration contains typos: 'Catetory' should be 'Category' and 'Algin Entities' should be 'Align Entities'.","section":"Figure 1"},{"comment":"The experimental setup says all backbone calls use the provider-default reasoning configuration, but does not state temperature, top-p, or any determinism setting. Please specify the sampling parameters or explicitly state that they are left at provider defaults and that this limits reproducibility.","section":"§A.7"},{"comment":"Please state how the 60 released tasks were selected (e.g., random stratified by domain and modality) and whether the withheld 350 are fixed and versioned, so that external evaluations can be compared across time.","section":"Appendix B"},{"comment":"For the video predicate-abstraction strategy, the paper says the question is rejected when a selected span cannot be removed without changing semantics, but no statistics are reported on how often this occurs or how many video-augmented tasks required repair. A sentence with these counts would help calibrate the reliability of the video evidence chain.","section":"§4.3"}],"recommendation":"major_revision","confidential_remarks":"The main risk is reference-answer auditability rather than novelty or scope. If the authors can release the full gold set (or a clearly representative sample with provenance) and add uncertainty quantification for the headline numbers, the paper will be a strong benchmark contribution. The single-run stochastic issue is particularly important given the small absolute differences between top backbones."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"DataSpace is a genuinely useful benchmark. The fixed complete-table contract, deterministic schema-invariant evaluator, and cross-language heterogeneous workspaces fill a real gap: most prior benchmarks isolate retrieval, SQL, or open-ended analysis, and this one holds the output contract fixed across the whole test set. The construction pipeline is the most careful I have seen in this area—every stage is execution-grounded, with round-trip artifact checks and dual expert review. The KDD Cup deployment gives it immediate reach.\n\nWhat is new is the package: the DataSpace-Builder framework turning Text-to-SQL corpora into multimodal tasks, the query-conditioned video rendering with predicate abstraction, and the evaluator's header-invariant column alignment plus order-aware row comparison. The empirical section is thoughtful: fixing the backbone while varying harnesses isolates harness effects, and the root-cause audit distinguishes evaluator symptoms from upstream errors. The error analysis is a real plus.\n\nThe soft spots are real but not fatal. First, gold correctness is load-bearing: source golds from EHRSQL and BULL are assumed correct; execution equivalence plus an LLM judge validates translation, but judge agreement is not reported; document and video artifact checks can miss semantic drift; and the two-reviewer consensus process reports no inter-reviewer reliability, repair counts, or removal counts. Only 60 of 410 gold tables are released, so the 66.34% ceiling and the 76-task hard core cannot be fully checked. Second, all headline numbers are single runs of stochastic API models with no confidence intervals; the 15.36-point harness spread could shift on repeat. Third, the source base is narrow—88.5% BULL and 11.5% EHRSQL—so domain coverage is really finance plus health.\n\nProportionately, the construction is credible, the paper owns its limitations, and the qualitative conclusion—unsaturation, with joins and multimodal integration hurting—is likely robust even if a few golds are wrong. The 56-asil74 structure (solved by all vs. solved by none, with 81.46% oracle union) is large enough that small gold noise will not flip it.\n\nThis is for researchers building data agents and evaluation benchmarks. It deserves a serious referee. I would recommend accept with revision: release at least a stratified sample of the withheld golds or inter-reviewer agreement statistics, report repeat-run variance for at least one backbone, and disclose the LLM-judge pass rate in cross-language transformation.","headline":"A solid, genuinely useful benchmark with a careful construction pipeline; the two things to press on are gold-answer verifiability and missing error bars.","tokens_in":34782,"tokens_out":2315,"would_cite":true,"duration_ms":22430,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DataSpace argues that data agents should be judged by whether they return the exact complete table requested from a heterogeneous workspace, and shows that the best controlled model reaches only 66.34 percent accuracy.","keywords":["data agents","benchmark","heterogeneous workspaces","tabular output","deterministic evaluation","text-to-SQL","multimodal evidence","cross-language analytics"],"falsifier":"Have an independent team reconstruct the reference answer for the 60 released tasks directly from each workspace without seeing the paper's gold tables, then run the official evaluator on both gold sets; if the two gold sets disagree on even a small share of tasks, the reported accuracy numbers are not a stable benchmark property. A second check is to give an agent an oracle that reveals exactly which artifacts are required: if accuracy rises far above 66.34%, the bottleneck is discovery and multimodal reading; if not, the residual failures sit in computation and output materialization.","tokens_in":33772,"feed_emoji":"📊","tokens_out":8417,"duration_ms":69716,"temperature":0.7,"pith_summary":"DataSpace is a benchmark that asks a data agent to take a single natural-language question and a task-local workspace, find the relevant evidence across CSV, JSON, SQLite, Markdown, PDF, and video, and return one complete typed table. The paper's central claim is that this setting can be evaluated deterministically, without an LLM judge, and that it is not yet saturated: the best controlled model reaches 66.34% Task Accuracy, 76 of 410 tasks are missed by all six tested backbones, and the choice of agent harness alone moves accuracy by 15.36 points. A sympathetic reader should care because the benchmark unifies three properties that earlier suites kept separate: workspace-scale discovery, a complete-table output contract, and reproducible evaluation. The paper also isolates where agents currently fail, pointing to output materialization and cross-modal integration rather than mere evidence retrieval.","feed_headline":"Best data agent solves only 66% of workspace tasks","feed_subtitle":"A 410-task benchmark makes agents hunt across CSVs, databases, PDFs, and video to return one exact table.","key_machinery":"The load-bearing mechanism is DataSpace-Builder, a four-stage construction pipeline that turns executable text-to-SQL instances into heterogeneous workspace tasks. It jointly translates question, database, and SQL across languages, samples a task-local relational instance while preserving primary and foreign keys and query conditions, routes tables to CSV, JSON, SQLite, Markdown, PDF, and query-conditioned video renderers, and then subjects every task to independent expert review and repair. The reference answer is produced by executing the transformed SQL against the sampled database, so the gold tables inherit an execution-grounded origin. The evaluation side uses a deterministic comparator that ignores prediction headers, aligns columns by trying all permutations, canonicalizes types, precision, and units, and compares rows either as ordered sequences or as multisets depending on the task configuration.","core_discovery":"On its own terms, the paper establishes that workspace-style data agents can be benchmarked by exact tabular output, and that current frontier models are far from solving such tasks. It reports that with a fixed agent harness, the best controlled backbone achieves 66.34% Task Accuracy, the second-best trails by 1.71 points while using 74.2% fewer tokens, and the strongest harness outperforms the weakest by 15.36 points with the same model. It further reports that multimodal tasks underperform single-modal tasks for every backbone by 1.8–14.0 points, that join-requiring tasks drop by 9.7–19.8 points, and that 76 tasks are solved by none of the six backbones while 56 are solved by all. Materialization errors dominate the strongest model's failures: 52.2% of 136 audited failures originate in output materialization, and target-result misunderstanding plus faulty column projection account for 56.6%. The paper's finding is that DataSpace remains unsaturated and that its deterministic evaluator provides a stable target for measuring future progress.","pith_inferences":["A testable extension is to release the remaining 350 reference tables; if independent re-annotation disagrees on a non-negligible share of tasks, the reported accuracy scale would need revisiting.","Because video tasks relocate a predicate or answer value into the video, progress in video-language models could saturate those tasks faster than the document tasks, shifting the difficulty mix unevenly over time.","The same construction pipeline could generate fresh tasks from new text-to-SQL sources on demand, allowing the benchmark to grow without the expert-review step becoming the limiting cost.","The 15.36-point harness spread suggests that stronger agent designs may lift scores substantially before raw model capability becomes the binding constraint, so the 66.34% figure is not a ceiling."],"forward_implications":["If DataSpace's output contract is the right one, partial or factoid answers must be scored wrong even when some returned values are correct.","The 15.36-point harness spread with a fixed backbone means benchmark reports must pin down the harness as carefully as the model, or leaderboard comparisons are not meaningful.","Because multimodal integration and joins degrade every tested backbone, these operations, not basic retrieval, should be the focus of the next round of data-agent training and tooling.","The 56.6% share of failures from output materialization and target-result misunderstanding implies that projects that improve an agent's ability to emit exactly the requested schema may yield larger gains than better source selection.","The 76 tasks missed by all six backbones and the 81.46% oracle-union ceiling define a hard core that future benchmark suites can use to measure saturation."],"supporting_citations":[{"why":"supplies the clinical text-to-SQL source instances whose questions, databases, and executable SQL are transformed into tasks.","marker":"[12]"},{"why":"supplies the financial text-to-SQL source instances; 88.5% of the benchmark tasks originate from this corpus.","marker":"[56]"},{"why":"supplies the tabular-data-to-video rendering technique that the query-conditioned video branch extends.","marker":"[50]"},{"why":"supplies the ReAct-style reasoning-and-acting loop that defines the controlled DataSpace-Agent baseline.","marker":"[54]"},{"why":"establishes the workspace-style enterprise evaluation setting that DataSpace extends toward multimodal artifacts.","marker":"[13]"},{"why":"is the closest prior data-agent benchmark over heterogeneous data and serves as the comparison point for the new task contract.","marker":"[48]"},{"why":"supplies the execution-based deterministic evaluation tradition that DataSpace's tabular-result evaluator adapts.","marker":"[55]"}],"fun_headline_variants":["Data agents flunk heterogeneous analytics: 66% top score","Workspace data agents top out at 66% exact-table accuracy","Benchmark: agents solve barely two-thirds of workspace tasks","DataSpace benchmark: best agent hits 66% only"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's gold answers are only as trustworthy as the source text-to-SQL instances they were built from, plus the translation, rendering, and expert-review steps applied to them, and since only 60 of 410 reference tables are publicly released that chain cannot be fully audited from outside.","fun_headline_variants_meta":{"raw":{"variants":["Data agents flunk heterogeneous analytics: 66% top score","Workspace data agents top out at 66% exact-table accuracy","Benchmark: agents solve barely two-thirds of workspace tasks","DataSpace benchmark: best agent hits 66% only"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000164,"raw_usage":{"total_tokens":1299,"prompt_tokens":1047,"completion_tokens":252,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":663,"completion_tokens_details":{"reasoning_tokens":182}},"tokens_in":663,"tokens_out":252,"duration_ms":2683,"temperature":1.0,"reasoning_tokens":182,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:49:31.669979+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have an independent team reconstruct the reference answer for the 60 released tasks directly from each workspace without seeing the paper's gold tables, then run the official evaluator on both gold sets; if the two gold sets disagree on even a small share of tasks, the reported accuracy numbers are not a stable benchmark property. A second check is to give an agent an oracle that reveals exactly which artifacts are required: if accuracy rises far above 66.34%, the bottleneck is discovery and multimodal reading; if not, the residual failures sit in computation and output materialization.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the financial text-to-SQL source instances; 88.5% of the benchmark tasks originate from this corpus."},{"cited_title":"DataMagic: Transforming Tabular Data into Data Insight Video","cited_arxiv_id":"2606.20388","evidence_quote":"supplies the tabular-data-to-video rendering technique that the query-conditioned video branch extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the execution-based deterministic evaluation tradition that DataSpace's tabular-result evaluator adapts."}],"review_version":1}