The reviewed record of science sign in
Pith

arxiv: 2607.06482 · v1 · pith:LRAN53FV · submitted 2026-07-07 · cs.CL · cs.AI

Data Analysis in the Wild: Benchmarking Large Language Models Against Real-World Data Complexities

So Hasegawa , Shailaja Keyur Sampat , Lei Liu , Wei-Peng Chen This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-07-08 04:13 UTCglm-5.2pith:LRAN53FVrecord.jsonopen to challenge →

classification cs.CL cs.AI
keywords datallmsanalysisbenchmarkdatagovbenchlargemodelsreal-world
0
0 comments X

The pith

Top LLMs score below 0.4 on real-world table QA benchmark

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper introduces DataGovBench, a benchmark built from 178 governmental open datasets designed to test LLMs against the messiness of real-world data analysis: large multi-tabular datasets (averaging 210K rows, up to 11.9M), metadata, external knowledge files, decomposable multi-part questions, and open-ended exploratory insight generation. The authors argue that existing benchmarks are too clean and simple, focusing on fact retrieval from small single tables, and thus fail to expose the gap between what LLMs can do in controlled settings and what real data analysis demands. The benchmark comprises two tasks: Table QA, which requires answering complex decomposable questions with text or visualizations, and Table Insight, which requires models to proactively generate expert-level findings through exploratory analysis without a specific query. The central mechanism carrying the argument is the combination of data complexity (scale, multiple tables, metadata, external knowledge) with task complexity (decomposable questions and open-ended insight generation). When state-of-the-art models including GPT-5.1, Claude Sonnet 4.6, and Gemini 2.5 Flash are evaluated, even with agentic frameworks providing code generation, self-correction, and reflection loops, the top Whole accuracy on Table QA remains below 0.4, and the top Summary-level score on Table Insight remains below 0.5. The paper identifies two core missing capabilities through qualitative analysis: narrative-level reasoning over tabular data (models can identify the general topic of an insight but fail to reproduce the central analytical argument or supporting entities) and accurate fact retrieval from complex tables (the most common error, at 32.4%, involves failing to apply implicit filtering conditions such as avoiding double-counting when a dataset contains both aggregated and disaggregated rows). The authors also find that their feature type-specific table serialization, which summarizes each column by its data type rather than dumping raw rows, outperforms providing more raw rows to the model, and that Python code generation substantially outperforms SQL for these tasks.

Core claim

The core discovery is that when LLMs are tested against data analysis tasks that mirror real-world complexity, their performance drops to levels far below what existing benchmarks suggest. The paper constructs this demonstration by building a benchmark from governmental open data with characteristics that existing benchmarks lack simultaneously: large tables, multiple tables per dataset, metadata, external knowledge, decomposable questions, and exploratory insight generation. The key empirical finding is that the best models achieve below 0.4 Whole accuracy on Table QA and below 0.5 Summary-level score on Table Insight even with agentic support, and that two specific capabilities are missing

What carries the argument

DataGovBench, a benchmark with two tasks (Table QA and Table Insight) built from 178 governmental open datasets averaging 210K rows, featuring multi-tabular datasets, metadata, external knowledge, decomposable questions, and ground-truth insights extracted from official expert reports. The evaluation uses an Answer Agent (with feature type-specific table serialization, coding with self-correction, and reflection modules) and an Insight Agent (with a DAG-based question-answer-insight exploration cycle), evaluated against ground truth via exact match for text answers, MLLM-as-judge for visualizations, and GPT-4o-based scoring for insights.

If this is right

  • The gap between benchmark performance and real-world readiness is large enough that deploying LLM-based data analysis systems on governmental or enterprise data without substantial human oversight carries significant risk of incorrect calculations and misleading conclusions.
  • The finding that feature type-specific table serialization outperforms raw row provision suggests that the field should invest in better compact table representations rather than simply scaling context windows.
  • The identification of narrative-level reasoning as a missing capability implies that current code-generation approaches can compute values but cannot reliably construct the analytical argument that makes those values meaningful, pointing to a need for systems that bridge numerical computation and narrative synthesis.
  • The poor performance of SQL relative to Python code generation on these tasks challenges the dominance of text-to-SQL as the paradigm for table question answering when tasks involve complex data transformations, multiple tables, and visualization outputs.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The benchmark's ground truth for Table QA is generated by LLMs and human-verified, and the ground truth for Table Insight is partly extracted using LLMs, which means the evaluation instrument itself is subject to the same class of errors the benchmark is designed to expose. The absolute scores may be noisy, though the relative ranking of models is likely more robust.
  • With only 6 datasets for Table Insight and 211 QA pairs for Table QA, the benchmark is small relative to existing alternatives. The performance gaps may partly reflect the difficulty of specific datasets rather than a systematic deficiency, and scaling the benchmark would strengthen the claims.
  • The moderate correlation between human judgment and metric scores for Table Insight (Pearson 0.669, Spearman 0.663) means that the Insight-level scores should be interpreted as approximate rankings rather than precise measurements of insight quality.

Load-bearing premise

The ground-truth answers and insights used to score models are themselves partly generated by LLMs and then human-verified, with no reported inter-annotator agreement and with LLMs used to extract insights from free-text reports, meaning the evaluation standard may contain the same kinds of errors the benchmark is designed to expose in the models it tests.

What would settle it

If a model or agent system were to achieve Whole accuracy above 0.7 on Table QA and Summary-level score above 0.7 on Table Insight on this benchmark, the central claim that current systems are far from satisfying real-world data analysis demands would be weakened, suggesting instead that the gap was an artifact of benchmark immaturity rather than a genuine capability deficit.

Figures

Figures reproduced from arXiv: 2607.06482 by Lei Liu, Shailaja Keyur Sampat, So Hasegawa, Wei-Peng Chen.

Figure 1
Figure 1. Figure 1: DataGovBench evaluates LLMs and agents on two table reasoning tasks [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: 1) curating datasets from open data portals through a systematic filter [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the three-stage construction process of DataGovBench [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Distributions of (a) number of tables per dataset, (b) format of external [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Architecture of (a) Answer Agent and (b) Insight Agent [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Error distribution with Answer Agent Errors also arise from the inher￾ent complexity of the tasks. Con￾text Handling Errors (8.7%) oc￾cur in decomposable questions where the model incorrectly uses the output from a flawed previous turn though the generated logic is correct in most cases. Visualization Errors (6.8%) typically involve incorrect axis ranges, such as a timeline that does not match the period s… view at source ↗
Figure 6
Figure 6. Figure 6: Insight perspective distributions by Insight Agent and AgentPoirot. [PITH_FULL_IMAGE:figures/full_fig_p012_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Annotation GUI for revising questions and answers [PITH_FULL_IMAGE:figures/full_fig_p024_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Annotation GUI for checking the revised QA pairs [PITH_FULL_IMAGE:figures/full_fig_p024_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Insight generaion process by (a) tree-based process used in AgentPoirot, [PITH_FULL_IMAGE:figures/full_fig_p028_9.png] view at source ↗
read the original abstract

Current benchmarks for evaluating Large Language Models (LLMs) in data analysis often fail to reflect real-world settings. They typically focus on fact retrieval from small tables and overlook the challenges of large multi-tabular datasets, external knowledge integration, and exploratory insight discovery. We introduce DataGovBench, a benchmark derived from governmental open data designed to evaluate LLMs in practical scenarios. The benchmark includes two tasks: Table QA that requires solving complex decomposable questions and producing textual answers or visualizations, and Table Insight that evaluates the ability of models to generate expert-level findings through exploratory data analysis. Comprehensive experiments with state-of-the-art LLMs, both with and without agentic frameworks, reveal significant performance gaps across both tasks. These results suggest that current LLM-based systems remain far from satisfying the demands of real-world data analytics. DataGovBench provides a challenging benchmark for advancing research on LLMs capable of both answering analytical queries and discovering insights from data. Code and sample data are available at https://github.com/SoHasegawa/datagovbench.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 8 minor

Summary. The paper introduces DataGovBench, a benchmark for evaluating LLMs on real-world data analysis tasks, comprising two sub-tasks: Table QA (211 question sets over 178 datasets, with decomposable questions and visualization outputs) and Table Insight (exploratory insight generation over 6 datasets with expert-authored reports as ground truth). The benchmark features large multi-tabular datasets with metadata and external knowledge, addressing gaps in existing benchmarks. The authors evaluate several closed- and open-source LLMs with and without agentic frameworks (Answer Agent, Insight Agent, AgentPoirot), finding that even top models achieve low accuracy (below 0.4 Whole on Table QA, below 0.5 Summary-level on Table Insight). The paper also provides qualitative error analyses and ablation studies on the Answer Agent components.

Significance. The benchmark addresses a genuine gap: most existing table QA benchmarks use small, single-table datasets without metadata or external knowledge, and few benchmarks formalize insight generation as a primary task. The feature type-specific table serialization (§2.1, Appendix A.2) is a practical contribution for handling large tables within context limits. The human-in-the-loop QA annotation pipeline with two-stage verification is well-designed. The SQL-vs-Python comparison (Table 5) and the ablation study (Table 4) provide useful empirical signals. The qualitative error taxonomy (Figure 5) and the four-perspective insight analysis (Figure 6) are informative. Code and sample data are publicly available, which supports reproducibility.

major comments (3)
  1. §3.3, Table 3 (Table Insight columns): GPT-4o is used as the sole evaluator for the Table Insight task, yet GPT-4o is also one of the four evaluated closed-source models (scoring 0.292/0.345 with AgentPoirot and 0.319/0.399 with Insight Agent). This is a direct self-evaluation conflict: GPT-4o may systematically prefer its own output style over that of Claude Sonnet 4.6 or Gemini 2.5 Flash. The meta-evaluation in Appendix B.3 (Pearson 0.669, Spearman 0.663) checks whether the metric agrees with humans on relative quality of insight pairs in general, but it does not test for evaluator self-preference bias specifically — a distinct bias that would not necessarily appear in a general correlation check. Given that the Table Insight task uses only 6 datasets (Table 9), even a modest systematic bias could reorder model rankings. The paper should either (a) use a different evaluator model that,
  2. §3.4, Table 3 (Table Insight columns): The paper reports 5-run averaging for Table Insight but does not report per-dataset variance or per-dataset scores. With only 6 datasets, aggregate scores could mask high cross-dataset variability that would render the model rankings in Table 3 unstable. Reporting per-dataset scores (or at minimum standard deviations across datasets) is necessary for readers to assess whether the differences between, e.g., Claude Sonnet 4.6 (0.453 Summary) and Gemini 2.5 Flash (0.405 Summary) are meaningful rather than noise.
  3. §2.1 (Annotation of Insight): The ground-truth construction for Table Insight uses NotebookLM and Gemini 2.5 Flash to extract insights from free-text reports, introducing LLM-generated content into the ground truth that is then used to evaluate LLMs. While the paper notes manual verification of extracted insights, it does not report what fraction of extractions were modified versus accepted as-is, nor any inter-annotator agreement for this verification step. This matters because the absolute scores in Table 3 depend on ground-truth quality. The paper should report these statistics or at minimum acknowledge this as a limitation.
minor comments (8)
  1. §2.1 (Annotation of Question and Answer Pairs): The paper states annotators have 'expertise in data analysis' but does not report the number of annotators, their specific qualifications, or inter-annotator agreement for the QA verification stage. Adding these details would strengthen the credibility of the ground truth.
  2. Table 3: The open-source models are not evaluated on the Table Insight task. The paper briefly notes 'relatively low performance of open-source models on Table QA' as justification (§3.2), but including at least one open-source model on Table Insight would make the comparison more complete.
  3. §4.2, Figure 5: The error analysis is conducted only on Gemini 2.5 Flash with Answer Agent. It would be useful to note whether the error distribution is similar for other models, or at least to acknowledge this limitation.
  4. §4.3, Table 5: The SQL-vs-Python comparison excludes visualization-based QA pairs, reducing the evaluation to 145 text-based pairs. The paper should clarify what fraction of the full benchmark this represents and whether the text-only subset is representative.
  5. Table 11: The example comparison is informative but only shows two ground-truth insights. Including a brief note on how representative these examples are would help readers gauge generality.
  6. §5 (Related Works): 'extenral' should be 'external' (typo). Also, reference [1] title reads 'ntroducing claude sonnet 4.6' — should be 'Introducing'.
  7. §6 (Conclusion): 'difficulity' should be 'difficulty' (typo).
  8. The paper uses future-dated references (e.g., GPT-5.1 accessed March 2026, Claude Sonnet 4.6 accessed March 2026). If these are real, the dates are fine; if placeholder, they should be corrected.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the thorough and constructive review. The referee correctly identifies the benchmark's contributions and raises three substantive concerns about the Table Insight evaluation methodology. We agree that all three points are valid and warrant revision. Below we address each comment point by point.

read point-by-point responses

  1. Referee: §3.3, Table 3 (Table Insight columns): GPT-4o is used as the sole evaluator for the Table Insight task, yet GPT-4o is also one of the four evaluated closed-source models. This is a direct self-evaluation conflict. The meta-evaluation in Appendix B.3 checks general correlation with human judgment but does not test for evaluator self-preference bias specifically. The paper should either (a) use a different evaluator model, or (b) explicitly test for self-preference bias.

    Authors: The referee is correct that using GPT-4o as both an evaluated model and the sole evaluator for Table Insight creates a potential self-evaluation conflict, and that the general correlation meta-evaluation in Appendix B.3 would not necessarily detect a systematic self-preference bias. This is a valid concern. We will address it in the revision by re-running the Table Insight evaluation with a different evaluator model (Claude Sonnet 4.6, which is also one of the evaluated models, creating a symmetric cross-evaluation setup) and reporting both sets of scores. We will also add an explicit discussion of the self-preference bias risk as a limitation. If the rankings change between the two evaluator configurations, we will report this transparently. We note that the overall conclusion—that all models score below 0.5 on Summary-level and below 0.35 on Insight-level—is unlikely to be affected, but the relative rankings among the top models could shift, and the revised paper will make this clear. revision: yes

  2. Referee: §3.4, Table 3 (Table Insight columns): The paper reports 5-run averaging for Table Insight but does not report per-dataset variance or per-dataset scores. With only 6 datasets, aggregate scores could mask high cross-dataset variability that would render model rankings unstable. Reporting per-dataset scores or at minimum standard deviations across datasets is necessary.

    Authors: We agree. With only 6 datasets, aggregate scores can mask important variability, and readers need per-dataset breakdowns to assess whether differences between models (e.g., Claude Sonnet 4.6 at 0.453 vs. Gemini 2.5 Flash at 0.405 on Summary-level) are meaningful. In the revision, we will add a per-dataset results table showing both Insight-level and Summary-level scores for each of the 6 datasets, along with standard deviations across datasets. This will allow readers to judge the stability of the rankings directly. revision: yes

  3. Referee: §2.1 (Annotation of Insight): The ground-truth construction for Table Insight uses NotebookLM and Gemini 2.5 Flash to extract insights from free-text reports, introducing LLM-generated content into the ground truth used to evaluate LLMs. The paper does not report what fraction of extractions were modified versus accepted as-is, nor any inter-annotator agreement for this verification step.

    Authors: This is a fair point. The ground-truth construction pipeline for Table Insight does involve LLM-generated content (NotebookLM for insight extraction, Gemini 2.5 Flash for summary generation), and the current manuscript does not adequately report the verification statistics. In the revision, we will add: (1) the fraction of LLM-extracted insights that were modified versus accepted as-is during manual verification, and (2) inter-annotator agreement for the verification step. We will also add an explicit acknowledgment in the limitations section that LLM-generated content is part of the ground-truth pipeline, along with a discussion of the steps taken to mitigate any resulting bias (manual verification, synthesis into standardized declarative sentences). If the inter-annotator agreement data were not systematically recorded during annotation in a form that supports a clean IAA statistic, we will state this honestly and report what verification statistics we do have. revision: partial

Circularity Check

2 steps flagged

No significant circularity: GPT-4o as both evaluated model and Table Insight evaluator is a self-evaluation concern, but the central claim (LLMs perform poorly) is not forced by construction.

specific steps
  1. fitted input called prediction [§3.3 (Table Insight evaluation) and Table 3]
    "We employ GPT-4o as the evaluator by replacing LLaMA3-70b [7]. We verified that the evaluator aligns with human perception through a meta-evaluation, as shown in Appendix B.3."

    GPT-4o is both one of the four evaluated closed-source models in Table 3 (scoring 0.292/0.345 with AgentPoirot, 0.319/0.399 with Insight Agent) and the sole evaluator for Table Insight. This creates a potential self-evaluation bias where GPT-4o may prefer its own output style. However, this is not circularity in the strict sense: the central claim (LLMs perform poorly, with top scores below 0.5) is not forced by construction. GPT-4o scores are not the highest among evaluated models (Claude Sonnet 4.6 scores higher), so the evaluator is not systematically favoring itself in a way that would circularly guarantee the paper's conclusion. The meta-evaluation (Pearson 0.669, Spearman 0.663) provides independent, albeit moderate, support for the evaluator's validity. This is a methodological bias

  2. fitted input called prediction [§2.1 (Annotation of Insight) and §3.3 (Table Insight evaluation)]
    "If the report expresses findings in free text, we uploaded the report to NotebookLM [6] and prompted it to suggest ten insights... we then synthesized the extracted insights into a standardized set of declarative sentences, and converted the set of sentences into a summary via Gemini 2.5 Flash [4]."

    The ground-truth insights and summaries for Table Insight are partly constructed using LLMs (NotebookLM, Gemini 2.5 Flash), and then LLM outputs are evaluated against this LLM-influenced ground truth using GPT-4o as evaluator. This creates a mild LLM-in-the-loop dependency. However, the ground truth originates from human-authored official reports, and the LLM is used only for extraction/summarization, not for creating the findings themselves. The central claim (poor LLM performance) is not circularly forced: if the ground truth were biased toward LLM-generated patterns, we would expect higher scores, not lower ones. The low scores (below 0.5) argue against circular reinforcement.

full rationale

The paper has two mild LLM-in-the-loop concerns: (1) GPT-4o serves as both an evaluated model and the sole Table Insight evaluator, and (2) ground-truth insights are partly extracted using LLMs. Neither reduces to circularity by construction. The central claim—that current LLMs perform poorly on real-world data analysis—is not forced by either the ground-truth construction or the evaluator choice. If anything, LLM involvement in ground-truth construction would bias scores upward, not downward, making the low scores more credible rather than less. The meta-evaluation provides independent (if moderate) support for evaluator validity. No self-citation chain is load-bearing for the central claim. The paper is substantially self-contained against external benchmarks. Score 2 reflects the methodological self-evaluation concern without rising to genuine circularity.

Axiom & Free-Parameter Ledger

7 free parameters · 5 axioms · 3 invented entities

The benchmark introduces 7 free parameters set by the authors, 5 domain assumptions that are plausible but not fully validated, and 3 invented entities (serialization technique, Answer Agent, Insight Agent) that have some independent evidence from ablation studies. The main concern is that the ground-truth construction and evaluation both rely on LLMs, creating a mild circularity, and the Table Insight task uses only 6 datasets.

free parameters (7)
  • Minimum row threshold (5,000) = 5,000
    Chosen as the filtering criterion for dataset inclusion (Section 2.1, Appendix A.1). Not derived from data; set by the authors.
  • Top-10 questions per dataset = 10
    Number of questions selected after scoring stage (Section 2.1). Set by the authors.
  • Number of LLM judges (4) = 4
    GPT-4o, GPT-4o-mini, Gemini 2.0 Flash, Gemini 1.5 Pro used as judges (Section 2.1). Set by the authors.
  • Max revision attempts (3) = 3
    Self-correction and reflection loops allow up to 3 revision attempts (Section 3.1). Set by the authors.
  • Insight Agent: initial questions (3), iterations (4), total insights (12) = 3, 4, 12
    Insight Agent configuration parameters (Appendix B.1). Set by the authors to match AgentPoirot's 12-insight output.
  • Categorical column unique value threshold (20) = 20
    Serialization lists all categories if ≤20 unique values, otherwise samples 20 (Appendix A.2). Set by the authors.
  • Random sample size for other feature types (10) = 10
    Serialization provides 10 random unique values for non-numerical, non-categorical, non-datetime columns (Appendix A.2). Set by the authors.
axioms (5)
  • domain assumption Government open data is representative of real-world data analysis scenarios
    Section 1 and Section 2.1 justify the benchmark's realism by citing the complexity of government data. This is plausible but unproven — government data may over-represent certain domains (demographics, environment) and under-represent others (finance, healthcare operations).
  • domain assumption LLM-generated questions, after human verification, are valid test items for evaluating LLM data analysis capability
    Section 2.1 describes the four-stage pipeline where LLMs generate questions that are then human-verified. The paper assumes this process yields valid, unbiased test items, but does not report inter-annotator agreement or compare against fully human-authored questions.
  • domain assumption Official reports accompanying datasets serve as valid ground truth for insight generation
    Section 2.1, 'Annotation of Insight': 'we adopted official reports that accompany the open datasets' as ground truth. This assumes the reports capture the full set of important insights, which may not hold — reports may omit findings that are derivable from the data.
  • domain assumption GPT-4o as evaluator produces scores that align with human judgment
    Section 3.3 uses GPT-4o as evaluator for Table Insight, replacing LLaMA3-70b from InsightBench. The meta-evaluation (Appendix B.3) reports Pearson 0.669 and Spearman 0.663, which the paper calls 'solid alignment.' This is moderate, not strong, correlation.
  • domain assumption Majority consensus of 4 MLLMs is a reliable judge for visualization correctness
    Section 3.3: 'a prediction is deemed correct upon a majority consensus, requiring positive assessments from at least three of the four models.' No meta-evaluation is reported for this protocol.
invented entities (3)
  • Feature type-specific table serialization independent evidence
    purpose: Compact textual representation of large tables for LLM context windows
    Ablation study (Table 4) shows serialization improves performance over baseline (+0.05 for Gemini, +0.015 for GPT-4o). The technique is described in detail in Appendix A.2 with concrete examples.
  • Answer Agent independent evidence
    purpose: Agentic framework for Table QA with coding, self-correction, and reflection modules
    Ablation study (Table 4) shows each module contributes to performance. The agent is compared against standalone LLMs and other agents.
  • Insight Agent (DAG-based) independent evidence
    purpose: Agentic framework for Table Insight using directed acyclic graph exploration
    Compared against AgentPoirot (tree-based) in Table 3. The DAG structure is described in Appendix B.2 and Figure 9. However, only 6 datasets are used for evaluation.

pith-pipeline@v1.1.0-glm · 23550 in / 4009 out tokens · 491996 ms · 2026-07-08T04:13:38.057958+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

36 extracted references · 36 canonical work pages · 10 internal anchors

  1. [1]

    Blog Post (February 2026),https:// www.anthropic.com/news/claude-sonnet-4-6, accessed: March 5, 2026

    Anthropic: ntroducing claude sonnet 4.6. Blog Post (February 2026),https:// www.anthropic.com/news/claude-sonnet-4-6, accessed: March 5, 2026

    work page 2026

  2. [2]

    Chen, L., Zhao, X., Zeng, Z., Huang, J., Zhong, Y., Ma, L.: Chart-r1: Chain- of-thought supervision and reinforcement for advanced chart reasoner (2025), arXiv:2507.15509

    work page arXiv 2025

  3. [3]

    In: International Conference on Learning Repre- sentations (2021)

    Chen, W., Chang, M.W., Schlinger, E., Wang, W.Y., Cohen, W.W.: Open question answering over tables and text. In: International Conference on Learning Repre- sentations (2021)

    work page 2021

  4. [4]

    Comanici, G., et al: Gemini 2.5: Pushing the frontier with advanced reason- ing, multimodality, long context, and next generation agentic capabilities (2025), arXiv:2507.06261

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  5. [5]

    Blog Post (December 2024),https://blog.google/technology/google-deepmind/ google-gemini-ai-update-december-2024/, accessed: May 30, 2025

    DeepMind, G.: Introducing gemini 2.0: our new ai model for the agentic era. Blog Post (December 2024),https://blog.google/technology/google-deepmind/ google-gemini-ai-update-december-2024/, accessed: May 30, 2025

    work page 2024

  6. [6]

    Google: Notebooklm (accessed July 10, 2025),https://notebooklm.google.com/

    work page 2025

  7. [7]

    Grattafiori, A., et al: The llama 3 herd of models (2024), arXiv:2407.21783

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  8. [8]

    Nature645(8081), 633–638 (Sep 2025)

    Guo, D., et al: Deepseek-r1 incentivizes reasoning in llms through reinforcement learning. Nature645(8081), 633–638 (Sep 2025)

    work page 2025

  9. [9]

    In: Pro- ceedings of the 41st International Conference on Machine Learning (2024)

    Hu, X., et al: Infiagent-dabench: evaluating agents on data analysis tasks. In: Pro- ceedings of the 41st International Conference on Machine Learning (2024)

    work page 2024

  10. [10]

    In: Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track (2023)

    Li, J., et al: Can LLM already serve as a database interface? a BIg bench for large- scale database grounded text-to-SQLs. In: Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track (2023)

    work page 2023

  11. [11]

    In: Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering

    Liu, L., Hasegawa, S., Sampat, S.K., Xenochristou, M., Chen, W.P., Kato, T., Kakibuchi, T., Asai, T.: Autodw: Automatic data wrangling leveraging large lan- guage models. In: Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering. p. 2041–2052 (2024) 16 So Hasegawa et al

    work page 2041

  12. [12]

    Majumder, B.P., Surana, H., Agarwal, D., Mishra, B.D., Meena, A., Prakhar, A., Vora, T., Khot, T., Sabharwal, A., Clark, P.: Discoverybench: Towards data-driven discovery with large language models (2024), arXiv:2407.01725

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  13. [13]

    In: Proceed- ings of the 9th Python in Science Conference

    Wes McKinney: Data Structures for Statistical Computing in Python. In: Proceed- ings of the 9th Python in Science Conference. pp. 56 – 61 (2010)

    work page 2010

  14. [14]

    MistralAI: Upgrading agentic coding capabilities with the new devstral models (accessed July 10, 2025),https://mistral.ai/news/devstral-2507

    work page 2025

  15. [15]

    Transactions of the Association for Computational Linguistics10, 35–49 (2022)

    Nan, L., et al: FeTaQA: Free-form table question answering. Transactions of the Association for Computational Linguistics10, 35–49 (2022)

    work page 2022

  16. [16]

    Blog Post (November 2025),https://openai.com/index/gpt-5-1/, accessed: March 5, 2026

    OpenAI: Gpt-5.1: A smarter, more conversational chatgpt. Blog Post (November 2025),https://openai.com/index/gpt-5-1/, accessed: March 5, 2026

    work page 2025

  17. [17]

    OpenAI, et al: Gpt-4 technical report (2024), arXiv:2303.08774

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  18. [18]

    OpenAI, et al: Gpt-4o system card (2024), arXiv:2410.21276

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  19. [19]

    In: Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)

    Osés Grijalba, J., Ureña-López, L.A., Martínez Cámara, E., Camacho-Collados, J.: Question answering over tabular data with DataBench: A large-scale empirical evaluation of LLMs. In: Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024). pp. 13471–13488 (May 2024)

    work page 2024

  20. [20]

    Pasupat, P., Liang, P.: Compositional semantic parsing on semi-structured tables. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Pro- cessing (Volume 1: Long Papers). pp. 1470–1480 (2015)

    work page 2015

  21. [21]

    Qwen: Qwen3-coder: Agentic coding in the world (accessed March 6, 2026),https: //qwen.ai/blog?id=qwen3-coder

    work page 2026

  22. [22]

    In: ICLR (2025)

    Sahu, G., Puri, A., Rodriguez, J.A., Abaskohi, A., Chegini, M., Drouin, A., Taslakian,P.,Zantedeschi,V.,Lacoste,A.,Vázquez,D.,Chapados,N.,Pal,C.,Ra- jeswar,S.,Laradji,I.H.:Insightbench:Evaluatingbusinessanalyticsagentsthrough multi-step insight generation. In: ICLR (2025)

    work page 2025

  23. [23]

    In: Proceedings of the63rdAnnualMeetingoftheAssociationforComputationalLinguistics(Volume 1: Long Papers)

    Seo,K.,Kwon,D.,Lee,D.:MT-RAIG:Novelbenchmarkandevaluationframework for retrieval-augmented insight generation over multiple tables. In: Proceedings of the63rdAnnualMeetingoftheAssociationforComputationalLinguistics(Volume 1: Long Papers). pp. 23142–23172 (2025)

    work page 2025

  24. [24]

    Su, A., et al: Tablegpt2: A large multimodal model with tabular data integration (2024), arXiv:2411.02059

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  25. [25]

    Gemini: A Family of Highly Capable Multimodal Models

    Team, G., et al: Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  26. [26]

    Team, G., et al: Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context (2024), arXiv:2403.05530

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  27. [27]

    UK Government: Data.gov.uk,https://www.data.gov.uk/, accessed: 2026-03-11

    work page 2026

  28. [28]

    General Services Administration: Data.gov,https://www.data.gov/, ac- cessed: 2026-03-11

    U.S. General Services Administration: Data.gov,https://www.data.gov/, ac- cessed: 2026-03-11

    work page 2026

  29. [29]

    Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Scao, T.L., Gugger, S., Drame, M., Lhoest, Q., Rush, A.M.: Huggingface’s transformers: State-of-the-art natural language pro- cessing (2020)

    work page 2020

  30. [30]

    In: The Thirteenth International Conference on Learning Representations (2025) Data Analysis in the Wild 17

    Wu,J.,Yang,L.,Li,D.,Ji,Y.,Okumura,M.,Zhang,Y.:MMQA:EvaluatingLLMs with multi-table multi-hop complex questions. In: The Thirteenth International Conference on Learning Representations (2025) Data Analysis in the Wild 17

    work page 2025

  31. [31]

    In: Proceedings of the AAAI Conference on Artificial Intelligence

    Wu,X.,etal:Tablebench:Acomprehensiveandcomplexbenchmarkfortableques- tion answering. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 39, pp. 25497–25506 (2025)

    work page 2025

  32. [32]

    Yang, A., et al: Qwen3 technical report (2025), arXiv:2505.09388

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  33. [33]

    In: Proceedings of the 2018 Confer- ence on Empirical Methods in Natural Language Processing

    Yu, T., et al: Spider: A large-scale human-labeled dataset for complex and cross- domain semantic parsing and text-to-SQL task. In: Proceedings of the 2018 Confer- ence on Empirical Methods in Natural Language Processing. pp. 3911–3921 (2018)

    work page 2018

  34. [34]

    In: Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing

    Zhang, J., et al: T2R-BENCH: A benchmark for real world table-to-report task. In: Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. pp. 22427–22451 (Nov 2025)

    work page 2025

  35. [35]

    Data-to-Dashboard: Multi-Agent LLM Framework for Insightful Visualization in Enterprise Analytics

    Zhang, R., Elhamod, M.: Data-to-dashboard: Multi-agent llm framework for in- sightful visualization in enterprise analytics (2025), arXiv:2505.23695 A Benchmark Construction A.1 Data Curation Data CollectionTable 6 lists 53 websites, from which we downloaded all datasets along with their metadata via APIs (e.g., CKAN API2). Data FilteringAll downloaded da...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  36. [36]

    generally showed an in- crease in concentrations with in- creasing age across all cycles, with the 60–79 years age group consistently exhibiting the highest arithmetic means (e.g., 140 ng/g lipid in Cycle 1). Polychlorinated biphenyls (PCBs), particularly ’Marker polychlorinated biphenyls (sum of PCB 138, 153, 180)’, show a clear age-related accumulation,...

    work page