Pith. sign in

REVIEW 4 major objections 6 minor 71 references

DataLab: A Unified Platform for LLM-Powered Business Intelligence

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read DataLab claims one notebook can handle all four major BI tasks at the level of specialized agents.

desk verdict Solid systems paper with real engineering, but the abstract overclaims SOTA against its own Table I and the enterprise evaluation is too thin to carry the headline numbers. read the letter →

arxiv 2412.02205 v3 pith:JQCH2FXO submitted 2024-12-03 cs.DB cs.AIcs.CL

Luoxuan Weng , Yinghao Tang , Yingchaojie Feng , Zhuo Chang , Ruiqin Chen , Haozhe Feng , Chen Hou , Danqing Huang
show 13 more authors
Yang Li Huaming Rao Haonan Wang Canshi Wei Xiaofeng Yang Yuhui Zhang Yifeng Zheng Xiuqi Huang Minfeng Zhu Yuxin Ma Bin Cui Peng Chen Wei Chen
This is my paper · ORCID
classification cs.DBcs.AIcs.CL
keywords businessintelligenceLLMagentsNL2SQLNL2VISdataanalysisvisualizationmulti-agentsystemsnotebookinterfaces
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

DataLab is a proposal to collapse the usual business-intelligence workflow—data preparation, analysis, visualization, and insight delivery—into one notebook augmented by LLM agents. The paper argues that prior agents are fragmented across tasks and tools, and that a unified platform can work if it automates domain knowledge, structures communication between agents, and manages notebook context by dependency. On research benchmarks it reports the best pass rate on code-generation and visualization tasks and competitive results on the rest. On enterprise data, it reports that domain knowledge raises schema linking recall from 41.02% to 79.49% and NL2DSL accuracy from 32.52% to 91.10%, while dependency-aware context pruning cuts token cost by 61.65%. The payoff, if these results hold, is that one environment can serve data engineers, scientists, and analysts without the usual handoffs between tools.

What carries the argument

The argument is carried by three modules. The Domain Knowledge Incorporation module automatically generates knowledge about databases, tables, columns, and values from historical SQL/Python scripts and data lineage, organizes it in a knowledge graph, and uses it to rewrite and translate queries into a structured DSL. The Inter-Agent Communication module represents each agent's output as a six-field information unit and routes those units through a shared buffer according to a finite state machine. The Cell-based Context Management module builds a directed acyclic graph of notebook-cell dependencies from variable references, traverses it to find relevant cells, and prunes by task type to supply a minimal context set. These three mechanisms are what the paper claims allow a single notebook to match task-specific agents and to cut token cost.

What would settle it

A human audit of generated knowledge on a sample far larger than the 50 tables used in the paper—counting how often expert annotations contradict the LLM-generated descriptions and calculation logic—would settle whether the reported gains on schema linking and NL2DSL come from the knowledge itself or from the DSL and retrieval machinery around it.

Watch

Extended reading notes

Core claim

The central claim is that unification does not cost accuracy: a single multi-agent notebook can carry a BI task from a natural-language query to SQL, Python analysis, chart, and insight with results comparable to or better than agents built for one task each. The paper identifies three mechanisms as the reason. Domain knowledge is generated automatically from historical SQL/Python scripts and data lineage, organized in a knowledge graph, and used to rewrite ambiguous queries into a structured DSL. Agents exchange six-field information units through a shared buffer under a finite state machine instead of free-form text, and notebook contexts are selected by traversing a dependency DAG of cells and pruning by task type. The enterprise experiments are where the claim is sharpest: schema linking recall rises from 41.02% to 79.49%, NL2DSL accuracy from 32.52% to 91.10%, and token cost per query falls by 61.65%.

Load-bearing premise

The load-bearing premise is that LLM summaries of historical SQL/Python scripts and data lineage correctly capture what ambiguous enterprise table and column names actually mean, with only the LLM's own self-calibration score as the quality gate.

Editorial extensions

If this is right

  • A user could move from a natural-language question to a finished chart or report inside one notebook, with generated SQL, Python, and chart specifications that remain editable.
  • Enterprise-specific jargon and ambiguous column names stop blocking LLM agents, because knowledge extracted once from script history is reused across schema linking, DSL translation, and downstream generation.
  • Multi-agent collaboration becomes more reliable: on the paper's enterprise question set, the structured communication module raises success rate from 73% to 92% and accuracy from 56% to 84%.
  • Context pruning makes LLM-powered notebooks affordable at scale: a 61.65% token-cost reduction per query with accuracy nearly unchanged translates directly into lower API spending.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that its knowledge-generation recipe—parse script history, summarize with an LLM, organize as a graph—could serve as a general self-updating data-catalog mechanism for any organization with query logs.
  • Because DataLab reduces everything to an intermediate DSL, the same design could make NL2SQL, NL2DSCode, and NL2VIS agents interchangeable: any agent that can read and emit the DSL could drop into the workflow.
  • The DAG-based context-selection idea is not tied to BI. It could be tested in ordinary Python notebooks and IDE assistants, where cell and file dependencies play the same role as the notebook cells here.
  • A natural stress test is to replace the LLM's self-calibration score with expert labels during knowledge generation; if accuracy holds, the knowledge is genuinely captured, and if it drops, the reported gains depend on the model's own confidence rather than on the knowledge itself.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. DataLab is a unified BI platform that integrates an LLM-based agent framework with an augmented computational notebook interface. It supports data preparation, analysis, and visualization tasks—including NL2SQL, NL2DSCode, NL2VIS, and NL2Insight—within one environment. The paper proposes three modules: automated domain knowledge incorporation from enterprise scripts and lineage; FSM-based structured inter-agent communication; and cell-dependency DAG-based context management for cost-efficient notebook execution. The authors evaluate on public research benchmarks and on proprietary Tencent datasets, reporting competitive results across tasks, up to a 58.58% accuracy gain and 61.65% token cost reduction on enterprise-specific tasks, plus deployment statistics at Tencent.

Significance. If the claims are supported, DataLab is a noteworthy systems contribution: unifying several previously fragmented BI tasks in a single notebook environment, with a practical automated knowledge-generation pipeline for enterprise data and a principled context-pruning mechanism. The paper's strengths include evaluation across multiple public benchmarks spanning four BI stages, real-world deployment at Tencent with sizable scale (2,426 databases, 262,041 tables), and clear descriptions of the three modules with algorithmic pseudocode. The work also makes a genuinely falsifiable claim about cost-effectiveness via token reduction. However, the headline 'state-of-the-art' claim is internally inconsistent with the reported table, and the enterprise-specific gains rest on small, self-constructed datasets with no statistical validation, so the significance is real but currently over-stated.

major comments (4)
  1. [Abstract and Table I] The abstract claims DataLab 'achieves state-of-the-art performance on various BI tasks across popular research benchmarks,' but Table I directly contradicts this: DataLab trails the best baseline on 6 of 10 metric/benchmark combinations, including Spider (80.70 vs. CHESS 87.20 and PURPLE 87.80), BIRD (61.33 vs. CHESS 68.31), nvBench (53.90 vs. LIDA 54.71), DABench (75.10 vs. AgentPoirot 75.88), InsightBench ROUGE-1 (0.33 vs. 0.35), and VisEval Readability (3.73 vs. 3.80). Section VII-A2 itself only claims 'comparable performance' and concedes that 'certain baselines excel in individual tasks.' The abstract must be revised to accurately represent these results, e.g., as competitive or comparable rather than state-of-the-art, or the claim must be qualified as state-of-the-art for a unified platform rather than for individual tasks.
  2. [Section VII-C, Tables II and IV] The headline enterprise-specific improvements—38.47% Schema Linking recall increase, 58.58% NL2DSL accuracy increase, and 61.65% token cost reduction—are all measured on small author-collected datasets (439, 326, and 150 samples respectively) with no error bars, no multiple independent runs, and no significance tests. For example, Table II reports S3 vs. S1 differences of 38.47 and 58.58 percentage points on sample sizes below 500, and Table IV reports a 4.67% accuracy drop with DAG pruning alongside a 61.65% token reduction, but the reader cannot assess whether these differences are stable. The authors should report confidence intervals, per-run variance, or at least multiple seeds, and should specify the exact evaluation protocol (e.g., GPT-4 temperature, number of runs) for these private datasets.
  3. [Section IV-A, Algorithm 1] Algorithm 1's self-calibration loop uses the LLM's own 1–5 score as the quality gate, with the threshold T as a free parameter. This is a circularity risk: the gate checks whether the LLM is satisfied with its own output, not whether that output matches ground-truth semantics. The only external validation in Section VII-C1 is embedding similarity (SES 0.712 for tables, 0.677 for columns) on 50 tables with no human agreement rate or comparison baseline. To make the domain-knowledge claims load-bearing, the authors should validate the self-calibration score against expert judgments (e.g., correlation or agreement) or show that results are insensitive to T across a reasonable range. Without this, the 'high-quality knowledge' claim lacks independent support.
  4. [Section VII-C2 and VII-D] The downstream benefit of domain knowledge is measured on tasks (Schema Linking and NL2DSL) that are designed by the authors and annotated by (presumably) the same team, using the authors' own generated knowledge as the intervention. This is not circular in the benchmark sense, but it is a potential confirmation-bias channel because the test queries and the knowledge generation are both tuned to the same enterprise context. The authors should either release these datasets to enable external evaluation, or provide an independent human annotation protocol with inter-annotator agreement, and should compare against a simple baseline (e.g., manually curated table/column descriptions) to show the advantage is not merely from having any descriptions at all.
minor comments (6)
  1. [Section VII-A2] The text says 'DataLab outperforms all baselines on benchmarks including DS-1000, DSEval, InsightBench, and VisEval,' but Table I shows that on VisEval Readability, DataLab (3.73) is below CoML4VIS (3.80) and LIDA (3.77). This sentence should be corrected to reflect that DataLab is best on Pass Rate but not on Readability.
  2. [Section II-B and Section III] The paper would benefit from a precise definition of what 'state-of-the-art' means in the context of a unified platform versus a task-specific agent; several baselines (e.g., PURPLE, CHESS) are specifically optimized for a single task, so a direct comparison on each task is inherently unfavorable to a unified system. This trade-off should be stated explicitly in the abstract or conclusion.
  3. [Section VI-E2, Table IV] The token cost reduction of 61.65% is reported without specifying the tokenization method, the average number of cells in the notebooks, or the variance across the 150 queries. Since this is a central cost-efficiency claim, these details should be added to allow reproduction.
  4. [Section VII-C1] The SES similarity scores (0.712 and 0.677) are described as 'indicating practical utility,' but there is no comparison to a random or naive baseline (e.g., using column names directly), nor a threshold indicating what SES value constitutes acceptable quality. Adding such a baseline would strengthen the interpretation.
  5. [Section V] The FSM-based information-sharing protocol is described at a high level, but the paper does not specify how the FSM is constructed from a user query or how the proxy agent decides which agents to include. A concrete example of an FSM transition for a multi-task query would improve clarity.
  6. [General] The paper has numerous formatting issues: reference [47] has a rendering artifact ('XingyuHan'), the table captions in Table I are ambiguous about which baseline corresponds to which column, and Figure 6's y-axis labels are truncated. These should be cleaned up before publication.

Circularity Check

1 steps flagged · score 2.0 of 10

Mild self-referential knowledge-quality gate and author-collected enterprise evaluation; public-benchmark claims are externally benchmarked and the SOTA overstatement is a correctness issue, not circularity.

  1. other [Section IV-A, Algorithm 1 (map phase, Self-Calibration) and the paragraph following Algorithm 1]
    "we integrate a self-calibration mechanism that leverages LLMs' self-reflection abilities [33] to evaluate the intermediate results using a numerical score ranging from 1 to 5. ... Should the rating score si fall below the predefined threshold T, the knowledge generation process must be repeated. Therefore, this feedback loop ensures the generation quality of each iteration."

    In Algorithm 1, the same LLM that generates the knowledge components (di, ti, ci) also produces the quality score si, and the loop terminates when si >= T. The accepted knowledge is therefore, by construction, knowledge that the generator rates highly; 'quality' in this loop is defined by the generator's self-agreement. The sentence 'this feedback loop ensures the generation quality' is tautological with respect to the loop's own criterion. The circularity is partial and non-central because the paper separately validates generated knowledge against expert-annotated ground truth via SES (Section VII-C1) and evaluates downstream tasks with objective metrics, so the main benchmark results do not reduce to this self-score.

full rationale

The paper's headline benchmark comparisons (Spider, BIRD, DS-1000, DSEval, DABench, InsightBench, nvBench, VisEval) are run against external public datasets with external ground truth, so the central performance claims are not self-referential. The enterprise-specific gains (Table II: Schema Linking 41.02 to 79.49, NL2DSL 32.52 to 91.10) are measured on author-collected Tencent datasets, and the domain knowledge is generated from the same enterprise's script history and lineage; this creates a mild risk that the evaluation measures self-consistency rather than external utility, but the knowledge is not fitted to the test query-DSL labels and the metrics are objective, so this is a limitation rather than a reduction. The clearest self-referential component is Algorithm 1's self-calibration gate, where the LLM scores its own generated knowledge and the paper claims this 'ensures' quality; taken alone that claim is tautological. The SES evaluation against expert annotations (0.712 for tables, 0.677 for columns) provides partial external support, but it covers only 50 of 262,041 processed tables, leaving the bulk of deployed knowledge quality gated by the self-score. The abstract's 'state-of-the-art' claim is not supported by the paper's own Table I, where DataLab trails the best baseline on 6 of 10 reported benchmark metrics; that is an internal inconsistency and a correctness risk, not a circularity. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from the authors' prior work, and no central derivation reduces to an input by construction. Overall circularity is minor and non-load-bearing, warranting a score of 2.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The platform's central performance claims rely on several unverified premises: that LLMs can generate trustworthy domain knowledge from code, that script usage history reflects column semantics, that self-calibration scores are a valid quality signal, that embedding similarity is an adequate proxy for knowledge quality, and that AST-based cell dependency tracking captures all context needed. The retrieval weights and thresholds in Algorithm 2 are fitted per task but not reported. No invented entities are introduced.

free parameters (3)
  • Knowledge retrieval weights (omega1, omega2, omega3) = Not reported
    Algorithm 2 computes matching scores as a weighted sum of lexical, semantic, and LLM evaluations, with weights said to be tailored to each BI task but values not given.
  • Top-K knowledge node count K = Not reported
    Section IV-C says K is set to a relatively large value for coverage; the exact K is not disclosed.
  • Self-calibration threshold T = Not reported
    Algorithm 1 uses a predefined score threshold T between 1 and 5 to accept generated knowledge; the value is not given.
assumptions (5)
  • domain assumption LLMs can generate accurate domain knowledge from historical data processing scripts via Map-Reduce with self-calibration.
    Algorithm 1 assumes that LLM-generated descriptions of databases, tables, and columns from SQL/Python script history are correct enough to improve downstream BI tasks. Section VII-C provides only embedding-similarity validation, not human audit.
  • domain assumption Script history and data lineage reflect the semantic meanings and usage patterns of tables and columns.
    Section IV-A motivates knowledge generation from script history and lineage, claiming that these reflect common business usage. If scripts are outdated, anomalous, or unrepresentative, the generated knowledge is misleading.
  • ad hoc to paper The LLM's self-calibration score is a valid proxy for knowledge quality.
    Algorithm 1 loops until the LLM's own 1-5 rating exceeds threshold T, but no evidence connects the rating to external correctness. This is an unverified internal quality gate.
  • domain assumption Sentence embedding similarity between generated descriptions and expert annotations is an adequate quality measure for domain knowledge.
    Section VII-C.1 uses M3-Embedding SES with 0.712 for tables and 0.677 for columns, with no baseline or human agreement comparison, treating similarity as accuracy.
  • domain assumption AST-based variable reference tracking captures the dependency structure needed for context selection in multi-language notebooks.
    Algorithm 3 assumes Python and SQL cell dependencies are fully determined by global variables and SELECT outputs, and that Markdown relevance can be found by embedding similarity. The paper acknowledges Markdown retrieval failures in Section VII-E.2.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DataLab: A Unified Platform for LLM-Powered Business Intelligence." pith.science (2026). https://pith.science/paper/JQCH2FXO

@misc{pith2026241202205,
  author       = {Pith},
  title        = {Pith review of: DataLab: A Unified Platform for LLM-Powered Business Intelligence},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JQCH2FXO}},
  note         = {Machine review of arXiv:2412.02205}
}
read the original abstract

Business intelligence (BI) transforms large volumes of data within modern organizations into actionable insights for informed decision-making. Recently, large language model (LLM)-based agents have streamlined the BI workflow by automatically performing task planning, reasoning, and actions in executable environments based on natural language (NL) queries. However, existing approaches primarily focus on individual BI tasks such as NL2SQL and NL2VIS. The fragmentation of tasks across different data roles and tools lead to inefficiencies and potential errors due to the iterative and collaborative nature of BI. In this paper, we introduce DataLab, a unified BI platform that integrates a one-stop LLM-based agent framework with an augmented computational notebook interface. DataLab supports various BI tasks for different data roles in data preparation, analysis, and visualization by seamlessly combining LLM assistance with user customization within a single environment. To achieve this unification, we design a domain knowledge incorporation module tailored for enterprise-specific BI tasks, an inter-agent communication mechanism to facilitate information sharing across the BI workflow, and a cell-based context management strategy to enhance context utilization efficiency in BI notebooks. Extensive experiments demonstrate that DataLab achieves state-of-the-art performance on various BI tasks across popular research benchmarks. Moreover, DataLab maintains high effectiveness and efficiency on real-world datasets from Tencent, achieving up to a 58.58% increase in accuracy and a 61.65% reduction in token cost on enterprise-specific BI tasks.

Figures

Figures reproduced from arXiv: 2412.02205 by the authors.

Figure 1
Figure 1. Overview of DataLab and its three critical modules. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Example agent workflow for NL2VIS. by other auxiliary components like memory modules, each BI agent is represented as a DAG for high flexibility and easy extensibility. Within the DAG, nodes depict reusable components (e.g., LLM APIs, tools) and edges depict their connections (e.g., file transfer across tools) [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The notebook interface of DataLab. Meanwhile, a context management strategy ( [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Structure of the knowledge graph. C. Knowledge Utilization Algorithm 2 Knowledge Retrieval Input: User Query Q, Knowledge Graph G Output: Knowledge Nodes VQ 1: VQ ← ∅ 2: Coarse-Grained Retrieval: 3: VQ ← lex search(Q, G) + sem search(Q, G) 4: Fine-Grained Ordering: 5: …
Figure 5
Figure 5. Figure 5: Workflow of Inter-Agent Communication. queries, Python code, and charts. This variety poses a sig￾nificant challenge in ensuring accurate information sharing without introducing redundancy or miscommunication. Exist￾ing frameworks [17], [18] that rely on unstructured n…
Figure 6
Figure 6. Figure 6: Performance of DataLab using various underlying LLMs. [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Time cost of DAG construction. TABLE IV ABLATION STUDY ON CELL-BASED CONTEXT MANAGEMENT Metric S1 S2 Accuracy (%) 86.67 82.00 Token Cost per Query (K) 10.69 4.10 2) Task Completion: For each notebook in our collected dataset, we derive 3 real-world user queries, which …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

71 extracted references · 51 canonical work pages

  1. [1]

    Conversational BI: an ontology-driven conversationsystem for business intelligence applications,

    A. Quamar, F. ¨Ozcan, D. Miller, R. J. Moore, R. Niehus, and J. T. Kreulen, “Conversational BI: an ontology-driven conversationsystem for business intelligence applications,” Proc. VLDB Endow., vol. 13, no. 12, pp. 3369–3381, 2020

    work page 2020

  2. [2]

    Spider2- v: How far are multimodal agents from automating data science and engineering workflows?

    R. Cao, F. Lei, H. Wu, J. Chen, Y . Fu, H. Gao, X. Xiong, H. Zhang, Y . Mao, W. Hu, T. Xie, H. Xu, D. Zhang, S. Wang, R. Sun, P. Yin, C. Xiong, A. Ni, Q. Liu, V . Zhong, L. Chen, K. Yu, and T. Yu, “Spider2- v: How far are multimodal agents from automating data science and engineering workflows?” CoRR, vol. abs/2407.10956, 2024

    arXiv 2024

  3. [3]

    Openagents: An open platform for language agents in the wild,

    T. Xie, F. Zhou, Z. Cheng, P. Shi, L. Weng, Y . Liu, T. J. Hua, J. Zhao, Q. Liu, C. Liu, Z. Liu, Y . Xu, H. SU, D. Shin, C. Xiong, and T. Yu, “Openagents: An open platform for language agents in the wild,” in COLM, 2024

    work page 2024

  4. [4]

    DS-1000: A natural and reliable benchmark for data science code generation,

    Y . Lai, C. Li, Y . Wang, T. Zhang, R. Zhong, L. Zettlemoyer, W. Yih, D. Fried, S. I. Wang, and T. Yu, “DS-1000: A natural and reliable benchmark for data science code generation,” in ICML, ser. Proceedings of Machine Learning Research, vol. 202. PMLR, 2023, pp. 18 319– 18 345

    work page 2023

  5. [5]

    Automated data visualization from natural language via large language models: An exploratory study,

    Y . Wu, Y . Wan, H. Zhang, Y . Sui, W. Wei, W. Zhao, G. Xu, and H. Jin, “Automated data visualization from natural language via large language models: An exploratory study,” Proc. ACM Manag. Data , vol. 2, no. 3, p. 115, 2024

    work page 2024

  6. [6]

    Insightlens: Discovering and exploring insights from conversational contexts in large- language-model-powered data analysis,

    L. Weng, X. Wang, J. Lu, Y . Feng, Y . Liu, and W. Chen, “Insightlens: Discovering and exploring insights from conversational contexts in large- language-model-powered data analysis,” CoRR, vol. abs/2404.01644, 2024

    arXiv 2024

  7. [7]

    Notebooks for data analysis and visualization: Moving beyond the data,

    R. Kosara, “Notebooks for data analysis and visualization: Moving beyond the data,” IEEE Computer Graphics and Applications , vol. 43, no. 1, pp. 91–96, 2023

    work page 2023

  8. [8]

    BI-REC: Guided Data Analysis for Conversational Business Intelligence

    V . V . Meduri, A. Quamar, C. Lei, V . Efthymiou, and F. Ozcan, “BI-REC: guided data analysis for conversational business intelligence,”CoRR, vol. abs/2105.00467, 2021

    work page Pith review arXiv 2021

Show all 71 references
  1. [9]

    Data interpreter: An LLM agent for data science,

    S. Hong, Y . Lin, B. Liu, B. Liu, B. Wu, D. Li, J. Chen, J. Zhang, J. Wang, L. Zhang, L. Zhang, M. Yang, M. Zhuge, T. Guo, T. Zhou, W. Tao, W. Wang, X. Tang, X. Lu, X. Zheng, X. Liang, Y . Fei, Y . Cheng, Z. Xu, and C. Wu, “Data interpreter: An LLM agent for data science,” CoR...

    2024 arXiv

  2. [10]

    Tablegpt2: A large multimodal model with tabular data integration,

    A. Su, A. Wang, C. Ye, C. Zhou, G. Zhang, G. Zhu, H. Wang, H. Xu, H. Chen, H. Li, H. Lan, J. Tian, J. Yuan, J. Zhao, J. Zhou, K. Shou, L. Zha, L. Long, L. Li, P. Wu, Q. Zhang, Q. Huang, S. Yang, T. Zhang, W. Ye, W. Zhu, X. Hu, X. Gu, X. Sun, X. Li, Y . Yang, and Z. Xiao, “Tabl...

    2024 arXiv

  3. [11]

    BEA VER: an enterprise benchmark for text-to-sql,

    P. B. Chen, F. Wenz, Y . Zhang, M. Kayali, N. Tatbul, M. J. Cafarella, C ¸ . Demiralp, and M. Stonebraker, “BEA VER: an enterprise benchmark for text-to-sql,” CoRR, vol. abs/2409.02038, 2024

    2024 arXiv

  4. [12]

    Chatbi: Towards natural language to complex business intelligence SQL,

    J. Lian, X. Liu, Y . Shao, Y . Dong, M. Wang, Z. Wei, T. Wan, M. Dong, and H. Yan, “Chatbi: Towards natural language to complex business intelligence SQL,” CoRR, vol. abs/2405.00527, 2024

    2024 arXiv

  5. [13]

    Fine tuning LLM for enterprise: Practical guidelines and recommendations,

    M. R. J, K. VM, H. Warrier, and Y . Gupta, “Fine tuning LLM for enterprise: Practical guidelines and recommendations,” CoRR, vol. abs/2404.10779, 2024

    2024 arXiv

  6. [14]

    Simple and scalable strategies to continually pre-train large language models,

    A. Ibrahim, B. Th ´erien, K. Gupta, M. L. Richter, Q. G. Anthony, E. Belilovsky, T. Lesort, and I. Rish, “Simple and scalable strategies to continually pre-train large language models,” Trans. Mach. Learn. Res., vol. 2024, 2024

    2024

  7. [15]

    Chat2data: An interactive data analysis system with rag, vector databases and llms,

    X. Zhao, X. Zhou, and G. Li, “Chat2data: An interactive data analysis system with rag, vector databases and llms,” Proc. VLDB Endow. , vol. 17, no. 12, pp. 4481–4484, 2024

    2024

  8. [16]

    Autogen: Enabling next-gen LLM applications via multi-agent conversation framework,

    Q. Wu, G. Bansal, J. Zhang, Y . Wu, S. Zhang, E. Zhu, B. Li, L. Jiang, X. Zhang, and C. Wang, “Autogen: Enabling next-gen LLM applications via multi-agent conversation framework,” CoRR, vol. abs/2308.08155, 2023

    2023 arXiv

  9. [17]

    Chatdev: Commu- nicative agents for software development,

    C. Qian, W. Liu, H. Liu, N. Chen, Y . Dang, J. Li, C. Yang, W. Chen, Y . Su, X. Cong, J. Xu, D. Li, Z. Liu, and M. Sun, “Chatdev: Commu- nicative agents for software development,” in ACL. Association for Computational Linguistics, 2024, pp. 15 174–15 186

    2024

  10. [18]

    CAMEL: communicative agents for

    G. Li, H. Hammoud, H. Itani, D. Khizbullin, and B. Ghanem, “CAMEL: communicative agents for ”mind” exploration of large language model society,” in NeurIPS, 2023

    2023

  11. [19]

    Metagpt: Meta programming for A multi-agent collaborative framework,

    S. Hong, M. Zhuge, J. Chen, X. Zheng, Y . Cheng, J. Wang, C. Zhang, Z. Wang, S. K. S. Yau, Z. Lin, L. Zhou, C. Ran, L. Xiao, C. Wu, and J. Schmidhuber, “Metagpt: Meta programming for A multi-agent collaborative framework,” in ICLR. OpenReview.net, 2024

    2024

  12. [20]

    Large language models can be easily distracted by irrelevant context,

    F. Shi, X. Chen, K. Misra, N. Scales, D. Dohan, E. H. Chi, N. Sch ¨arli, and D. Zhou, “Large language models can be easily distracted by irrelevant context,” in ICML, ser. Proceedings of Machine Learning Research, vol. 202. PMLR, 2023, pp. 31 210–31 227

    2023

  13. [21]

    PURPLE: making a large language model a better SQL writer,

    T. Ren, Y . Fan, Z. He, R. Huang, J. Dai, C. Huang, Y . Jing, K. Zhang, Y . Yang, and X. S. Wang, “PURPLE: making a large language model a better SQL writer,” in ICDE. IEEE, 2024, pp. 15–28

    2024

  14. [22]

    Chatpipe: Orchestrating data preparation pipelines by optimiz- ing human-chatgpt interactions,

    S. Chen, H. Liu, W. Jin, X. Sun, X. Feng, J. Fan, X. Du, and N. Tang, “Chatpipe: Orchestrating data preparation pipelines by optimiz- ing human-chatgpt interactions,” in SIGMOD Conference Companion . ACM, 2024, pp. 484–487

    2024

  15. [23]

    Chat2query: A zero-shot automatic exploratory data analysis system with large language models,

    J. Zhu, P. Cai, B. Niu, Z. Ni, K. Xu, J. Huang, J. Wan, S. Ma, B. Wang, D. Zhang, L. Tang, and Q. Liu, “Chat2query: A zero-shot automatic exploratory data analysis system with large language models,” in ICDE. IEEE, 2024, pp. 5429–5432

    2024

  16. [24]

    A survey on large language model based autonomous agents,

    L. Wang, C. Ma, X. Feng, Z. Zhang, H. Yang, J. Zhang, Z. Chen, J. Tang, X. Chen, Y . Lin, W. X. Zhao, Z. Wei, and J. Wen, “A survey on large language model based autonomous agents,” Frontiers Comput. Sci., vol. 18, no. 6, p. 186345, 2024

    2024

  17. [25]

    Text- to-sql empowered by large language models: A benchmark evaluation,

    D. Gao, H. Wang, Y . Li, X. Sun, Y . Qian, B. Ding, and J. Zhou, “Text- to-sql empowered by large language models: A benchmark evaluation,” Proc. VLDB Endow., vol. 17, no. 5, pp. 1132–1145, 2024

    2024

  18. [26]

    Insightbench: Evaluating business analytics agents through multi-step insight generation,

    G. Sahu, A. Puri, J. A. Rodriguez, A. Drouin, P. Taslakian, V . Zant- edeschi, A. Lacoste, D. V ´azquez, N. Chapados, C. Pal, S. Rajeswar, and I. H. Laradji, “Insightbench: Evaluating business analytics agents through multi-step insight generation,” CoRR, vol. abs/2407.06423, 2024

    2024 arXiv

  19. [27]

    Chat2vis: Generating data visualizations via natural language using chatgpt, codex and GPT-3 large language models,

    P. Maddigan and T. Susnjak, “Chat2vis: Generating data visualizations via natural language using chatgpt, codex and GPT-3 large language models,” IEEE Access, vol. 11, pp. 45 181–45 193, 2023

    2023

  20. [28]

    Pygwalker: On-the-fly as- sistant for exploratory visual data analysis,

    Y . Yu, L. Shen, F. Long, H. Qu, and H. Chen, “Pygwalker: On-the-fly as- sistant for exploratory visual data analysis,” CoRR, vol. abs/2406.11637, 2024

    2024 arXiv

  21. [29]

    CHESS: contextual harnessing for efficient SQL synthesis,

    S. Talaei, M. Pourreza, Y . Chang, A. Mirhoseini, and A. Saberi, “CHESS: contextual harnessing for efficient SQL synthesis,” CoRR, vol. abs/2405.16755, 2024

    2024 arXiv

  22. [30]

    LIDA: A tool for automatic generation of grammar-agnostic visualizations and infographics using large language models,

    V . Dibia, “LIDA: A tool for automatic generation of grammar-agnostic visualizations and infographics using large language models,” in ACL (demo). Association for Computational Linguistics, 2023, pp. 113– 126

    2023

  23. [31]

    SAC: A system for big data lineage tracking,

    M. Tang, S. Shao, W. Yang, Y . Liang, Y . Yu, B. Saha, and D. Hyun, “SAC: A system for big data lineage tracking,” in ICDE. IEEE, 2019, pp. 1964–1967

    2019

  24. [32]

    Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback,

    K. Tian, E. Mitchell, A. Zhou, A. Sharma, R. Rafailov, H. Yao, C. Finn, and C. D. Manning, “Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback,” in EMNLP. Association for Computational Linguistics...

    2023

  25. [33]

    Towards mitigating LLM hallucination via self reflection,

    Z. Ji, T. Yu, Y . Xu, N. Lee, E. Ishii, and P. Fung, “Towards mitigating LLM hallucination via self reflection,” in EMNLP (Findings). Associ- ation for Computational Linguistics, 2023, pp. 1827–1843

    2023

  26. [34]

    Gormley and Z

    C. Gormley and Z. Tong, Elasticsearch: the definitive guide: a dis- tributed real-time search and analytics engine. ” O’Reilly Media, Inc.”, 2015

    2015

  27. [35]

    Starrocks: A high-performance analytical database,

    StarRocks, “Starrocks: A high-performance analytical database,” https: //www.starrocks.io/, 2024

    2024

  28. [36]

    Can large language models be an alternative to human evaluations?

    D. C. Chiang and H. Lee, “Can large language models be an alternative to human evaluations?” in ACL. Association for Computational Linguistics, 2023, pp. 15 607–15 631

    2023

  29. [37]

    Foundations of JSON schema,

    F. Pezoa, J. L. Reutter, F. Su ´arez, M. Ugarte, and D. Vrgoc, “Foundations of JSON schema,” in WWW. ACM, 2016, pp. 263–273

    2016

  30. [38]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V . Le, and D. Zhou, “Chain-of-thought prompting elicits reasoning in large language models,” in NeurIPS, 2022

    2022

  31. [39]

    Self-polish: Enhance reasoning in large language models via problem refinement,

    Z. Xi, S. Jin, Y . Zhou, R. Zheng, S. Gao, J. Liu, T. Gui, Q. Zhang, and X. Huang, “Self-polish: Enhance reasoning in large language models via problem refinement,” in EMNLP (Findings) . Association for Computational Linguistics, 2023, pp. 11 383–11 406

    2023

  32. [40]

    Mixlora: Enhancing large language models fine-tuning with lora-based mixture of experts,

    D. Li, Y . Ma, N. Wang, Z. Ye, Z. Cheng, Y . Tang, Y . Zhang, L. Duan, J. Zuo, C. Yang, and M. Tang, “Mixlora: Enhancing large language models fine-tuning with lora-based mixture of experts,” 2024. [Online]. Available: https://arxiv.org/abs/2404.15159

    2024 arXiv

  33. [41]

    K. R. Fall and W. R. Stevens, Tcp/ip illustrated . Addison-Wesley Professional, 2012, vol. 1

    2012

  34. [42]

    Spider: A large- scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task,

    T. Yu, R. Zhang, K. Yang, M. Yasunaga, D. Wang, Z. Li, J. Ma, I. Li, Q. Yao, S. Roman, Z. Zhang, and D. R. Radev, “Spider: A large- scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task,” in EMNLP. Association for Computational Linguist...

    2018

  35. [43]

    Can LLM already serve as A database interface? A big bench for large-scale database grounded text-to-sqls,

    J. Li, B. Hui, G. Qu, J. Yang, B. Li, B. Li, B. Wang, B. Qin, R. Geng, N. Huo, X. Zhou, C. Ma, G. Li, K. C. Chang, F. Huang, R. Cheng, and Y . Li, “Can LLM already serve as A database interface? A big bench for large-scale database grounded text-to-sqls,” in NeurIPS, 2023

    2023

  36. [44]

    Mlcopilot: Unleashing the power of large language models in solving machine learning tasks,

    L. Zhang, Y . Zhang, K. Ren, D. Li, and Y . Yang, “Mlcopilot: Unleashing the power of large language models in solving machine learning tasks,” in EACL. Association for Computational Linguistics, 2024, pp. 2931– 2959

    2024

  37. [45]

    Open source implementation of the chatgpt code in- terpreter,

    shroominic, “Open source implementation of the chatgpt code in- terpreter,” https://github.com/shroominic/codeinterpreter-api, [Accessed 18-10-2024]

    2024

  38. [46]

    A natural language interface for computers,

    KillianLucas, “A natural language interface for computers,” https:// github.com/OpenInterpreter/open-interpreter, [Accessed 13-02-2025]

    2025

  39. [47]

    Benchmarking data science agents,

    Y . Zhang, Q. Jiang, X. XingyuHan, N. Chen, Y . Yang, and K. Ren, “Benchmarking data science agents,” in ACL. Association for Compu- tational Linguistics, 2024, pp. 5677–5700

    2024

  40. [48]

    Infiagent-dabench: Evaluating agents on data analysis tasks,

    X. Hu, Z. Zhao, S. Wei, Z. Chai, Q. Ma, G. Wang, X. Wang, J. Su, J. Xu, M. Zhu, Y . Cheng, J. Yuan, J. Li, K. Kuang, Y . Yang, H. Yang, and F. Wu, “Infiagent-dabench: Evaluating agents on data analysis tasks,” in ICML. OpenReview.net, 2024

    2024

  41. [49]

    Synthesizing natural language to visualization (NL2VIS) benchmarks from NL2SQL benchmarks,

    Y . Luo, N. Tang, G. Li, C. Chai, W. Li, and X. Qin, “Synthesizing natural language to visualization (NL2VIS) benchmarks from NL2SQL benchmarks,” in SIGMOD Conference. ACM, 2021, pp. 1235–1247

    2021

  42. [50]

    Viseval: A benchmark for data visualization in the era of large language models,

    N. Chen, Y . Zhang, J. Xu, K. Ren, and Y . Yang, “Viseval: A benchmark for data visualization in the era of large language models,” IEEE Transactions on Visualization and Computer Graphics , 2024

    2024

  43. [51]

    GPT-4 technical report,

    OpenAI, “GPT-4 technical report,” CoRR, vol. abs/2303.08774, 2023

    2023 arXiv

  44. [52]

    Visualization generation with large language models: An evaluation,

    G. Li, X. Wang, G. Aodeng, S. Zheng, Y . Zhang, C. Ou, S. Wang, and C. H. Liu, “Visualization generation with large language models: An evaluation,” CoRR, vol. abs/2401.11255, 2024

    2024

  45. [53]

    The dawn of lmms: Preliminary explorations with gpt-4v(ision),

    Z. Yang, L. Li, K. Lin, J. Wang, C. Lin, Z. Liu, and L. Wang, “The dawn of lmms: Preliminary explorations with gpt-4v(ision),” CoRR, vol. abs/2309.17421, 2023

    2023 arXiv

  46. [54]

    React: Synergizing reasoning and acting in language models,

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y . Cao, “React: Synergizing reasoning and acting in language models,” in ICLR. OpenReview.net, 2023

    2023

  47. [55]

    Grammar prompting for domain-specific language generation with large language models,

    B. Wang, Z. Wang, X. Wang, Y . Cao, R. A. Saurous, and Y . Kim, “Grammar prompting for domain-specific language generation with large language models,” in NeurIPS, 2023

    2023

  48. [56]

    Qwen2.5: A party of foundation models,

    Q. Team, “Qwen2.5: A party of foundation models,” September 2024. [Online]. Available: https://qwenlm.github.io/blog/qwen2.5/

    2024

  49. [57]

    The llama 3 herd of models,

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, A. Goyal, A. Hartshorn, A. Yang, A. Mitra, A. Sravankumar, A. Korenev, A. Hinsvark, A. Rao, A. Zhang, A. Rodriguez, A. Gregerson, A. Spataru, B. Rozi`ere, B. Biron, B. T...

    2024 arXiv

  50. [58]

    M3- embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation,

    J. Chen, S. Xiao, P. Zhang, K. Luo, D. Lian, and Z. Liu, “M3- embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation,” in ACL (Findings) . Association for Computational Linguistics, 2024, pp. 2318–2335

    2024

  51. [59]

    Re-examining the role of schema linking in text-to-sql,

    W. Lei, W. Wang, Z. Ma, T. Gan, W. Lu, M. Kan, and T. Chua, “Re-examining the role of schema linking in text-to-sql,” in EMNLP. Association for Computational Linguistics, 2020, pp. 6943–6954

    2020

  52. [60]

    A DSL for modeling application-specific functionalities of business applications,

    A. Popovic, I. Lukovic, V . Dimitrieski, and V . Djukic, “A DSL for modeling application-specific functionalities of business applications,” Comput. Lang. Syst. Struct. , vol. 43, pp. 69–95, 2015

    2015

  53. [61]

    Is cosine-similarity of embed- dings really about similarity?

    H. Steck, C. Ekanadham, and N. Kallus, “Is cosine-similarity of embed- dings really about similarity?” in WWW (Companion Volume). ACM, 2024, pp. 887–890

    2024

  54. [62]

    Tableau einstein,

    Tableau, “Tableau einstein,” https://www.tableau.com/, 2024

    2024

  55. [63]

    Power bi,

    Microsoft, “Power bi,” https://www.microsoft.com/en-us/ power-platform/products/power-bi, 2024

    2024

  56. [64]

    Databricks data intelligence platform,

    Databricks, “Databricks data intelligence platform,” https://www. databricks.com/, 2024

    2024

  57. [65]

    Do what I mean, not what I say! design con- siderations for supporting intent and context in analytical conversation,

    M. Tory and V . Setlur, “Do what I mean, not what I say! design con- siderations for supporting intent and context in analytical conversation,” in VAST. IEEE, 2019, pp. 93–103

    2019

  58. [66]

    XNLI: explaining and diagnosing nli-based visual data analysis,

    Y . Feng, X. Wang, B. Pan, K. Wong, Y . Ren, S. Liu, Z. Yan, Y . Ma, H. Qu, and W. Chen, “XNLI: explaining and diagnosing nli-based visual data analysis,” IEEE Trans. Vis. Comput. Graph. , vol. 30, no. 7, pp. 3813–3827, 2024

    2024

  59. [67]

    Demystifying data management for large language models,

    X. Miao, Z. Jia, and B. Cui, “Demystifying data management for large language models,” in SIGMOD Conference Companion . ACM, 2024, pp. 547–555

    2024

  60. [68]

    Llm-enhanced data management,

    X. Zhou, X. Zhao, and G. Li, “Llm-enhanced data management,” CoRR, vol. abs/2402.02643, 2024

    2024 arXiv

  61. [69]

    Conversational challenges in ai-powered data science: Obstacles, needs, and design opportunities,

    B. Chopra, A. Singha, A. Fariha, S. Gulwani, C. Parnin, A. Tiwari, and A. Z. Henley, “Conversational challenges in ai-powered data science: Obstacles, needs, and design opportunities,” CoRR, vol. abs/2310.16164, 2023

    2023 arXiv

  62. [70]

    Table-gpt: Table fine-tuned GPT for diverse table tasks,

    P. Li, Y . He, D. Yashar, W. Cui, S. Ge, H. Zhang, D. R. Fainman, D. Zhang, and S. Chaudhuri, “Table-gpt: Table fine-tuned GPT for diverse table tasks,” Proc. ACM Manag. Data , vol. 2, no. 3, p. 176, 2024

    2024

  63. [71]

    Insightpilot: An llm- empowered automated data exploration system,

    P. Ma, R. Ding, S. Wang, S. Han, and D. Zhang, “Insightpilot: An llm- empowered automated data exploration system,” in EMNLP (Demos) . Association for Computational Linguistics, 2023, pp. 346–352

    2023

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.