Pith

open record

sign in

arxiv: 2607.06229 · v1 · pith:EYNBZNXV · submitted 2026-07-07 · cs.CL · cs.AI· cs.DB

Spider 2.0-AIFunc: Extending Real-World Text-to-SQL to AI-Native SQL Workflows

Tianyang Liu , Canwen Xu , Fangyu Lei , Nikki Lijing Kuang , Jixuan Chen , Tao Yu , Julian McAuley , Zhewei Yao
show 1 more author
Yuxiong He
This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 reserved 2026-07-08 12:36 UTCglm-5.2pith:EYNBZNXVrecord.jsonopen to challenge →

Figure 1
Figure 1. Figure 1: An example of transforming a traditional SQL task into an AI-native SQL instance [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] reproduced from arXiv: 2607.06229
classification cs.CL cs.AIcs.DB
keywords ai-nativetext-to-sqlaifunclanguagemodelsspideracrossagent
0
0 comments X

The pith

First benchmark tests LLMs writing AI-native SQL

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces Spider 2.0-AIFunc, a benchmark of 465 verified instances across 125 real-world databases that tests whether language models can generate SQL queries incorporating AI functions—classification, sentiment analysis, similarity search, extraction, filtering, and aggregation—alongside conventional relational operators. Cloud platforms like Snowflake now expose large language model capabilities as native SQL functions, enabling analysts to perform semantic operations directly within queries, but no prior benchmark evaluated whether models can generate such queries. The benchmark is constructed by rewriting existing enterprise text-to-SQL tasks through an agent-based pipeline that injects AI function calls into target SQL queries while refining natural language instructions to make the intended AI-native solution explicit. Every instance passes a multi-round execution verification protocol across temporally separated windows to confirm result stability, addressing the fact that AI functions invoke language models whose outputs may vary even under deterministic settings. Evaluating ten state-of-the-art models, the paper finds that the strongest proprietary models reach 67-70% execution accuracy while the best open-source model achieves 58.1%, with errors concentrated in predicate specification, schema grounding, query logic, and AI function parameterization. A secondary finding is that agent frameworks designed for traditional text-to-SQL challenges—schema retrieval, relevant table selection, iterative refinement—do not transfer effectively to AI-native SQL: a minimal agent setup consistently matches or outperforms more elaborate alternatives.

Core claim

The central object the paper establishes is the AI-native SQL benchmark task: given a natural language instruction, a database schema, auxiliary documentation, and AI function reference documentation, a system must produce a SQL query that correctly composes AI function calls (such as AI_CLASSIFY, AI_FILTER, AI_SENTIMENT, AI_SIMILARITY, AI_EXTRACT, AI_AGG) with conventional SQL operators. The paper's core claim is that this task is meaningfully distinct from traditional text-to-SQL and that current models exhibit a measurable capability gap on it, with proprietary models at 67-70% accuracy and open-source models at 44.9-58.1%. The paper also discovers that the error profile shifts: AI-native

What carries the argument

AI-native SQL benchmark task: given a natural language instruction, a database schema, auxiliary documentation, and AI function reference documentation, a system must produce a SQL query that correctly composes AI function calls with conventional SQL operators. The paper's core claim is that this task is meaningfully distinct from traditional text-to-SQL and that current models exhibit a measurable capability gap on it, with proprietary models at 67-70% accuracy and open-source models at 44.9-58.1%. The paper also discovers that the error profile shifts: AI-native SQL correctness often depends on small semantic decisions—such as whether filtering happens before or after an AI call, or exact

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The finding that elaborate agent frameworks do not outperform a minimal setup suggests that the bottleneck in AI-native SQL has shifted from schema navigation to semantic reasoning about AI function placement and parameterization—a shift that may require fundamentally different agent architectures.
  • If AI functions become standard in enterprise SQL, the gap between proprietary and open-source models on this benchmark could widen further, since open-source models show systematic weaknesses specifically in AI function usage (C4 errors), a failure mode absent from traditional text-to-SQL.
  • The benchmark's dependence on a single construction model (Claude Opus 4.5) for both rewriting and verification may introduce systematic biases in instruction wording and function selection patterns that could advantage models with similar stylistic tendencies.

Load-bearing premise

The benchmark's validity depends on the assumption that the agent-based construction pipeline produces gold SQL queries and instructions that are both correct and unambiguously determinate. The paper acknowledges that it is difficult to fully verify that every instance rules out all reasonable alternative interpretations, and the error analysis finds that 23% of universal-failure instances show signs of annotation issues where the gold query encodes constraints absent from or

What would settle it

If a substantial fraction of the 465 gold queries are ambiguous or incorrect, the execution accuracy numbers are noisy and model rankings may be partially artifacts of benchmark limitations rather than model capability differences.

Figures

Figures reproduced from arXiv: 2607.06229 by Canwen Xu, Fangyu Lei, Jixuan Chen, Julian McAuley, Nikki Lijing Kuang, Tao Yu, Tianyang Liu, Yuxiong He, Zhewei Yao.

Figure 2
Figure 2. Figure 2: (a) Distribution of AI function types across the 465 released instances. A single task [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Distribution of error types across three failure strata, based on manual analysis of [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
read the original abstract

Major cloud data platforms now expose large language model capabilities as native SQL functions, enabling analysts to perform classification, filtering, sentiment analysis, extraction, similarity search, and aggregation within ordinary SQL queries. Yet existing text-to-SQL benchmarks evaluate only conventional SQL and provide no signal on whether models can generate such AI-native SQL. We introduce Spider 2.0-AIFunc, a benchmark of 465 verified instances across 125 real-world databases covering six types of AI functions on the Snowflake platform. Starting from an existing enterprise text-to-SQL benchmark, we construct Spider 2.0-AIFunc through an agent-based pipeline that rewrites source tasks into AI-native form, simultaneously transforming target queries and refining natural language instructions to make the intended AI-native solution explicit and reduce ambiguity. All instances pass a multi-round repeated execution protocol across temporally separated windows to confirm result stability before release. Evaluating ten state-of-the-art language models, we find that the strongest proprietary models reach 67-70% execution accuracy while the best open-source model achieves 58.1%, a gap driven primarily by errors in predicate specification, schema grounding, and AI function parameterization. Agent frameworks designed for traditional text-to-SQL challenges, such as schema retrieval and relevant table selection, do not transfer effectively to AI-native SQL: a minimal agent setup consistently matches or outperforms more elaborate alternatives, suggesting that the strategies these frameworks employ are less critical in this setting. Data are available at https://github.com/Leolty/Spider2-AIFunc .

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 7 minor

Summary. The paper introduces Spider 2.0-AIFunc, a benchmark of 465 verified instances across 125 real-world databases for evaluating text-to-SQL systems on AI-native SQL queries that incorporate Snowflake Cortex AI functions (e.g., AI_CLASSIFY, AI_SENTIMENT, AI_SIMILARITY). The benchmark is constructed by an agent-based pipeline that rewrites Spider2-Snow tasks to include AI function calls, with a multi-pass determinism verification protocol (35+ executions across four passes, plus 30 additional executions across three time windows). The authors evaluate ten state-of-the-art LLMs, finding proprietary models reach 67–70% execution accuracy while open-source models reach 44.9–58.1%, and compare four agent frameworks, finding that traditional text-to-SQL agent frameworks do not outperform a minimal baseline. The paper is well-motivated, addresses a genuine gap in existing benchmarks, and provides a thorough evaluation methodology with co-execution of gold and predicted SQL within the same time window.

Significance. This work makes a timely and substantive contribution by identifying and filling a real gap in text-to-SQL evaluation: no existing benchmark covers AI-native SQL functions that are now deployed on major cloud data platforms. The determinism verification protocol is thorough and addresses a genuine technical challenge (LLM-backed SQL functions producing non-deterministic outputs even at temperature zero). The co-execution evaluation methodology (Eq. 1) is a sound solution to the problem of drifting gold results. The benchmark is released with code and data on GitHub and HuggingFace, supporting reproducibility. The error analysis (§4.4) with stratified sampling and a defined taxonomy (C1–C5) adds diagnostic value beyond headline accuracy numbers. The per-function-type breakdowns and function selection accuracy metric (Appendix B) provide useful additional signal.

major comments (2)
  1. §4.2, Table 2: The claim that agent frameworks 'do not transfer effectively to AI-native SQL workflows' is the paper's most prominent comparative finding, yet the evidence is based on differences of 0.4pp (Spider-Agent 69.0% vs AutoLink 68.6%) and 0.5pp (vs ReFoRCE 68.5%) with a single agent trajectory per task at temperature 1 (§4.1) and no variance estimates. With 465 instances and stochastic agent behavior, these gaps are well within expected sampling noise. The paper should either (a) soften the claim to acknowledge that the evaluation cannot distinguish 'no benefit' from 'small benefit masked by noise,' or (b) provide variance estimates (e.g., via multiple trajectories or bootstrap confidence intervals) to support the current framing. The DSR-SQL deficit (6.8pp) is more informative but may reflect framework-specific issues rather than a general transfer failure. As stated, the claim
  2. §3.2 and §7: The benchmark is constructed by Claude Opus 4.5 and the top evaluated models are Claude Opus 4.6 (70.3%) and Claude Sonnet 4.6 (69.0%). The paper acknowledges this construction-evaluation coupling in §7 as a limitation but does not discuss its potential consequences concretely. If the construction model's rewriting patterns, instruction phrasing, or AI-function parameterization conventions align more closely with Claude models' priors, the proprietary-vs-open-source gap could be partially inflated. The paper would benefit from either (a) a brief discussion of what specific biases might arise and how they would affect the rankings, or (b) a sensitivity check, such as having a different construction model rewrite a subset of instances and re-evaluating. Without any such analysis, the reader cannot assess the magnitude of this risk to the paper's three main findings.
minor comments (7)
  1. §3.2: The construction pipeline uses Claude Opus 4.5 with a maximum of 15 interaction rounds, but no statistics are reported on how many rounds were actually used on average, how many instances required repair during verification, or what fraction of candidate instances were ultimately filtered out at each stage. Reporting these pipeline statistics (e.g., a table showing: 513 main-round candidates → N after verification → 325 released; 227 diversity-round candidates → M after verification → 140 released) would strengthen the paper.
  2. §4.4, Figure 3: The C5 annotation issue rate (23% of S1 instances, which is 7/30 = 1.5% of total) is a real but lower-stakes concern. The paper is transparent about this in §4.4 and §7. A brief note estimating the upper bound on how much the headline accuracy numbers could change if all C5 instances were fixed or removed would make the analysis more complete.
  3. Table 1: The † footnote for GPT-5.4 references Appendix A, which is good. The ablation in Table 4 showing a 10pp improvement from the parameter optimization patch is important context. The paper should note in the main text (not just the appendix) that without the patch, GPT-5.4 would drop from 63.0% to 52.9%, falling below Gemini 3 Flash (60.9%), as this affects the model ranking.
  4. §3.3: The verification protocol allows the agent to modify evaluation configuration (e.g., whether row order should be compared). The paper should clarify what fraction of instances had their evaluation configuration modified during verification, and whether any instances had row-order comparison disabled, as this affects the strictness of the evaluation.
  5. Appendix B.2, Table 6: The AI_SENTIMENT category has only 22 instances, and accuracy on this category shows high variance across models (e.g., Qwen 3.5 Plus at 68.2% vs GPT-5.4 at 36.4%). A note acknowledging that per-function-type results for AI_SENTIMENT are based on small samples and should be interpreted cautiously would be appropriate.
  6. §2.2: The statement that AI function inference is run at temperature zero is attributed to Snowflake documentation. Given that the entire verification protocol is motivated by non-determinism at temperature zero, citing the specific evidence (He & Lab, 2025; Atil et al., 2025) more explicitly in this section, rather than only mentioning that 'prior work has shown' non-determinism, would clarify why 35+ executions are necessary.
  7. The paper uses model names with version numbers (e.g., Claude Opus 4.6, Gemini 3.1 Pro, GPT-5.4) that may not be widely recognizable at the time of review. Including a brief table mapping model names to their release dates or API identifiers in an appendix would improve reproducibility.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the thorough and constructive review. Both major comments identify legitimate methodological concerns. We agree that (1) the agent framework comparison claim is overstated given the absence of variance estimates, and (2) the construction-evaluation coupling deserves more concrete discussion. We will revise the manuscript accordingly.

read point-by-point responses

  1. Referee: §4.2, Table 2: The claim that agent frameworks 'do not transfer effectively to AI-native SQL workflows' is based on differences of 0.4pp and 0.5pp with a single agent trajectory per task at temperature 1 and no variance estimates. The paper should either soften the claim or provide variance estimates.

    Authors: The referee is correct. With 465 instances, single trajectories at temperature 1, and gaps of 0.4–0.5pp between Spider-Agent and AutoLink/ReFoRCE, we cannot statistically distinguish 'no benefit' from 'small benefit masked by noise.' A simple binomial check confirms this: at 69.0% accuracy (321/465), the 95% confidence interval is approximately ±4.2pp, far wider than the observed differences. We will revise the manuscript in two ways. First, we will soften the framing in §4.2 and the abstract to state that the three frameworks perform comparably, with the minimal Spider-Agent baseline matching but not exceeding the alternatives—a finding that elaborate framework complexity does not confer a measurable advantage on this benchmark. Second, we will add bootstrap confidence intervals for all entries in Table 2 to make the uncertainty explicit. We note that the DSR-SQL deficit (6.8pp) is large enough to be meaningful even under noise, but as the referee observes, this may reflect framework-specific issues rather than a general transfer failure; we will add this qualification. We agree that multiple trajectories per task would provide a stronger evaluation, and we acknowledge in §7 that cost constraints prevented this. Running 465 tasks × 4 frameworks × multiple trajectories, each involving up to 50 agent rounds with Snowflake AI function calls, is substantial. We will add bootstrap CIs as a practical middle ground and note multi-trajectory evaluation as a priority for future work. revision: partial

  2. Referee: §3.2 and §7: The benchmark is constructed by Claude Opus 4.5 and the top evaluated models are Claude Opus 4.6 (70.3%) and Claude Sonnet 4.6 (69.0%). The paper acknowledges this coupling in §7 but does not discuss its potential consequences concretely. The paper should discuss specific biases or provide a sensitivity check.

    Authors: The referee raises a valid concern that we underaddressed. We will expand §7 to discuss the specific biases that could arise. The most concrete risk is that Claude Opus 4.5's rewriting patterns—phrasing conventions for instructions, parameterization styles for AI function calls, and choices about which aspects of the SQL to transform—may align more closely with the priors of Claude-family evaluation models than with open-source models. If this bias exists, it could partially inflate the proprietary-vs-open-source gap. However, we note three mitigating factors that we will also include in the revised discussion. First, the construction pipeline starts from existing Spider2-Snow tasks with fixed databases, schemas, and reference SQL; the agent rewrites queries to incorporate AI functions but does not create the underlying schema or base query logic, limiting the surface area for construction-model-specific bias. Second, the instructions are refined to make AI function usage explicit (e.g., specifying exact label sets), which reduces the degree to which model-specific phrasing conventions matter for task interpretation. Third, the top non-Claude proprietary model, Gemini 3.1 Pro (67.1%), performs comparably to the Claude models, suggesting that the gap is not driven solely by Claude-family familiarity with construction patterns. We agree that a sensitivity check using a different construction model would be the strongest test, and we will note this as an important direction for future work. We cannot complete such an experiment within the revision timeframe because it would require re-running the full construction pipeline with a different model, re-verifying all instances for determinism, and re-evaluating all models—a process that took months for the original benchmark. revision: partial

Circularity Check

0 steps flagged

No significant circularity: benchmark construction and model evaluation are independent, with one minor self-citation that is not load-bearing

full rationale

The paper constructs a benchmark (Spider 2.0-AIFunc) using an agent pipeline powered by Claude Opus 4.5, then evaluates multiple models including Claude variants on that benchmark. The reader flags a potential construction-evaluation coupling since the same model family is used for both. However, this is not circularity in the sense of a derivation reducing to its inputs. The benchmark's gold SQL queries are derived from Spider2-Snow source tasks (an existing external benchmark by Lei et al., 2025) and verified through multi-pass execution stability checks, not defined by the construction model's outputs in a way that would make evaluation results tautological. The evaluation metric (execution accuracy, Eq. 1-2) compares predicted SQL results against gold SQL results executed in the same time window — this is an external, falsifiable comparison, not a self-referential definition. The paper's central claims (proprietary models reach 67-70%, open-source 44.9-58.1%, agent frameworks do not transfer) are empirical findings against an independently constructed benchmark, not predictions that reduce to fitted inputs. The self-citation to Spider2-Snow (Lei et al., 2025) provides the source benchmark but is not load-bearing for the novel contribution: the AI-function rewriting and verification pipeline is new. The authors transparently acknowledge in §7 that the construction model may introduce biases in wording style or rewrite patterns, which is a legitimate methodological limitation but not a circularity. The GPT-5.4 patch (Appendix A) is an evaluation adjustment, not a circular dependency. No step in the paper's derivation chain reduces to its inputs by construction.

Axiom & Free-Parameter Ledger

6 free parameters · 4 axioms · 0 invented entities

The benchmark introduces no new physical entities, particles, or theoretical constructs. The free parameters are operational settings (rounds, timeouts, tolerances) chosen for practical reasons. The axioms are domain assumptions about determinism, representativeness, and evaluation methodology. The construction model (Claude Opus 4.5) is an existing system, not an invented entity.

free parameters (6)
  • Max interaction rounds (construction) = 15
    Chosen for the construction agent pipeline; not derived from data.
  • Max interaction rounds (evaluation) = 50
    Chosen for the evaluation agent setup; not derived from data.
  • Per-query execution timeout = 120 seconds
    Chosen as the timeout threshold for SQL execution during evaluation.
  • Float comparison tolerance = 0.01
    Used for comparing float values in execution results; a design choice.
  • Temperature (evaluation) = 1
    Set uniformly for all evaluated models during evaluation.
  • Reasoning effort = medium
    Set uniformly for models that support this parameter.
axioms (4)
  • domain assumption AI function outputs at temperature zero are sufficiently stable for execution-based evaluation after multi-round verification.
    Invoked in §2.2 and §3.3; the entire verification protocol is built on this premise. The paper provides empirical evidence (465 instances pass) but no theoretical guarantee.
  • domain assumption Spider2-Snow source tasks are representative of real-world enterprise SQL workflows.
    Inherited from Lei et al. (2025); the benchmark's real-world validity depends on this.
  • domain assumption The column-subset matching criterion (Eq. 2) is an appropriate correctness measure for AI-native SQL.
    Invoked in §3.5; allows predicted results to contain extra columns, which is a lenient but standard choice from Spider2-Snow.
  • ad hoc to paper A single agent trajectory per model-task pair provides a meaningful accuracy estimate.
    Invoked in §4.1; the authors acknowledge this is a cost-driven limitation in §7.

pith-pipeline@v1.1.0-glm · 24493 in / 3134 out tokens · 453242 ms · 2026-07-08T12:36:56.307856+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

66 extracted references · 66 canonical work pages · 14 internal anchors

  1. [1]

    Introducing claude opus 4.5

    Anthropic . Introducing claude opus 4.5. https://www.anthropic.com/news/claude-opus-4-5, November 2025. Published Nov 24, 2025

    work page 2025

  2. [2]

    Introducing claude opus 4.6

    Anthropic . Introducing claude opus 4.6. https://www.anthropic.com/news/claude-opus-4-6, February 2026 a . Published Feb 5, 2026

    work page 2026

  3. [3]

    Introducing claude sonnet 4.6

    Anthropic . Introducing claude sonnet 4.6. https://www.anthropic.com/news/claude-sonnet-4-6, February 2026 b . Published Feb 17, 2026

    work page 2026

  4. [4]

    Non-Determinism of "Deterministic" LLM Settings

    Berk Atil, Sarp Aykent, Alexa Chittams, Lisheng Fu, Rebecca J. Passonneau, Evan Radcliffe, Guru Rajan Rajagopal, Adam Sloan, Tomasz Tudrej, Ferhan Ture, Zhe Wu, Lixinyu Xu, and Breck Baldwin. Non-determinism of "deterministic" llm settings, 2025. URL https://arxiv.org/abs/2408.04667

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  5. [5]

    APEX-SQL: Talking to the data via Agentic Exploration for Text-to-SQL

    Bowen Cao, Weibin Liao, Yushi Sun, Dong Fang, Haitao Li, and Wai Lam. Apex-sql: Talking to the data via agentic exploration for text-to-sql, 2026. URL https://arxiv.org/abs/2602.16720

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  6. [6]

    BEAVER: An Enterprise Benchmark for Text-to-SQL

    Peter Baile Chen, Fabian Wenz, Yi Zhang, Devin Yang, Justin Choi, Nesime Tatbul, Michael Cafarella, Çağatay Demiralp, and Michael Stonebraker. Beaver: An enterprise benchmark for text-to-sql, 2025. URL https://arxiv.org/abs/2409.02038

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  7. [7]

    Enrich data using ai functions

    Databricks . Enrich data using ai functions. https://docs.databricks.com/aws/en/large-language-models/ai-functions, 2026. Accessed Apr 1, 2026

    work page 2026

  8. [8]

    DeepSeek-AI, Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenhao Xu, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Erhang Li, Fangqi Zhou, Fangyun Lin, Fucong Dai, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Hanwei Xu, Ha...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  9. [9]

    ReFoRCE: A Text-to-SQL Agent with Self-Refinement, Consensus Enforcement, and Column Exploration

    Minghang Deng, Ashwin Ramachandran, Canwen Xu, Lanxiang Hu, Zhewei Yao, Anupam Datta, and Hao Zhang. Reforce: A text-to-sql agent with self-refinement, consensus enforcement, and column exploration, 2025. URL https://arxiv.org/abs/2502.00675

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  10. [10]

    GLM-5: from Vibe Coding to Agentic Engineering

    GLM-5-Team, :, Aohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, Bin Chen, Da Yin, Chendi Ge, Chenghua Huang, Chengxing Xie, Chenzheng Zhu, Congfeng Yin, Cunxiang Wang, Gengzheng Pan, Hao Zeng, Haoke Zhang, Haoran Wang, Huilong Chen, Jiajie Zhang, Jian Jiao, Jiaqi Guo, Jingsen Wang, Jingzhao Du, Jinzhu Wu, Kedong Wang, Lei Li, Lin Fan, Lucen Zho...

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  11. [11]

    Gemini 3 flash: frontier intelligence built for speed

    Google . Gemini 3 flash: frontier intelligence built for speed. https://blog.google/products-and-platforms/products/gemini/gemini-3-flash/, December 2025. Published Dec 17, 2025

    work page 2025

  12. [12]

    Gemini 3.1 pro: A smarter model for your most complex tasks

    Google . Gemini 3.1 pro: A smarter model for your most complex tasks. https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-3-1-pro/, February 2026. Published Feb 19, 2026

    work page 2026

  13. [13]

    Introduction to ai in bigquery

    Google Cloud . Introduction to ai in bigquery. https://docs.cloud.google.com/bigquery/docs/ai-introduction, 2026. Accessed Apr 1, 2026

    work page 2026

  14. [14]

    Text-to-sql as dual-state reasoning: Integrating adaptive context and progressive generation, 2025

    Zhifeng Hao, Qibin Song, Ruichu Cai, and Boyan Xu. Text-to-sql as dual-state reasoning: Integrating adaptive context and progressive generation, 2025. URL https://arxiv.org/abs/2511.21402

    work page arXiv 2025

  15. [15]

    Defeating nondeterminism in llm inference

    Horace He and Thinking Machines Lab. Defeating nondeterminism in llm inference. Thinking Machines Lab: Connectionism, 2025. doi:10.64434/tml.20250910. https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/

    work page doi:10.64434/tml.20250910 2025

  16. [16]

    KaggleDBQA: Realistic Evaluation of Text-to-SQL Parsers

    Chia-Hsuan Lee, Oleksandr Polozov, and Matthew Richardson. Kaggledbqa: Realistic evaluation of text-to-sql parsers, 2021. URL https://arxiv.org/abs/2106.11455

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  17. [17]

    Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows

    Fangyu Lei, Jixuan Chen, Yuxiao Ye, Ruisheng Cao, Dongchan Shin, Hongjin Su, Zhaoqing Suo, Hongcheng Gao, Wenjing Hu, Pengcheng Yin, Victor Zhong, Caiming Xiong, Ruoxi Sun, Qian Liu, Sida Wang, and Tao Yu. Spider 2.0: Evaluating language models on real-world enterprise text-to-sql workflows, 2025. URL https://arxiv.org/abs/2411.07763

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  18. [18]

    Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Rongyu Cao, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin C. C. Chang, Fei Huang, Reynold Cheng, and Yongbin Li. Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls, 2023. URL https://arxiv.org...

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  19. [19]

    Agent bain vs

    Yue Li, Ran Tao, Derek Hommel, Yusuf Denizay Dönder, Sungyong Chang, David Mimno, and Unso Eun Seo Jo. Agent bain vs. agent mckinsey: A new text-to-sql benchmark for the business domain, 2026. URL https://arxiv.org/abs/2510.07309

    work page arXiv 2026

  20. [20]

    A survey of text-to-sql in the era of llms: Where are we, and where are we going?, 2025

    Xinyu Liu, Shuyu Shen, Boyan Li, Peixian Ma, Runzhi Jiang, Yuxin Zhang, Ju Fan, Guoliang Li, Nan Tang, and Yuyu Luo. A survey of text-to-sql in the era of llms: Where are we, and where are we going?, 2025. URL https://arxiv.org/abs/2408.05109

    work page arXiv 2025

  21. [21]

    Minimax m2.5: Built for real-world productivity

    MiniMax . Minimax m2.5: Built for real-world productivity. https://www.minimax.io/news/minimax-m25, February 2026. Published Feb 11, 2026

    work page 2026

  22. [22]

    Introducing gpt-5.4

    OpenAI . Introducing gpt-5.4. https://openai.com/index/introducing-gpt-5-4/, March 2026. Published Mar 18, 2026

    work page 2026

  23. [24]

    Semantic Operators: A Declarative Model for Rich, AI-based Data Processing

    Liana Patel, Siddharth Jha, Melissa Pan, Harshit Gupta, Parth Asawa, Carlos Guestrin, and Matei Zaharia. Semantic operators: A declarative model for rich, ai-based data processing, 2025. URL https://arxiv.org/abs/2407.11418

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  24. [25]

    Multilingual text-to-sql: Benchmarking the limits of language models with collaborative language agents, 2025

    Khanh Trinh Pham, Thu Huong Nguyen, Jun Jo, Quoc Viet Hung Nguyen, and Thanh Tam Nguyen. Multilingual text-to-sql: Benchmarking the limits of language models with collaborative language agents, 2025. URL https://arxiv.org/abs/2509.24405

    work page arXiv 2025

  25. [26]

    Qwen3.5: Towards native multimodal agents

    Qwen Team . Qwen3.5: Towards native multimodal agents. https://qwen.ai/blog?id=qwen3.5, February 2026. Published Feb 27, 2026

    work page 2026

  26. [27]

    On the Potential of Lexico-logical Alignments for Semantic Parsing to SQL Queries

    Tianze Shi, Chen Zhao, Jordan Boyd-Graber, Hal Daumé III, and Lillian Lee. On the potential of lexico-logical alignments for semantic parsing to sql queries, 2020. URL https://arxiv.org/abs/2010.11246

    work page internal anchor Pith review Pith/arXiv arXiv 2020

  27. [28]

    Snowflake cortex ai functions (including llm functions)

    Snowflake . Snowflake cortex ai functions (including llm functions). https://docs.snowflake.com/en/user-guide/snowflake-cortex/aisql, 2026. Accessed Apr 1, 2026

    work page 2026

  28. [29]

    Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao, S. H. Cai, Yuan Cao, Y. Charles, H. S. Che, Cheng Chen, Guanduo Chen, Huarong Chen, Jia Chen, Jiahao Chen, Jianlong Chen, Jun Chen, Kefan Chen, Liang Chen, Ruijue Chen, Xinhao Chen, Yanru Chen, Yanxu Chen, Yicun Chen, Yimin Chen, Yingjiang Chen, Yuankun Chen, Yujie Chen, Yutian Chen, Zhirong Chen, Ziwei Chen...

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  29. [30]

    Autolink: Autonomous schema exploration and expansion for scalable schema linking in text-to-sql at scale, 2025

    Ziyang Wang, Yuanlei Zheng, Zhenbiao Cao, Xiaojin Zhang, Zhongyu Wei, Pei Fu, Zhenbo Luo, Wei Chen, and Xiang Bai. Autolink: Autonomous schema exploration and expansion for scalable schema linking in text-to-sql at scale, 2025. URL https://arxiv.org/abs/2511.17190

    work page arXiv 2025

  30. [31]

    Parameswaran

    Lindsey Linxi Wei, Shreya Shankar, Sepanta Zeighami, Yeounoh Chung, Fatma Ozcan, and Aditya G. Parameswaran. Multi-objective agentic rewrites for unstructured data processing, 2026. URL https://arxiv.org/abs/2512.02289

    work page arXiv 2026

  31. [32]

    Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task

    Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task, 2019. URL https://arxiv.org/abs/1809.08887

    work page internal anchor Pith review Pith/arXiv arXiv 2019

  32. [33]

    Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning

    Victor Zhong, Caiming Xiong, and Richard Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning, 2017. URL https://arxiv.org/abs/1709.00103

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  33. [34]

    2017 , eprint=

    Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning , author=. 2017 , eprint=

    work page 2017

  34. [35]

    2019 , eprint=

    Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task , author=. 2019 , eprint=

    work page 2019

  35. [36]

    2023 , eprint=

    Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs , author=. 2023 , eprint=

    work page 2023

  36. [37]

    2025 , eprint=

    A Survey of Text-to-SQL in the Era of LLMs: Where are we, and where are we going? , author=. 2025 , eprint=

    work page 2025

  37. [38]

    2025 , eprint=

    BEAVER: An Enterprise Benchmark for Text-to-SQL , author=. 2025 , eprint=

    work page 2025

  38. [39]

    2025 , eprint=

    Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows , author=. 2025 , eprint=

    work page 2025

  39. [40]

    Deterministic

    Non-Determinism of "Deterministic" LLM Settings , author=. 2025 , eprint=

    work page 2025

  40. [41]

    and Harman, Mark and Wang, Meng , year=

    Ouyang, Shuyin and Zhang, Jie M. and Harman, Mark and Wang, Meng , year=. An Empirical Study of the Non-Determinism of. ACM Transactions on Software Engineering and Methodology , publisher=. doi:10.1145/3697010 , number=

    work page doi:10.1145/3697010

  41. [42]

    Thinking Machines Lab: Connectionism , year =

    Horace He and Thinking Machines Lab , title =. Thinking Machines Lab: Connectionism , year =

    work page

  42. [43]

    2025 , eprint=

    ReFoRCE: A Text-to-SQL Agent with Self-Refinement, Consensus Enforcement, and Column Exploration , author=. 2025 , eprint=

    work page 2025

  43. [44]

    2025 , eprint=

    Text-to-SQL as Dual-State Reasoning: Integrating Adaptive Context and Progressive Generation , author=. 2025 , eprint=

    work page 2025

  44. [45]

    2025 , eprint=

    AutoLink: Autonomous Schema Exploration and Expansion for Scalable Schema Linking in Text-to-SQL at Scale , author=. 2025 , eprint=

    work page 2025

  45. [46]

    2026 , howpublished =

    Snowflake Cortex AI Functions (including LLM functions) , author =. 2026 , howpublished =

    work page 2026

  46. [47]

    2026 , howpublished =

    Introduction to AI in BigQuery , author =. 2026 , howpublished =

    work page 2026

  47. [48]

    2026 , howpublished =

    Enrich data using AI Functions , author =. 2026 , howpublished =

    work page 2026

  48. [49]

    2025 , month = nov, howpublished =

    Introducing Claude Opus 4.5 , author =. 2025 , month = nov, howpublished =

    work page 2025

  49. [50]

    2026 , month = feb, howpublished =

    Introducing Claude Opus 4.6 , author =. 2026 , month = feb, howpublished =

    work page 2026

  50. [51]

    2026 , month = feb, howpublished =

    Introducing Claude Sonnet 4.6 , author =. 2026 , month = feb, howpublished =

    work page 2026

  51. [52]

    2026 , month = feb, howpublished =

    Gemini 3.1 Pro: A smarter model for your most complex tasks , author =. 2026 , month = feb, howpublished =

    work page 2026

  52. [53]

    2025 , month = dec, howpublished =

    Gemini 3 Flash: frontier intelligence built for speed , author =. 2025 , month = dec, howpublished =

    work page 2025

  53. [54]

    2026 , month = feb, howpublished =

    Qwen3.5: Towards Native Multimodal Agents , author =. 2026 , month = feb, howpublished =

    work page 2026

  54. [55]

    2025 , month = nov, howpublished =

    GPT-5.1: A smarter, more conversational ChatGPT , author =. 2025 , month = nov, howpublished =

    work page 2025

  55. [56]

    2026 , month = mar, howpublished =

    Introducing GPT-5.4 , author =. 2026 , month = mar, howpublished =

    work page 2026

  56. [57]

    2026 , month = feb, howpublished =

    MiniMax M2.5: Built for Real-World Productivity , author =. 2026 , month = feb, howpublished =

    work page 2026

  57. [58]

    2026 , eprint=

    Kimi K2.5: Visual Agentic Intelligence , author=. 2026 , eprint=

    work page 2026

  58. [59]

    2025 , eprint=

    DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models , author=. 2025 , eprint=

    work page 2025

  59. [60]

    2026 , eprint=

    GLM-5: from Vibe Coding to Agentic Engineering , author=. 2026 , eprint=

    work page 2026

  60. [61]

    2021 , eprint=

    KaggleDBQA: Realistic Evaluation of Text-to-SQL Parsers , author=. 2021 , eprint=

    work page 2021

  61. [62]

    2020 , eprint=

    On the Potential of Lexico-logical Alignments for Semantic Parsing to SQL Queries , author=. 2020 , eprint=

    work page 2020

  62. [63]

    2025 , eprint=

    Multilingual Text-to-SQL: Benchmarking the Limits of Language Models with Collaborative Language Agents , author=. 2025 , eprint=

    work page 2025

  63. [64]

    2025 , eprint=

    Semantic Operators: A Declarative Model for Rich, AI-based Data Processing , author=. 2025 , eprint=

    work page 2025

  64. [65]

    2026 , eprint=

    APEX-SQL: Talking to the data via Agentic Exploration for Text-to-SQL , author=. 2026 , eprint=

    work page 2026

  65. [66]

    2026 , eprint=

    Multi-Objective Agentic Rewrites for Unstructured Data Processing , author=. 2026 , eprint=

    work page 2026

  66. [67]

    Agent McKinsey: A New Text-to-SQL Benchmark for the Business Domain , author=

    Agent Bain vs. Agent McKinsey: A New Text-to-SQL Benchmark for the Business Domain , author=. 2026 , eprint=

    work page 2026