Pith. sign in

REVIEW 2 major objections 4 minor 88 references

An automated loop that invents and evolves failure patterns finds far more latent Text-to-SQL brittleness than fixed expert rules.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 23:37 UTC pith:C2PUINO3

load-bearing objection Solid automated red-team loop for Text-to-SQL with real VER gains over static expert rules; Checker validity is the main soft spot, not a collapse of the claim. the 2 major comments →

arxiv 2607.03833 v1 pith:C2PUINO3 submitted 2026-07-04 cs.CL cs.AI

Beyond Static Rules: Automated Discovery of Latent Vulnerabilities in Text-to-SQL

Hanqing Wang , Yongdong Chi , Jian Yang , Lei Yang , Jiehui Zhao , Yun Chen , Guanhua Chen This is my paper
classification cs.CL cs.AI
keywords Text-to-SQLLLM robustnessvulnerability discoveryadversarial perturbationVulnerability Codexcross-model transfersemantic equivalence checking
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Current Text-to-SQL models look strong on standard benchmarks, yet they still break under small, meaning-preserving changes to the question or schema. Existing diagnostics mostly check a fixed list of expert-written traps, so they cannot systematically hunt for unknown failure modes. This paper introduces SAGE, a closed loop that first invents vulnerability hypotheses for each correctly solved sample, then uses a living Vulnerability Codex of prior failures to craft targeted perturbations, checks that the answer should still be the same, and finally abstracts confirmed failures back into the Codex. On open-source models the method exposes failures on roughly 77% of previously correct BIRD examples and 58% of Spider examples—substantially more than the static-expert baseline—while needing fewer attempts per discovery. The same Codex also transfers across models, and a small fine-tuning set built from the discovered failures begins to harden a different model, sketching a path from diagnosis to repair.

Core claim

SAGE’s evolving Vulnerability Codex, guided by hypothesis generation and semantic compression, systematically uncovers latent Text-to-SQL vulnerabilities that static expert rule sets miss: average Vulnerability Exposure Rates reach 76.98% on BIRD and 58.45% on Spider (versus 50.66% and 39.79% for the expert baseline), the discovered patterns transfer across models, and the resulting adversarial samples can be used for lightweight remediation.

What carries the argument

The Vulnerability Codex—a continuously updated repository of generalized error archetypes that is retrieved to guide new perturbations and refined by abstracting raw failures then compressing near-duplicates—turns one-off discovery into an iterative, self-improving audit loop.

Load-bearing premise

That the automated checker correctly decides when a rewritten question or schema still has the same right answer, so a model mistake is a true robustness failure rather than an accidental change of meaning.

What would settle it

A larger human audit of Checker decisions, especially on relevant-schema rewrites, that shows a high fraction of “valid” failures actually change the gold answer; those cases would inflate VER and collapse the claim that the failures are genuine latent vulnerabilities.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper proposes SAGE, a closed-loop framework that generates vulnerability hypotheses for Text-to-SQL samples, retrieves guidance from an evolving Vulnerability Codex, synthesizes semantic-preserving perturbations over query/schema scopes, verifies failures by execution mismatch, and abstracts validated failures into generalized error archetypes via summarization and embedding-based compression. On initially-correct subsets of BIRD and Spider, SAGE is compared to a static Expert baseline built from Dr.Spider/SpiderSyn/ADVETA rules across Gemma-3, Inf-rl-qwen, and OmniSQL, reporting substantially higher Vulnerability Exposure Rates (average VER 76.98% vs 50.66% on BIRD; 58.45% vs 39.79% on Spider), lower Attempts per Discovery, cross-model Codex transfer gains (Fig. 2), a budgeted GPT-4o transfer result (Table 4), and a small SFT remediation experiment on Qwen2.5-Coder-7B using ~1.5k adversarial samples (Table 3).

Significance. If the validity filter is reliable, the work is a clear advance over static robustness benchmarks for Text-to-SQL: it turns expert heuristics into an adaptive discovery loop, introduces usable metrics (VER, ApD), shows that discovered patterns transfer across models, and takes a first step toward a detect-and-repair reliability loop. Strengths include multi-model evaluation on BIRD/Spider, an explicit Expert baseline, scope and iteration ablations, cross-model transfer, a proprietary-model transfer experiment under budget, and released code. These make the contribution actionable for robustness auditing even if remediation remains preliminary.

major comments (2)
  1. §3.2.2 Eqs. (5)–(6) and Appendix B.6: every VER/ApD number treats IsCorrect=false after ValidCheck as a true robustness defect. The Checker (Qwen3-32B) is only partially validated—89.3% agreement with three annotators on 100 cases and 93.15% GPT-5.1 proxy agreement on 2000 cases, dropping to 87.44% on Relevant Schema. A non-trivial invalid-pass rate can inflate absolute VER, especially under multi-iteration full-scope search that favors hard schema rewrites. Relative SAGE–Expert margins may be partly preserved because both use the same Checker, but the absolute “latent vulnerability” interpretation of Codex transfer and SFT gains remains sensitive to this filter. A larger stratified human audit (by scope and iteration), error-rate-adjusted VER bounds, or a stricter multi-judge gate is needed before the headline rates can be treated as definitive.
  2. §5.2 / Table 3: the remediation claim is load-bearing for the “closing the reliability loop” narrative, yet the experiment is thin—1,580 samples discovered on Gemma-3, SFT only on Qwen2.5-Coder-7B, and no control that matches data volume with non-adversarial or Expert-generated rewrites. The Adv. EX gain (4.37%→9.58%) and Std. EX gain are encouraging, but without those controls it is hard to separate genuine structural repair from generic extra Text-to-SQL supervision. A matched-budget baseline and at least one same-model repair setting would make the remediation claim proportionate to the paper’s framing.
minor comments (4)
  1. Limitations correctly notes coarse Codex updates and limited component ablations; a short sensitivity check on τ (semantic compression) and Top-K retrieval would strengthen §3.3.2.
  2. Figure 3’s “Novel Patterns” slice (~10–11%) is interesting but the taxonomy and inter-annotator process for reclassifying OTHER errors should be stated more explicitly in Appendix A.1.
  3. Table 5 consolidates |D| and search settings well; ensure the main text points readers there when VER denominators are first introduced.
  4. Clarify whether Expert and SAGE share identical hypothesis budgets and stopping rules beyond the static vs evolving Codex distinction, so efficiency (ApD) comparisons remain apples-to-apples.

Circularity Check

1 steps flagged

No significant circularity: VER is defined by external execution mismatch after a validity filter, not by restating the Codex; Codex evolution is adaptive search, not tautological derivation.

specific steps
  1. other [Sec. 3.2-3.3, Eqs. (3)-(9); Fig. 1 closed loop]
    "SAGE references a continuously evolving Vulnerability Codex to design targeted perturbations, thereby iteratively verifying and documenting potential defects. ... Ej = RetrieveTop5(Me, hi,k, Vt-1) ... xj,m = SampleGenerate(Mg, Cj, sm, Ej) ... Vt = SemanticComp(Me, Vt-1 ∪ V(t)_new, τ)"

    Past abstracted failures are retrieved to guide generation of new candidates. This is self-referential search (the system uses its own prior discoveries as experience), but the success metric remains external execution mismatch after ValidCheck, not identity with the Codex. Not self-definitional or fitted-as-prediction; mild closed-loop self-reference only.

full rationale

SAGE's central claims (higher VER than Expert, Codex transfer, remediation via SFT) rest on an operational pipeline: start from samples where Mt already succeeds (Exec(Mt(qi,si))=y*_i), generate hypotheses, retrieve from Vt-1, produce xj,m, require ValidCheck(Mc,...), then flag a vulnerability only if Exec(Mt(xj,m)) != y*_i (Eqs. 1-6, Sec. 3.1-3.2). That failure criterion is external (database execution on held-out schemas), not definitionally equal to the Codex contents. The Codex is initialized from prior expert literature (Dr.Spider, SpiderSyn, ADVETA) and then updated by abstracting observed failures (Eqs. 7-9); using past failures to guide later search is adaptive exploration, not circular derivation of the reported rates. The Expert baseline uses the same evaluation harness without Codex evolution, so relative VER/ApD margins are not forced by construction. Mild self-reference exists (LLM agents generate hypotheses and abstract failures that later guide the same agents), but this is ordinary closed-loop search, not self-definitional prediction or fitted-input-as-prediction. Checker reliability is a correctness/validity risk (Appendix B.6), not a circularity reduction. Score 1 for that mild self-reference only; no load-bearing step reduces the main claims to their inputs by definition.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 3 invented entities

The central empirical claims rest on standard Text-to-SQL evaluation assumptions plus several framework hyperparameters and the invented Codex/agent machinery. No physical constants; free parameters are search/compression knobs. Invented entities are methodological constructs, not new physical objects.

free parameters (5)
  • max iterations T
    Search budget; main results use T=3; Table 6 shows VER rises and ApD falls as T increases, so headline rates depend on this choice.
  • hypotheses per sample K
    K=3 initial hypotheses per sample controls exploration breadth and cost.
  • semantic compression threshold τ
    τ=0.1 decides when archetypes merge; shapes Codex diversity and retrieval.
  • Top-5 Codex retrieval
    Fixed retrieval depth for experience-guided generation.
  • LLM sampling settings (e.g. Qwen3-32B temp=0.6, top_k=20, top_p=0.95)
    Stochastic generation of hypotheses and perturbations; affects discovered set.
axioms (5)
  • domain assumption A vulnerability is robustness brittleness under semantic-preserving perturbations of query/schema, measured by execution mismatch, not security exploits.
    Stated in footnote 1 and problem formulation §3.1; defines the entire evaluation target.
  • domain assumption Restricting evaluation to the initially-correct subset D isolates latent fragility rather than base incompetence.
    §3.1 and §4.1; VER is defined only on this subset.
  • domain assumption SQL execution result equality is a sufficient ground truth for correctness of Text-to-SQL outputs.
    Standard EX/VES practice used throughout Tables 1–4.
  • domain assumption Expert-curated patterns from Dr.Spider, SpiderSyn, and ADVETA form a strong static baseline for discovery.
    §4.1 Baselines; comparison fairness depends on this choice of fixed repository.
  • ad hoc to paper Embedding similarity (Euclidean distance below τ) is an adequate redundancy criterion for merging error archetypes.
    §3.3.2 Semantic Compression; not derived from first principles.
invented entities (3)
  • Vulnerability Codex no independent evidence
    purpose: Dynamic repository of generalized error archetypes that guides subsequent perturbations and is refined by semantic compression.
    Core invented knowledge structure of SAGE; independent evidence is empirical transfer and discovery gains, not external measurement.
  • SAGE closed-loop (Generator / Checker / Summarizer agents + Φdisc/Φevol) no independent evidence
    purpose: Autonomous hypothesis-driven probing and abstraction of Text-to-SQL failures.
    Methodological system introduced in §3; validated only by the paper’s own experiments.
  • VER and ApD metrics no independent evidence
    purpose: Quantify discovery capability and efficiency on initially-correct instances.
    Paper-defined evaluation quantities (§4.1 Metrics).

pith-pipeline@v1.1.0-grok45 · 22791 in / 3354 out tokens · 29563 ms · 2026-07-11T23:37:16.972380+00:00 · methodology

0 comments
read the original abstract

While Large Language Models (LLMs) have achieved remarkable success in Text-to-SQL tasks, their deployment in real-world environments is hindered by latent reliability issues. Identifying these latent weaknesses is critical for building trustworthy database interfaces, yet current diagnostic approaches rely heavily on static, expert-defined rules, which lack the capability for systematic and automated exploration. To bridge this gap, we propose SAGE (Systematic Automated Guided Exploration), a novel framework designed to autonomously uncover latent failure patterns in LLM-based Text-to-SQL generation. Specifically, SAGE generates vulnerability hypotheses for given samples and references a continuously evolving Vulnerability Codex to design targeted perturbations, thereby iteratively verifying and documenting potential defects. Extensive experiments on state-of-the-art open-source LLMs demonstrate that SAGE uncovers a substantial number of failure cases, highlighting the significant fragility of current models. Furthermore, our analysis reveals that the Vulnerability Codex exhibits strong cross-model transferability, indicating that the discovered patterns represent generalized structural weaknesses. Finally, we explore SAGE's potential for remediation. Although preliminary, lightweight fine-tuning on the generated samples yields promising improvements, suggesting a scalable pathway for closing the reliability loop in future work.

Figures

Figures reproduced from arXiv: 2607.03833 by Guanhua Chen, Hanqing Wang, Jian Yang, Jiehui Zhao, Lei Yang, Yongdong chi, Yun Chen.

Figure 1
Figure 1. Figure 1: The architecture of the proposed automated vulnerability discovery framework, [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Cross-model strategy transfer performance. [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Distribution and hierarchical analysis of failure patterns across Gemma-3, Inf-rl-qwen, and OmniSQL. [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The prompt templates for perturbations ap [PITH_FULL_IMAGE:figures/full_fig_p015_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: The prompt templates for perturbations applied to schema-relevant elements. [PITH_FULL_IMAGE:figures/full_fig_p016_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

88 extracted references · 4 canonical work pages

  1. [1]

    FANNO : Augmenting High-Quality Instruction Data with Open-Sourced LLM s Only

    Zhu, He and Ding, Yifan and Tao, Yicheng and Ruan, Zhiwen and Li, Yixia and Zhang, Wenjia and Chen, Yun and Chen, Guanhua. FANNO : Augmenting High-Quality Instruction Data with Open-Sourced LLM s Only. Findings of the Association for Computational Linguistics: ACL 2025. 2025. doi:10.18653/v1/2025.findings-acl.906

    work page doi:10.18653/v1/2025.findings-acl.906 2025

  2. [2]

    Tag-Instruct: Controlled Instruction Complexity Enhancement through Structure-based Augmentation

    Zhu, He and Ruan, Zhiwen and Su, Junyou and He, Xingwei and Chen, Yun and Zhang, Wenjia and Chen, Guanhua. Tag-Instruct: Controlled Instruction Complexity Enhancement through Structure-based Augmentation. Findings of the Association for Computational Linguistics: ACL 2025. 2025. doi:10.18653/v1/2025.findings-acl.911

    work page doi:10.18653/v1/2025.findings-acl.911 2025

  3. [3]

    BiasScope: Towards Automated Detection of Bias in

    Peng Lai and Zhihao Ou and Yong Wang and Longyue Wang and Jian Yang and Yun Chen and Guanhua Chen , booktitle=. BiasScope: Towards Automated Detection of Bias in. 2026 , url=

    2026

  4. [4]

    Pi- SQL : Enhancing Text-to- SQL with Fine-Grained Guidance from Pivot Programming Languages

    Chi, Yongdong and Wang, Hanqing and Chen, Yun and Yang, Yan and Yang, Jian and Yang, Zonghan and Yan, Xiao and Chen, Guanhua. Pi- SQL : Enhancing Text-to- SQL with Fine-Grained Guidance from Pivot Programming Languages. Findings of the Association for Computational Linguistics: EMNLP 2025. 2025. doi:10.18653/v1/2025.findings-emnlp.1369

    work page doi:10.18653/v1/2025.findings-emnlp.1369 2025

  5. [5]

    Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages=

    Exploring Underexplored Limitations of Cross-Domain Text-to-SQL Generalization , author=. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages=

    2021

  6. [6]

    2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018 , pages=

    Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task , author=. 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018 , pages=. 2018 , organization=

    2018

  7. [7]

    Companion Proceedings of the ACM on Web Conference 2025 , pages=

    SQLord: A Robust Enterprise Text-to-SQL Solution via Reverse Data Generation and Workflow Decomposition , author=. Companion Proceedings of the ACM on Web Conference 2025 , pages=

    2025

  8. [8]

    arXiv preprint arXiv:2403.02951 , year=

    Benchmarking the text-to-sql capability of large language models: A comprehensive evaluation , author=. arXiv preprint arXiv:2403.02951 , year=

    arXiv

  9. [9]

    arXiv preprint arXiv:2503.19786 , year=

    Gemma 3 technical report , author=. arXiv preprint arXiv:2503.19786 , year=

    Pith/arXiv arXiv

  10. [10]

    CoRR , volume =

    Victor Zhong and Caiming Xiong and Richard Socher , title =. CoRR , volume =

  11. [11]

    KaggleDBQA: Realistic Evaluation of Text-to-SQL Parsers , author=. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages=

  12. [12]

    Transfer Learning for Natural Language Processing Workshop , pages=

    Exploring dimensions of generalizability and few-shot transfer for text-to-SQL semantic parsing , author=. Transfer Learning for Natural Language Processing Workshop , pages=. 2023 , organization=

    2023

  13. [13]

    Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    Benchmarking and Improving Text-to-SQL Generation under Ambiguity , author=. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    2023

  14. [14]

    Advances in Neural Information Processing Systems , volume=

    Ambrosia: A benchmark for parsing ambiguous questions into database queries , author=. Advances in Neural Information Processing Systems , volume=

  15. [15]

    Towards Robustness of Text-to-SQL Models against Synonym Substitution , author=. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages=

  16. [16]

    spider: A diagnostic evaluation benchmark towards text-to-sql robustness , author=

    Dr. spider: A diagnostic evaluation benchmark towards text-to-sql robustness , author=. arXiv preprint arXiv:2301.08881 , year=

    Pith/arXiv arXiv

  17. [17]

    Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Towards Robustness of Text-to-SQL Models Against Natural and Realistic Adversarial Table Perturbation , author=. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  18. [18]

    arXiv preprint arXiv:2501.09310 , year=

    A Study of In-Context-Learning-Based Text-to-SQL Errors , author=. arXiv preprint arXiv:2501.09310 , year=

    arXiv

  19. [19]

    arXiv preprint arXiv:2508.15276 , year=

    AmbiSQL: Interactive Ambiguity Detection and Resolution for Text-to-SQL , author=. arXiv preprint arXiv:2508.15276 , year=

    arXiv

  20. [20]

    2025 , url=

    Yue Gong and Chuan Lei and Xiao Qin and Kapil Vaidya and Balakrishnan Murali Narayanaswamy and Tim Kraska , booktitle=. 2025 , url=

    2025

  21. [21]

    arXiv preprint arXiv:2410.21276 , year=

    Gpt-4o system card , author=. arXiv preprint arXiv:2410.21276 , year=

    Pith/arXiv arXiv

  22. [22]

    Natural language engineering , volume=

    Natural language interfaces to databases--an introduction , author=. Natural language engineering , volume=. 1995 , publisher=

    1995

  23. [23]

    Proceedings of the VLDB Endowment , volume=

    Constructing an interactive natural language interface for relational databases , author=. Proceedings of the VLDB Endowment , volume=. 2014 , publisher=

    2014

  24. [24]

    Advances in Neural Information Processing Systems , volume=

    Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls , author=. Advances in Neural Information Processing Systems , volume=

  25. [25]

    arXiv preprint arXiv:2404.14453 , year=

    Epi-sql: Enhancing text-to-sql translation with error-prevention instructions , author=. arXiv preprint arXiv:2404.14453 , year=

    Pith/arXiv arXiv

  26. [26]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Magic: Generating self-correction guideline for in-context text-to-sql , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  27. [27]

    2015 2nd international conference on electrical information and communication technologies (EICT) , pages=

    A rule based approach for NLP based query processing , author=. 2015 2nd international conference on electrical information and communication technologies (EICT) , pages=. 2015 , organization=

    2015

  28. [28]

    arXiv preprint arXiv:2008.04759 , year=

    Hybrid ranking network for text-to-sql , author=. arXiv preprint arXiv:2008.04759 , year=

    Pith/arXiv arXiv 2008

  29. [29]

    2025 , publisher =

    inf-rl-qwen-coder-32b-2746 , author =. 2025 , publisher =

    2025

  30. [30]

    arXiv preprint arXiv:2505.09388 , year=

    Qwen3 technical report , author=. arXiv preprint arXiv:2505.09388 , year=

    Pith/arXiv arXiv

  31. [31]

    arXiv preprint arXiv:2506.05176 , year=

    Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models , author=. arXiv preprint arXiv:2506.05176 , year=

    Pith/arXiv arXiv

  32. [32]

    Dr.Spider: A Diagnostic Evaluation Benchmark towards Text-to-

    Shuaichen Chang and Jun Wang and Mingwen Dong and Lin Pan and Henghui Zhu and Alexander Hanbo Li and Wuwei Lan and Sheng Zhang and Jiarong Jiang and Joseph Lilien and Steve Ash and William Yang Wang and Zhiguo Wang and Vittorio Castelli and Patrick Ng and Bing Xiang , booktitle=. Dr.Spider: A Diagnostic Evaluation Benchmark towards Text-to-. 2023 , url=

    2023

  33. [33]

    Computational Linguistics , volume=

    Ryansql: Recursively applying sketch-based slot fillings for complex text-to-sql in cross-domain databases , author=. Computational Linguistics , volume=. 2021 , publisher=

    2021

  34. [34]

    arXiv preprint arXiv:1711.04436 , year=

    Sqlnet: Generating structured queries from natural language without reinforcement learning , author=. arXiv preprint arXiv:1711.04436 , year=

    Pith/arXiv arXiv

  35. [35]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Resdsql: Decoupling schema linking and skeleton parsing for text-to-sql , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  36. [36]

    Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages=

    TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data , author=. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages=

  37. [37]

    ACM Comput

    Shi, Liang and Tang, Zhengju and Zhang, Nan and Zhang, Xiaotong and Yang, Zhi , title =. ACM Comput. Surv. , month = jun, keywords =. 2025 , publisher =. doi:10.1145/3737873 , note =

    doi:10.1145/3737873 2025

  38. [38]

    Advances in Neural Information Processing Systems , volume=

    Din-sql: Decomposed in-context learning of text-to-sql with self-correction , author=. Advances in Neural Information Processing Systems , volume=

  39. [39]

    2025 , eprint=

    OpenSearch-SQL: Enhancing Text-to-SQL with Dynamic Few-shot and Consistency Alignment , author=. 2025 , eprint=

    2025

  40. [40]

    arXiv preprint arXiv:2505.23838 , year=

    Exploring the Landscape of Text-to-SQL with Large Language Models: Progresses, Challenges and Opportunities , author=. arXiv preprint arXiv:2505.23838 , year=

    Pith/arXiv arXiv

  41. [41]

    arXiv preprint arXiv:2502.11656 , year=

    Uncovering the Impact of Chain-of-Thought Reasoning for Direct Preference Optimization: Lessons from Text-to-SQL , author=. arXiv preprint arXiv:2502.11656 , year=

    Pith/arXiv arXiv

  42. [42]

    2024 International Joint Conference on Neural Networks (IJCNN) , pages=

    Chain-of-Program Prompting with Open-Source Large Language Models for Text-to-SQL , author=. 2024 International Joint Conference on Neural Networks (IJCNN) , pages=. 2024 , organization=

    2024

  43. [43]

    Xia, Hanchen and Jiang, Feng and Deng, Naihao and Wang, Cunxiang and Zhao, Guojiang and Mihalcea, Rada and Zhang, Yue , journal=. r

  44. [44]

    ICLR 2025 Workshop: VerifAI: AI Verification in the Wild , year=

    Reforce: A Text-to-SQL agent with self-refinement, format restriction, and column exploration , author=. ICLR 2025 Workshop: VerifAI: AI Verification in the Wild , year=

    2025

  45. [45]

    arXiv preprint arXiv:2503.02240 , year=

    Omnisql: Synthesizing high-quality text-to-sql data at scale , author=. arXiv preprint arXiv:2503.02240 , year=

    Pith/arXiv arXiv

  46. [46]

    SHARE : An SLM -based Hierarchical Action C or RE ction Assistant for Text-to- SQL

    Qu, Ge and Li, Jinyang and Qin, Bowen and Li, Xiaolong and Huo, Nan and Ma, Chenhao and Cheng, Reynold. SHARE : An SLM -based Hierarchical Action C or RE ction Assistant for Text-to- SQL. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025. doi:10.18653/v1/2025.acl-long.552

    work page doi:10.18653/v1/2025.acl-long.552 2025

  47. [47]

    SQLF orge: Synthesizing Reliable and Diverse Data to Enhance Text-to- SQL Reasoning in LLM s

    Guo, Yu and Jin, Dong and Ye, Shenghao and Chen, Shuangwu and Jianyang, Jianyang and Tan, Xiaobin. SQLF orge: Synthesizing Reliable and Diverse Data to Enhance Text-to- SQL Reasoning in LLM s. Findings of the Association for Computational Linguistics: ACL 2025. 2025. doi:10.18653/v1/2025.findings-acl.443

    doi:10.18653/v1/2025.findings-acl.443 2025

  48. [48]

    Proceedings of the ACM on Management of Data , volume=

    Codes: Towards building open-source language models for text-to-sql , author=. Proceedings of the ACM on Management of Data , volume=. 2024 , publisher=

    2024

  49. [49]

    arXiv preprint arXiv:2502.10739 , year=

    BASE-SQL: A powerful open source Text-To-SQL baseline approach , author=. arXiv preprint arXiv:2502.10739 , year=

    Pith/arXiv arXiv

  50. [50]

    2024 IEEE Calcutta Conference (CALCON) , pages=

    Adversarial text generation with dynamic contextual perturbation , author=. 2024 IEEE Calcutta Conference (CALCON) , pages=. 2024 , organization=

    2024

  51. [51]

    arXiv preprint arXiv:2502.17254 , year=

    Reinforce adversarial attacks on large language models: An adaptive, distributional, and semantic objective , author=. arXiv preprint arXiv:2502.17254 , year=

    Pith/arXiv arXiv

  52. [52]

    arXiv preprint arXiv:2505.12185 , year=

    EVALOOP: Assessing LLM Robustness in Programming from a Self-consistency Perspective , author=. arXiv preprint arXiv:2505.12185 , year=

    arXiv

  53. [53]

    arXiv preprint arXiv:2410.05295 , year=

    Autodan-turbo: A lifelong agent for strategy self-exploration to jailbreak llms , author=. arXiv preprint arXiv:2410.05295 , year=

    Pith/arXiv arXiv

  54. [54]

    arXiv preprint arXiv:2502.17392 , year=

    Emoti-Attack: Zero-Perturbation Adversarial Attacks on NLP Systems via Emoji Sequences , author=. arXiv preprint arXiv:2502.17392 , year=

    Pith/arXiv arXiv

  55. [55]

    arXiv preprint arXiv:2411.08599 , year=

    A preview of xiyan-sql: A multi-generator ensemble framework for text-to-sql , author=. arXiv preprint arXiv:2411.08599 , year=

    Pith/arXiv arXiv

  56. [56]

    arXiv preprint arXiv:2505.19988 , year=

    Automatic Metadata Extraction for Text-to-SQL , author=. arXiv preprint arXiv:2505.19988 , year=

    Pith/arXiv arXiv

  57. [57]

    arXiv preprint arXiv:2507.04701 , year=

    XiYan-SQL: A Novel Multi-Generator Framework For Text-to-SQL , author=. arXiv preprint arXiv:2507.04701 , year=

    Pith/arXiv arXiv

  58. [58]

    Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages=

    Structure-Grounded Pretraining for Text-to-SQL , author=. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages=

    2021

  59. [59]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Synthesizing Text-to-SQL Data from Weak and Strong LLMs , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  60. [60]

    Findings of the Association for Computational Linguistics: EMNLP 2022 , pages=

    Towards Generalizable and Robust Text-to-SQL Parsing , author=. Findings of the Association for Computational Linguistics: EMNLP 2022 , pages=

    2022

  61. [61]

    Proceedings of the 31st International Conference on Computational Linguistics , pages=

    Solid-SQL: Enhanced Schema-linking based In-context Learning for Robust Text-to-SQL , author=. Proceedings of the 31st International Conference on Computational Linguistics , pages=

  62. [62]

    Song, L and others , journal=

  63. [63]

    Tarbell, J and others , journal=

  64. [64]

    Datenbanksysteme f

    Utilising large language models for adversarial attacks in text-to-sql: A perpetrator and victim approach , author=. Datenbanksysteme f. 2025 , organization=

    2025

  65. [65]

    Lin, Meiyu and Zhang, Haichuan and Lao, Jiale and Li, Renyuan and Zhou, Yuanchun and Yang, Carl and Cao, Yang and Tang, Mingjie , journal=

  66. [66]

    Findings of the Association for Computational Linguistics: NAACL 2025 , year=

    Unmasking Database Vulnerabilities: Zero-Knowledge Schema Inference Attacks in Text-to-SQL Systems , author=. Findings of the Association for Computational Linguistics: NAACL 2025 , year=

    2025

  67. [67]

    Towards Robustness of Text-to-

    Gan, Yujian and Chen, Xinyun and Huang, Qiuping and Purver, Matthew and Woodward, John R and Xie, Jinxia and Huang, Pengsheng , booktitle=. Towards Robustness of Text-to-. 2021 , pages=

    2021

  68. [68]

    Lei, Fangyu and Chen, Jixuan and Ye, Yuxiao and Cao, Ruisheng and Shin, Dongchan and Su, Hongjin and Suo, Zhaoqing and Gao, Hongcheng and Hu, Wenjing and Yin, Pengcheng and others , booktitle=

  69. [69]

    Paulus, Anselm and Zharmagambetov, Arman and Guo, Chuan and Amos, Brandon and Tian, Yuandong , journal=

  70. [70]

    Towards Generalizable and Robust Text-to- SQL Parsing

    Gao, Chang and Li, Bowen and Zhang, Wenxuan and Lam, Wai and Li, Binhua and Huang, Fei and Si, Luo and Li, Yongbin. Towards Generalizable and Robust Text-to- SQL Parsing. Findings of the Association for Computational Linguistics: EMNLP 2022. 2022

    2022

  71. [71]

    arXiv preprint arXiv:2202.03286 , year=

    Red teaming language models with language models , author=. arXiv preprint arXiv:2202.03286 , year=

    Pith/arXiv arXiv

  72. [72]

    and Dahl, George E

    Bowman, Samuel R. and Dahl, George E. What Will it Take to Fix Benchmarking in NLU ?. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021

    2021

  73. [73]

    Yuhang Zhang and Zefang Liu and Zhaoran Wang and Jingfeng Yang and Tianyu Zhao and Jian-Guang Lou , year=

  74. [74]

    Towards Robustness of Text-to-

    Pi, Xinyu and Wang, Bing and Gao, Yan and Guo, Jiaqi and Li, Zhoujun and Lou, Jian-Guang , booktitle=. Towards Robustness of Text-to-. 2022 , pages=

    2022

  75. [75]

    NeurIPS , year=

    BIRD: A big bench for large-scale database grounded text-to-SQL , author=. NeurIPS , year=

  76. [76]

    ICLR , year=

    Spider 2.0: Evaluating language models on real-world enterprise text-to-SQL workflows , author=. ICLR , year=

  77. [77]

    2024 , journal=

    Next-generation database interfaces: A survey of LLM-based text-to-SQL , author=. 2024 , journal=

    2024

  78. [78]

    2024 , journal=

    FinStat2SQL: A text-to-SQL pipeline for financial statement analysis , author=. 2024 , journal=

    2024

  79. [79]

    2025 , journal=

    Are Your LLM-based Text-to-SQL Models Secure? Exploring SQL Injection via Backdoor Attacks , author=. 2025 , journal=

    2025

  80. [80]

    ACM Computing Surveys , year=

    A survey on employing large language models for text-to-SQL tasks , author=. ACM Computing Surveys , year=

Showing first 80 references.