Pith. sign in

REVIEW 5 cited by

Dr.Spider: A Diagnostic Evaluation Benchmark towards Text-to-SQL Robustness

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2301.08881 v2 pith:CNWEBXEC submitted 2023-01-21 cs.CL

Dr.Spider: A Diagnostic Evaluation Benchmark towards Text-to-SQL Robustness

Shuaichen Chang , Jun Wang , Mingwen Dong , Lin Pan , Henghui Zhu , Alexander Hanbo Li , Wuwei Lan , Sheng Zhang
show 8 more authors
Jiarong Jiang Joseph Lilien Steve Ash William Yang Wang Zhiguo Wang Vittorio Castelli Patrick Ng Bing Xiang
This is my paper
classification cs.CL
keywords robustnesstext-to-sqlmodelmodelsnaturalbenchmarklanguageperformance
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
read the original abstract

Neural text-to-SQL models have achieved remarkable performance in translating natural language questions into SQL queries. However, recent studies reveal that text-to-SQL models are vulnerable to task-specific perturbations. Previous curated robustness test sets usually focus on individual phenomena. In this paper, we propose a comprehensive robustness benchmark based on Spider, a cross-domain text-to-SQL benchmark, to diagnose the model robustness. We design 17 perturbations on databases, natural language questions, and SQL queries to measure the robustness from different angles. In order to collect more diversified natural question perturbations, we utilize large pretrained language models (PLMs) to simulate human behaviors in creating natural questions. We conduct a diagnostic study of the state-of-the-art models on the robustness set. Experimental results reveal that even the most robust model suffers from a 14.0% performance drop overall and a 50.7% performance drop on the most challenging perturbation. We also present a breakdown analysis regarding text-to-SQL model designs and provide insights for improving model robustness.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. EntSQL: A Benchmark for Grounding Text-to-SQL in Long-Context Enterprise Knowledge

    cs.CL 2026-06 unverdicted novelty 7.0

    EntSQL is a new benchmark with 1,066 examples across five domains where top systems reach only 15.9% accuracy on English inputs when long-form enterprise documents are provided.

  2. Beyond Static Rules: Automated Discovery of Latent Vulnerabilities in Text-to-SQL

    cs.CL 2026-07 conditional novelty 6.5

    An evolving Vulnerability Codex plus hypothesis-driven perturbations exposes latent Text-to-SQL failures in LLMs far better than fixed expert rules, with transferable patterns and early remediation gains.

  3. EntSQL: A Benchmark for Grounding Text-to-SQL in Long-Context Enterprise Knowledge

    cs.CL 2026-06 unverdicted novelty 6.0

    EntSQL benchmarks long-context enterprise Text-to-SQL and finds the best system reaches only 15.9% accuracy on English with full documents.

  4. EntSQL: A Benchmark for Grounding Text-to-SQL in Long-Context Enterprise Knowledge

    cs.CL 2026-06 unverdicted novelty 6.0

    EntSQL shows current systems reach only 15.9% accuracy when Text-to-SQL must ground in long enterprise business documents rather than schema alone.

  5. EGREFINE: An Execution-Grounded Optimization Framework for Text-to-SQL Schema Refinement

    cs.DB 2026-05 unverdicted novelty 6.0

    EGRefine optimizes column renamings via execution-grounded verification and view materialization to recover Text-to-SQL accuracy lost to schema naming issues while guaranteeing query equivalence.