Pith. sign in

REVIEW 2 major objections 6 minor 38 references

Correctness uncertainty for text-to-SQL lives in reasoning-based verification, not sampling agreement or log-probability.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-10 21:04 UTC pith:EF7H7GEU

load-bearing objection Solid empirical ranking study: black-box UQ plateaus ~0.61–0.68 AUROC on hard multi-table SQL while multi-provider verification clears it and enables real abstention; transfer claim is scoped but still the softest part. the 2 major comments →

arxiv 2607.06799 v1 pith:EF7H7GEU submitted 2026-07-07 cs.LG cs.AI

What Predicts Correctness in Text-to-SQL? A Selective-Prediction Study

Robert Richardson This is my paper
classification cs.LG cs.AI
keywords text-to-SQLselective predictionuncertainty quantificationself-consistencyLLM-as-judgeverificationcalibrationcross-schema transfer
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

When a system turns a natural-language question into SQL, it also needs a reliable way to know whether that SQL is correct so it can answer or abstain. On hard multi-table benchmarks, cheap signals that look only at how much a model agrees with itself—string, structural, and execution self-consistency, schema relevance, executability, and even the model’s own log-probability—all plateau in roughly the same band (about 0.61–0.68 AUROC). Verification changes the picture: an independent model that reads the question, schema, and candidate SQL and judges whether the computation matches the request moves past that ceiling, and judges from different providers make different errors. Combining two such judges yields the strongest and best-calibrated score, supporting useful abstention frontiers that self-consistency cannot form. Fine-tuned verifiers work well when the schemas are already known, but they do not transfer to unseen databases; generalizing across schemas currently requires a large frozen reasoning model rather than more fine-tuning.

Core claim

On hard multi-table text-to-SQL, black-box statistical signals and white-box log-probability form a correctness ceiling of roughly 0.61–0.68 AUROC, while LLM-as-judge verification exceeds it (up to about 0.78 alone and 0.82 for a calibrated two-provider ensemble). The pattern holds across two benchmarks, two generators, and two judge providers. Fine-tuned verifiers reach high in-domain AUROC but collapse on leave-one-database-out transfer, so cross-schema correctness prediction currently tracks model scale and reasoning rather than fine-tuning.

What carries the argument

AUROC ranking of candidate SQL correctness, grounded in execution match to a human reference, used to compare black-box self-consistency signals against logic-aware LLM judges and to measure selective-prediction risk–coverage frontiers.

Load-bearing premise

The claim that cross-schema transfer tracks reasoning scale rather than fine-tuning rests on leave-one-database-out tests over a modest number of schemas and generative verifiers no larger than 7B.

What would settle it

Train a larger generative verifier (or one trained on far more diverse schemas) and re-run leave-one-database-out transfer: if transfer AUROC rises to match a strong frozen judge, the open-versus-in-domain deployment split collapses.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper studies which signals predict execution correctness of generated SQL on hard multi-table text-to-SQL (BIRD and Spider), framing the problem as selective prediction. Black-box statistical signals (string, structural, and execution self-consistency, schema relevance, executability) and white-box log-probability occupy a narrow band of roughly 0.61–0.68 AUROC, with string self-consistency strongest at 0.675. LLM-as-judge verifiers clear that band (0.72–0.78 alone; 0.82 for a cross-fit two-provider ensemble with ECE 0.03) and yield usable risk–coverage frontiers that self-consistency cannot match. Fine-tuned encoder and generative verifiers reach ~0.77–0.79 AUROC in-distribution but fall to ~0.66 under leave-one-database-out transfer; scale to 7B, schema diversity, rationale distillation, and cross-benchmark training do not close the gap. The authors conclude that correctness uncertainty lives in reasoning-based signals, with fine-tuned verifiers useful in-domain and large frozen judges preferred for open/cross-schema use.

Significance. If the comparative ranking holds, the paper gives practitioners a clear, execution-grounded map of which UQ signals are worth paying for in text-to-SQL: sampling agreement and log-probability are weak correctness predictors on multi-table data, while logic-aware verification (especially multi-provider ensembles) enables calibrated abstention. The separation of selective prediction from self-correction, the paired-bootstrap AUROC comparisons, ECE, held-out risk–coverage, and LODO transfer ablations are concrete strengths. The work is timely for high-stakes database interfaces and complements agentic self-correction literature by asking when to trust a final query rather than how to improve it. The in-domain vs open-schema deployment split is a useful practical message even if the transfer evidence remains limited in scale.

major comments (2)
  1. [Section 4.4, Table 4, Appendix A, Discussion] Section 4.4, Table 4, Appendix A, and Discussion: the claim that “cross-schema transfer appears to track model scale and reasoning rather than fine-tuning” is only weakly supported by the reported experiments. LODO is run on 8–28 schemas with generative judges up to 7B; the frozen GPT-4o/Claude judges that transfer better are far larger and differently pretrained, so scale, reasoning ability, and model family are confounded. The negative ablations (diversity, distillation, cross-benchmark) are informative but do not isolate the causal claim. Soften the wording to what the data show—current fine-tunes up to 7B fail to transfer on these schema sets—and treat the open-vs-in-domain split as a practical recommendation under present evidence rather than a settled mechanism.
  2. [Section 3, Tables 1–3, Table 6] Section 3 and Tables 1–3: all primary BIRD numbers are on a fixed 800-question slice over 8 databases, scoring the modal query among K=8 samples. The Spider replication and second-generator check reduce the risk that the black-box ceiling is slice-specific, but the absolute AUROC band (0.61–0.68) and the selective-prediction coverages in Table 6 still depend on this design. Please report sensitivity of the ceiling and ensemble AUROC to (i) non-modal selection (e.g., highest-logprob or random sample) and (ii) a second independent BIRD slice or full available multi-table set, so readers can judge how tightly the “ceiling” is tied to the modal-query / fixed-slice protocol.
minor comments (6)
  1. [Table 7, Section 4.6] Table 7: the GROUP BY row is the only feature subset where the verifier underperforms string self-consistency (0.609 vs 0.651). A short discussion of why (e.g., grouping errors that are hard for the judge prompt, or small n) would help readers interpret the exception.
  2. [Figure 1, Section 4.4] Figure 1 is referenced for per-database transfer but the manuscript text does not state how many questions per held-out database or whether error bars are shown; add n per bar and, if possible, bootstrap intervals so the “leads on every schema” claim is visually checkable.
  3. [Section 4.3, Appendix B] Section 4.3 self-correction baseline: one reflection round with GPT-4o-mini is a useful negative control, but the prompt and revision protocol are not in Appendix B. Include them for reproducibility.
  4. [Abstract, Introduction, Tables] Abstract and Introduction use “auroc” in lowercase inconsistently with standard AUROC; pick one capitalization and stick to it. Also fix the spacing glitches such as “0 .68”, “0 .78”, “0 .66” that appear throughout.
  5. [Section 2] Related work cites conformal abstention and schema-linking graph models but does not position the present selective-prediction results against DIN-SQL / MAC-SQL style agentic pipelines beyond the one-round reflection check. A short paragraph clarifying complementarity would help.
  6. [Section 6] Limitations correctly note the lack of an open-weight generator; if space allows, even a small open-weight generator check (or an explicit “left to future work” with expected failure modes) would strengthen the multi-family claim.

Circularity Check

0 steps flagged

No significant circularity: empirical ranking of independent signals against execution-match labels, with cross-fit calibration and LODO transfer.

full rationale

This paper is a comparative empirical study, not a derivation of a first-principles quantity from fitted inputs. Correctness is the external benchmark definition (generated SQL result set matches the human reference). Black-box signals (string/structural/execution self-consistency, schema relevance, executability, log-probability) are computed from samples or model internals without using the correctness label, then ranked by AUROC against that label. LLM judges produce independent scores that are likewise evaluated against execution ground truth; the two-provider ensemble is cross-fit logistic so the combiner is never scored on its own fit data. Fine-tuned verifiers are trained on labeled pairs and evaluated with leave-one-database-out transfer. Thresholds for selective risk are chosen on a calibration half and reported on a disjoint test half. There are no self-definitional identities, no fitted parameters renamed as predictions of the same quantity, no load-bearing self-citations or uniqueness theorems imported from the author, and no ansatz smuggled in via citation. Standard use of Spider/BIRD reference queries as ground truth is ordinary benchmark practice, not circularity under this framework. The derivation chain is self-contained against external execution labels.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 1 invented entities

Empirical ML comparison paper. Load-bearing content is experimental protocol and evaluation definitions rather than new physical entities. Free parameters are sampling and evaluation choices that affect measured AUROC/frontiers; axioms are standard correctness and ranking definitions plus the assumption that the tested model/schema regime supports the transfer conclusion.

free parameters (5)
  • K samples per question
    K=8 at temperature 0.7 defines the self-consistency and modal-query signals; different K could move the black-box ceiling.
  • BIRD evaluation slice size and databases
    Fixed 800 questions over 8 local databases chosen before analysis; all primary BIRD numbers rest on this slice.
  • Cross-fit logistic ensemble / calibration
    Two-provider ensemble and ECE 0.03 depend on logistic combination fit on held-out splits; coefficients are data-fit free parameters of the best reported score.
  • Selective-risk target grid and split
    Empirical frontiers (e.g., 27% coverage at 24% risk) depend on calibration/test half-splits and chosen risk targets (0.20/0.30/0.40).
  • Fine-tune training set size and LoRA setup
    6400 execution-labeled pairs and LoRA on 1.5B/7B models set the in-domain vs transfer numbers; not uniquely determined by theory.
axioms (5)
  • domain assumption A generated query is correct iff its execution result set matches the human reference query on the database.
    Section 1 and Setup; standard Spider/BIRD criterion, but ignores equivalent SQL with different surface forms and any reference errors.
  • domain assumption AUROC of a score against execution correctness is the right primary ranking metric for comparing uncertainty signals.
    Stated in Introduction/Setup; drives all ceiling and verifier comparisons.
  • ad hoc to paper The modal query among K samples is the right unit to score for selective prediction.
    Section 3; alternative selection (best-of-K, execution-ranked) could change absolute accuracies and signal strengths.
  • ad hoc to paper Leave-one-database-out macro AUROC on the studied schema sets measures cross-schema transfer of verifiers.
    Section 4.4; underpins the claim that fine-tuning does not yield universal verifiers.
  • standard math Paired bootstrap CIs and cross-fit logistic combination yield valid comparisons without test leakage.
    Setup/Metrics; standard statistical practice as used here.
invented entities (1)
  • black-box correctness ceiling (≈0.61–0.68 AUROC band) no independent evidence
    purpose: Descriptive name for the empirical plateau of non-reasoning UQ signals on hard multi-table SQL.
    Not a new physical object; a summary of measured AUROCs. Independent evidence is only the paper’s own tables, so independent_evidence is false as a postulated entity, but it is an empirical finding rather than a free theoretical construct.

pith-pipeline@v1.1.0-grok45 · 15978 in / 3566 out tokens · 50349 ms · 2026-07-10T21:04:54.919271+00:00 · methodology

0 comments
read the original abstract

Evaluating uncertainty in AI-generated SQL queries requires estimating whether a query is correct, where correct means it executes to the same result as a human-written reference. We study which signals predict correctness on hard multi-table text-to-SQL, using AUROC to measure how well each ranks correct queries above incorrect ones. On BIRD and Spider, black-box signals such as string, structural, and execution self-consistency, a schema-relevance score, and query executability all fall between about 0.61 and 0.68 AUROC, with string self-consistency strongest at 0.675; white-box log-probability is similar (0.67). The signals that move past this ceiling are verification-based: an LLM judge scores from 0.72 (GPT-4o-mini) to 0.78 (Claude). Judges from different providers make different errors, so a two-provider ensemble reaches 0.82 AUROC with a well-calibrated probability (expected calibration error 0.03) and supports useful abstention frontiers (for example, answering 27% of questions at 24% selective risk) where self-consistency offers no valid low-risk subset. The pattern holds across two benchmarks, two generators, and two judge providers. We also ask whether a verifier can be trained. Fine-tuned verifiers, both encoder and generative, reach about 0.77 to 0.79 AUROC in-distribution but fall to about 0.66 on unseen schemas; scaling to 7B, adding schema diversity, distilling a strong judge's rationales, and cross-benchmark training all fail to close that gap. Cross-schema transfer appears to track model scale and reasoning rather than fine-tuning. In practice, correctness uncertainty for text-to-SQL lives in reasoning-based signals: a fine-tuned verifier is a good in-domain tool, but a verifier that generalizes across schemas currently means a large frozen reasoning model.

Figures

Figures reproduced from arXiv: 2607.06799 by Robert Richardson.

Figure 1
Figure 1. Figure 1: Per-database transfer. The frozen GPT-4o judge leads on every held-out schema, above both [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Left: risk–coverage, where the ensemble dominates self-consistency. Right: reliability, where the [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

38 extracted references · 38 canonical work pages · 2 internal anchors

  1. [1]

    EMNLP , year=

    Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task , author=. EMNLP , year=

    work page

  2. [2]

    ICLR , year=

    Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows , author=. ICLR , year=

    work page

  3. [3]

    NeurIPS Datasets and Benchmarks , year=

    Can LLM Already Serve as a Database Interface? A Big Bench for Large-Scale Database Grounded Text-to-SQLs (BIRD) , author=. NeurIPS Datasets and Benchmarks , year=

    work page

  4. [4]

    ACL , year=

    RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers , author=. ACL , year=

    work page

  5. [5]

    ACL , year=

    LGESQL: Line Graph Enhanced Text-to-SQL Model with Mixed Local and Non-Local Relations , author=. ACL , year=

    work page

  6. [6]

    ICLR , year=

    Self-Consistency Improves Chain of Thought Reasoning in Language Models , author=. ICLR , year=

    work page

  7. [7]

    ICLR , year=

    Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation , author=. ICLR , year=

    work page

  8. [8]

    Nature , volume=

    Detecting Hallucinations in Large Language Models Using Semantic Entropy , author=. Nature , volume=

    work page

  9. [9]

    The Annals of Probability , volume=

    The Two-Parameter Poisson--Dirichlet Distribution Derived from a Stable Subordinator , author=. The Annals of Probability , volume=

    work page

  10. [10]

    ACL , year=

    A Hierarchical Bayesian Language Model Based on Pitman--Yor Processes , author=. ACL , year=

    work page

  11. [11]

    Advances in Neural Information Processing Systems (NIPS) , pages=

    Adaptor Grammars: A Framework for Specifying Compositional Nonparametric Bayesian Models , author=. Advances in Neural Information Processing Systems (NIPS) , pages=

    work page

  12. [12]

    Biometrika , volume=

    The Population Frequencies of Species and the Estimation of Population Parameters , author=. Biometrika , volume=

    work page

  13. [13]

    ICML , year=

    On Calibration of Modern Neural Networks , author=. ICML , year=

    work page

  14. [14]

    NeurIPS , year=

    Selective Classification for Deep Neural Networks , author=. NeurIPS , year=

    work page

  15. [17]

    Proceedings of the ACM on Management of Data (SIGMOD) , year=

    Reliable Text-to-SQL with Adaptive Abstention , author=. Proceedings of the ACM on Management of Data (SIGMOD) , year=

    work page

  16. [18]

    arXiv preprint , year=

    Calibrating Large Language Models for Text-to-SQL , author=. arXiv preprint , year=

    work page

  17. [19]

    Advances in Neural Information Processing Systems (NeurIPS) , year=

    Reflexion: Language Agents with Verbal Reinforcement Learning , author=. Advances in Neural Information Processing Systems (NeurIPS) , year=

    work page

  18. [20]

    International Conference on Learning Representations (ICLR) , year=

    Teaching Large Language Models to Self-Debug , author=. International Conference on Learning Representations (ICLR) , year=

    work page

  19. [21]

    Pourreza, Mohammadreza and Rafiei, Davood , booktitle=

    work page

  20. [22]

    Wang, Bing and Ren, Changyu and Yang, Jian and others , journal=

    work page

  21. [23]

    Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control

    Anastasios N. Angelopoulos, Stephen Bates, Emmanuel J. Cand \`e s, Michael I. Jordan, and Lihua Lei. Learn then test: Calibrating predictive algorithms to achieve risk control. arXiv preprint arXiv:2110.01052, 2021

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  22. [24]

    Conformal Risk Control

    Anastasios N. Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster. Conformal risk control. arXiv preprint arXiv:2208.02814, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  23. [25]

    Lgesql: Line graph enhanced text-to-sql model with mixed local and non-local relations

    Ruisheng Cao, Lu Chen, Zhi Chen, Yanbin Zhao, Su Zhu, and Kai Yu. Lgesql: Line graph enhanced text-to-sql model with mixed local and non-local relations. In ACL, 2021

    work page 2021

  24. [26]

    Teaching large language models to self-debug

    Xinyun Chen, Maxwell Lin, Nathanael Sch \"a rli, and Denny Zhou. Teaching large language models to self-debug. In International Conference on Learning Representations (ICLR), 2024

    work page 2024

  25. [27]

    Detecting hallucinations in large language models using semantic entropy

    Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detecting hallucinations in large language models using semantic entropy. Nature, 630 0 (8017): 0 625--630, 2024

    work page 2024

  26. [28]

    Selective classification for deep neural networks

    Yonatan Geifman and Ran El-Yaniv. Selective classification for deep neural networks. In NeurIPS, 2017

    work page 2017

  27. [29]

    Weinberger

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In ICML, 2017

    work page 2017

  28. [30]

    Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation

    Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In ICLR, 2023

    work page 2023

  29. [31]

    Spider 2.0: Evaluating language models on real-world enterprise text-to-sql workflows

    Fangyu Lei et al. Spider 2.0: Evaluating language models on real-world enterprise text-to-sql workflows. In ICLR, 2025

    work page 2025

  30. [32]

    Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls (bird)

    Jinyang Li et al. Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls (bird). In NeurIPS Datasets and Benchmarks, 2023

    work page 2023

  31. [33]

    DIN-SQL : Decomposed in-context learning of text-to-sql with self-correction

    Mohammadreza Pourreza and Davood Rafiei. DIN-SQL : Decomposed in-context learning of text-to-sql with self-correction. In Advances in Neural Information Processing Systems (NeurIPS), 2023

    work page 2023

  32. [34]

    Calibrating large language models for text-to-sql

    Ashwin Ramachandran and Sunita Sarawagi. Calibrating large language models for text-to-sql. arXiv preprint, 2024

    work page 2024

  33. [35]

    Reflexion: Language agents with verbal reinforcement learning

    Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2023

    work page 2023

  34. [36]

    Reliable text-to-sql with adaptive abstention

    Oleg Somov et al. Reliable text-to-sql with adaptive abstention. In Proceedings of the ACM on Management of Data (SIGMOD), 2025

    work page 2025

  35. [37]

    Rat-sql: Relation-aware schema encoding and linking for text-to-sql parsers

    Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. Rat-sql: Relation-aware schema encoding and linking for text-to-sql parsers. In ACL, 2020

    work page 2020

  36. [38]

    MAC-SQL : A multi-agent collaborative framework for text-to-sql

    Bing Wang, Changyu Ren, Jian Yang, et al. MAC-SQL : A multi-agent collaborative framework for text-to-sql. arXiv preprint, 2024

    work page 2024

  37. [39]

    Self-consistency improves chain of thought reasoning in language models

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, et al. Self-consistency improves chain of thought reasoning in language models. In ICLR, 2023

    work page 2023

  38. [40]

    Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task

    Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, et al. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In EMNLP, 2018

    work page 2018