{"id":"3c054e0a-bfaa-4aa7-b2c4-46b25b36ae81","arxiv_id":"2501.09527","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Entropy-based selective classifiers can detect errors in text-to-SQL systems, and T5 models are better calibrated than GPT-4 and Llama 3 under distribution shift.","lead":"This paper tests whether a model's own confidence can catch wrong or unanswerable text-to-SQL queries before they reach a user. It finds that T5 models are better calibrated and more reliably flagged by an entropy-based selective classifier than GPT-4 or Llama 3.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The T5 calibration advantage is confounded with the uncertainty estimator: T5/Llama use max-token entropy (Eq. 1) while GPT-4/DAIL-SQL use sequence log-probability (Eq. 2), so 'better calibrated' may be an estimator artifact, not an architectural property.","rationale":"The paper is a careful empirical study with reproducible code, multiple distribution-shift splits, and a consistent finding that T5 achieves lower Brier scores after post-hoc calibration. It also honestly reports limitations. However, the strongest claim in the abstract makes an architectural attribution ('encoder-decoder T5 is better calibrated') that is not supported by the experimental design, because the uncertainty estimator is not held fixed across the compared families. The reader's weakest_assumption correctly identifies this confound. I agree with the reader: the verdict should remain CONDITIONAL, conditioned on a common-estimator comparison. The proposed test is straightforward because all five systems can produce token-level log-probabilities (GPT-4 via API logprobs), so the authors can rerun the analysis without changing the experimental protocol. If the T5 advantage persists under both estimators, the paper's claim is strengthened; if not, the conclusion must be restated as 'the max-token entropy estimator paired with T5 is better calibrated' rather than a property of the architecture. I do not see grounds for rejection: the data are internally consistent and the concern is addressable.","tokens_in":15759,"tokens_out":9161,"duration_ms":89569,"concrete_test":"Recompute Table 3 with a common estimator for all five systems. Use OpenAI API logprobs to compute max-token entropy (Eq. 1) for DAIL-SQL/GPT-4, and also compute normalized sequence log-probability (Eq. 2) for T5 and Llama from their token logprobs. Fit isotonic calibration on Dknown and evaluate Brier on Dunk for both estimators. If T5 remains best-calibrated under both estimators, the architecture claim holds; if the ranking changes (e.g., GPT-4 with Eq. 1 beats T5, or T5 with Eq. 2 loses to Llama), the headline conclusion is an artifact of estimator choice.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim is that encoder-decoder T5 is better calibrated than GPT-4 and Llama 3, and therefore the entropy-based selective classifier performs better. The evidence in Table 3 compares Brier scores across model families that do not use the same uncertainty score. T5 and Llama use max-token entropy (Eq. 1); DAIL-SQL/GPT-4 uses normalized sequence log-probability (Eq. 2). These are not interchangeable: the first is the entropy of the single weakest token, the second is an average log-likelihood over all tokens. They have different scales, different sensitivities to sequence length, and different behavior under distribution shift. Because model family is perfectly confounded with estimator choice, the observed Brier-score advantage of T5 over GPT-4 may reflect that max-token entropy is a more informative score for this task, not that T5's internal confidence estimates are better calibrated. The T5-vs-Llama comparison is not confounded, but the headline claim bundles GPT-4 and Llama. The same confound carries over to the selective-classifier comparison, since the classifier is built on the same score. A secondary issue is that 'Result EX = EM - FDR' in Table 1 is not a standard selective-classification metric and does not account for coverage; the primary concern remains the untested architecture-versus-estimator attribution.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies error detection and calibration for Text-to-SQL systems under distribution shift. It defines an uncertainty score for generated SQL queries, fits three selective classifiers (threshold, logistic regression, and Gaussian mixture) on a known OOD sample Dknown, and evaluates on a held-out OOD sample Dunk. Experiments compare T5-large, T5-3B, DAIL-SQL with ChatGPT-4, and Llama-3-8B in SFT and LoRA variants on PAUQ (cross-database, template, and length splits) and EHRSQL. The main findings are that Gaussian Mixture gives the best coverage/risk trade-off, that unanswerable questions are detected more reliably than incorrect generations, that post-hoc calibration (especially isotonic regression) improves Brier scores, and that T5 models are better calibrated than the other families, leading to better selective classification.","tokens_in":16115,"tokens_out":7158,"duration_ms":73221,"significance":"If the architecture-level calibration claim held, the paper would be a useful practical contribution: it identifies an entropy-based reject option as a workable safety mechanism for Text-to-SQL, shows which calibration method is most effective, and provides new shifted splits for PAUQ. The paper has notable strengths: an honest Dknown/Dunk protocol for fitting calibration and selective classifiers, multiple random seeds, three selective classifiers reported (two in the appendix), and released code for splits and models. The central limitation is that the comparison across model families is confounded with the uncertainty estimator, so the headline conclusion needs additional evidence or rephrasing.","major_comments":[{"comment":"The headline comparison of model families is confounded with the choice of uncertainty estimator. T5 and Llama use max-token entropy (Eq. 1), while DAIL-SQL/GPT-4 uses normalized sequence log-probability (Eq. 2). These are not interchangeable scores: one is the entropy of the weakest token, the other is an average log-likelihood over the generated sequence, and they have different scales, length sensitivities, and behavior under distribution shift. Because model family and estimator are perfectly aligned, the Brier-score advantage of T5 in Table 3 and the selective-classifier advantage in Table 1 may reflect the estimator rather than the architecture. Please re-run the comparison with at least one shared estimator across all models (e.g., normalized sequence log-probability for T5/Llama, and max-token entropy for DAIL-SQL if token log-probabilities are accessible), or explicitly restate the conclusion as a system-level comparison such as 'T5 with max-token entropy is better calibrated than DAIL-SQL with normalized sequence log-probability' rather than an architecture-level claim.","section":"3 (Uncertainty Estimate), Eqs. (1)-(2); Tables 1 and 3"},{"comment":"The metric 'Result EX = EM - FDR' is not a well-defined selective-classification performance measure. It subtracts the false discovery rate of the rejection decision from the execution match of the base generator, which conflates different populations and does not account for the reduction in coverage caused by abstention. A classifier that abstains on almost all correct queries can have a low FDR and therefore a high 'Result EX' even though the system is practically useless. The comparison of Gaussian Mixture versus Logistic Regression and Threshold in Table 2 is based on this metric, so the RQ1/RQ2 conclusion is not supported by the reported numbers. Please report standard risk-coverage curves or evaluate at fixed coverage levels, and use an aggregate such as the area under the risk-coverage curve.","section":"4 (Case Study #1), Tables 1-2"},{"comment":"The central calibration claim rests on Brier scores averaged across splits with no confidence intervals, per-split breakdowns, or significance tests. The differences between T5-3B (0.106) and Llama3-8B SFT (0.175) or DIAL-SQL (0.152) could be real or could be within seed and split variance. Please provide standard deviations or bootstrap intervals and show per-split values; otherwise the 'better calibrated' claim in the abstract is not quantified with uncertainty.","section":"5 (Case Study #2), Table 3"},{"comment":"The prediction rule is stated as 'the system predicts the query if u >= gamma; otherwise it abstains,' but Eq. (1) defines u as max-token entropy, for which higher values mean lower confidence. Under the stated rule the system would predict exactly the high-entropy generations it should abstain on. This is the sign convention for a confidence score, not an uncertainty score. Please clarify the intended direction of the threshold or correct the inequality, since the threshold-based selective classifier is otherwise not reproducible.","section":"3 (Selective Prediction Setting)"}],"minor_comments":[{"comment":"There are typos: 'descent ability' (Sec. 3), 'utulize' (Sec. 3), 'Platts' (Sec. 5), and 'ERHSQL' (Sec. 4) should be corrected.","section":"Throughout"},{"comment":"The caption calls recall 'the coverage of our selective classifier,' but recall is the fraction of detected errors, while coverage is the fraction of all test examples that are predicted; these concepts should be separated.","section":"4, Table 1 caption"},{"comment":"Gaussian Mixture appears to be selected based on the same evaluation data used to report final numbers; since the other classifiers are reported in the appendix, this is acceptable as an exploratory comparison, but it should be stated explicitly that this involves model selection on the evaluation set.","section":"4, Fig. 2"},{"comment":"The notation lists theta_0, ..., theta_M while the objective uses theta_m; the index ranges should be aligned.","section":"5, Eq. (6)"},{"comment":"The term 'inverted execution match metric (EX)' is confusing; execution match is normally denoted EM, and calling it 'inverted' obscures the standard definition.","section":"3 (Text-to-SQL Models)"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is worth a look if you work on Text-to-SQL reliability. It is the first systematic evaluation of selective classifiers (threshold, logistic regression, Gaussian mixture) for error detection in Text-to-SQL under distribution shift, over PAUQ and EHRSQL, with new compositional splits and released code. That part is genuinely useful, and the Dknown/Dunk split is an honest protocol: they fit calibrators and classifiers on one OOD sample and test on another, with three seeds.\n\nThe strongest clean result is the T5 versus Llama comparison. Both use the same max-token entropy score (Eq. 1), so the finding that T5-large and T5-3B are better calibrated under distribution shift is evidence about architecture, not just about the score. The observation that unanswerable questions are easier to flag than wrong generations is also a solid, reproducible empirical takeaway.\n\nThe soft spots are in the headline. The abstract claims T5 is better calibrated than GPT-4/DAIL-SQL, but that comparison is confounded: GPT-4 uses normalized sequence log-probability (Eq. 2), while T5 and Llama use max-token entropy. Different scales, different length behavior, different sensitivity to weak tokens. So 'T5 beats GPT-4' could easily be 'max-token entropy beats normalized log-prob' as an uncertainty estimator. The paper needs either a matched estimator (per-token logprobs from GPT-4) or a rewritten claim that limits the architectural statement to T5 versus Llama.\n\nTwo smaller issues. Table 3 reports Brier scores without error bars, so the calibration gaps between models could be seed noise; the sigma columns in the selective tables are nice but not present here. And the 'Result EX = EM - FDR' metric in Table 1 is nonstandard—subtracting a rate from an accuracy score without coverage weighting makes it hard to compare across rows. Standard risk-versus-coverage or selective-AUC plots are more interpretable; they have those too (Fig. 3, Appx. E), so the paper is recoverable in revision.\n\nRecommendation: send it to peer review. The core empirical contribution is reproducible and the confound is addressable in one revision cycle. The paper is not ready as-is, but it deserves a serious referee.","headline":"Careful empirical study of reject-option Text-to-SQL that ships code and uses an honest Dknown/Dunk protocol, but the abstract's T5-beats-GPT-4 claim is confounded by using different uncertainty estimators across model families.","tokens_in":16598,"tokens_out":3101,"would_cite":true,"duration_ms":29343,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that entropy-based confidence scores let a selective classifier detect erroneous SQL generations, with T5 models better calibrated than GPT-4 or Llama 3.","keywords":["text-to-SQL","selective classification","uncertainty estimation","calibration","entropy","distribution shift","error detection","large language models"],"falsifier":"Rerun the calibration and selective-classification experiments using the same uncertainty estimator for every model family—for example, sequence log-probability for T5 and Llama, or token-level probabilities from GPT-4 if they can be obtained—and check whether the T5 advantage in Brier score and selective AUC persists; if it disappears, the paper's headline claim fails.","tokens_in":15554,"feed_emoji":"🗄️","tokens_out":9484,"duration_ms":72974,"temperature":0.7,"pith_summary":"This paper asks whether a Text-to-SQL system can be made safer by letting it abstain from answering when it is not confident, and whether that confidence can be trusted. It shows that an entropy-based score—the uncertainty of the model's weakest predicted token—fed into a Gaussian mixture selective classifier can detect erroneous SQL generations under distribution shift, cutting risk by about 80 percent on average across the tested models and splits. The paper's headline claim is that encoder-decoder T5 models are better calibrated than the in-context-learning GPT-4 system and decoder-only Llama 3, so the same selective classifier works better on T5 outputs. It also finds that the detector is more likely to catch queries generated in response to irrelevant or unanswerable questions than to catch incorrect queries for relevant questions. The practical stake is that a reject option, with proper calibration, could reduce the false positives that make text-to-SQL unreliable in real deployments.","feed_headline":"T5's entropy confidence better catches bad SQL queries","feed_subtitle":"Entropy-based scores let a Gaussian mixture abstain on risky queries; T5's calibration best flags wrong SQL.","key_machinery":"The engine is the maximum-token-entropy uncertainty estimate: for an autoregressive model, each predicted token carries a softmax distribution, and the sequence's confidence is set by the worst token, $u = \\max_l H(p_l)$, where $H$ is Shannon entropy of the token distribution (for the closed GPT-4 system the paper substitutes the normalized sequence log-probability). This single score is the input to a Gaussian mixture selective classifier fit by expectation-maximization on a known out-of-distribution subset; the classifier treats the score as a draw from one of two components, correct or erroneous, and abstains when the erroneous component is more likely. Post-hoc calibration (Platt scaling and isotonic regression) maps the raw scores to probabilities, with isotonic regression giving the best Brier scores.","core_discovery":"The central claim is that a selective classifier built on a token-entropy confidence estimate can identify erroneous SQL generations in out-of-distribution settings, and that the reliability of this rejection mechanism tracks the model's calibration. Across four distribution-shift splits from PAUQ (cross-database, template, and target-length shifts) and EHRSQL (covariate shift with unanswerable questions), the entropy-based Gaussian mixture classifier reduces risk by an average of 80 percent, but at the cost of high false-discovery rates in full-shift settings. Under covariate shift alone, however, selective text-to-SQL keeps false discovery low—around 10 percent for the best models—and improves the net execution match. The paper's comparative conclusion is that encoder-decoder T5 models (large and 3B) are better calibrated after post-hoc calibration than the GPT-4-based DAIL-SQL and Llama 3 variants, and the Gaussian mixture selective classifier performs better on them; it is also more likely to flag generations for irrelevant questions than incorrect generations for relevant ones.","pith_inferences":["The paper's uncertainty score for fine-tuned models is not truly an entropy of the whole sequence but the maximum over tokens, which ignores token position and dependence; a sequence-level product or mutual-information-based score might change the ranking between architectures.","A natural extension is to apply the same pipeline to benchmarks with dirty values and external-knowledge requirements to see if the covariate-shift finding generalizes beyond the clinical domain.","Because the Gaussian mixture assumes exactly two normal components, the selective classifier may be miscalibrated for multimodal error score distributions; fitting a density-ratio or a nonparametric threshold could improve FDR on the hardest TSL SSP split.","The paper stops short of a decision-theoretic analysis of abstention cost; in practice the threshold should be set by a user-specified cost ratio between a wrong SQL and a refused answer, which the reported recall/FDR tables make possible."],"forward_implications":["If the central claim holds, a text-to-SQL deployment can attach a reject option to any generation model and abstain on the least-confident outputs, trading a bounded loss of coverage for a large drop in wrong SQL delivered to users.","The result that unanswerable questions are easier to catch than incorrect generations implies that production systems should treat OOD/unanswerable detection and in-scope error detection as separate channels with different thresholds.","Calibration under distribution shift does not restore reliability: even after isotonic regression, full-shift splits leave models uncalibrated, so a selective classifier trained on one OOD sample may not transfer to another.","The better calibration of encoder-decoder T5 suggests that for safety-critical text-to-SQL, architecture choice matters as much as accuracy when selecting a model."],"supporting_citations":[{"why":"Supplies the maximum-entropy uncertainty estimate that the selective classifier operates on.","marker":"(Yang et al. 2024)"},{"why":"Provides the risk-coverage selective prediction setup and the split of test data into known and unknown OOD samples.","marker":"(El-Yaniv and Wiener 2010)"},{"why":"Contributes the EHRSQL clinical benchmark with unanswerable questions used for the covariate-shift split.","marker":"(Lee et al. 2022)"},{"why":"Contributes the SPIDER benchmark from which the PAUQ splits are derived, the base for execution match evaluation.","marker":"(Yu et al. 2018)"},{"why":"Contributes the PAUQ dataset and its cross-database split used for the domain-shift experiments.","marker":"(Bakshandaeva et al. 2022)"},{"why":"Provides the template-split methodology that the compositional Template SSP split refines.","marker":"(Finegan-Dollak et al. 2018a)"},{"why":"Defines the Platt scaling calibration method compared across models.","marker":"(Platt et al. 1999)"},{"why":"Defines isotonic regression, the calibration method the paper finds best.","marker":"(Zadrozny and Elkan 2002)"},{"why":"Introduces the T5 encoder-decoder models whose calibration is the paper's headline result.","marker":"(Raffel et al. 2020)"},{"why":"Introduces DAIL-SQL, the GPT-4 in-context learning system compared against fine-tuned models.","marker":"(Gao et al. 2024)"}],"fun_headline_variants":["Entropy-based confidence flags bad SQL queries","T5 calibration improves error detection in Text-to-SQL","Selective classifiers catch wrong SQL with confidence scores","Better calibration leads to better SQL error detection","Gaussian mixture abstains on risky SQL queries"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison across model families uses different uncertainty scores—maximum token entropy for T5 and Llama, but normalized sequence log-probability for GPT-4—so the conclusion that T5 is better calibrated could be an artifact of the score choice rather than a property of the architecture.","fun_headline_variants_meta":{"raw":{"variants":["Entropy-based confidence flags bad SQL queries","T5 calibration improves error detection in Text-to-SQL","Selective classifiers catch wrong SQL with confidence scores","Better calibration leads to better SQL error detection","Gaussian mixture abstains on risky SQL queries"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00019,"raw_usage":{"total_tokens":1337,"prompt_tokens":939,"completion_tokens":398,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":555,"completion_tokens_details":{"reasoning_tokens":327}},"tokens_in":555,"tokens_out":398,"duration_ms":4945,"temperature":1.0,"reasoning_tokens":327,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T19:55:48.867359+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the calibration and selective-classification experiments using the same uncertainty estimator for every model family—for example, sequence log-probability for T5 and Llama, or token-level probabilities from GPT-4 if they can be obtained—and check whether the T5 advantage in Brier score and selective AUC persists; if it disappears, the paper's headline claim fails.","supporting_citations":[{"cited_title":"Towards Unbiased Evaluation of Detecting Unanswerable Questions in EHRSQL","cited_arxiv_id":"2405.01588","evidence_quote":"Supplies the maximum-entropy uncertainty estimate that the selective classifier operates on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the risk-coverage selective prediction setup and the split of test data into known and unknown OOD samples."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes the EHRSQL clinical benchmark with unanswerable questions used for the covariate-shift split."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes the PAUQ dataset and its cross-database split used for the domain-shift experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Platt scaling calibration method compared across models."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines isotonic regression, the calibration method the paper finds best."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces DAIL-SQL, the GPT-4 in-context learning system compared against fine-tuned models."}],"review_version":1}