Pith. sign in

REVIEW 4 major objections 5 minor 75 references

An Efficient and Effective Evaluator for Text2SQL Models on Unseen and Unlabeled Data

T0 review · 4 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A fixed Text2SQL model's execution accuracy on an unseen, unlabeled workload can be estimated from distribution-shift descriptors alone — no gold SQL or model retraining needed.

desk verdict A solid label-free accuracy estimator for Text2SQL with a strong zero-label transfer result, but the synthetic-to-real generalization story is asserted more than proven. read the letter →

arxiv 2603.07841 v2 pith:APYB5ZRV submitted 2026-03-08 cs.CL

Trinh Pham , Thanh Tam Nguyen , Viet Huynh , Hongzhi Yin , Quoc Viet Hung Nguyen This is my paper
classification cs.CL
keywords Text2SQLlabel-freeevaluationdistributionshiftexecutionaccuracymodel-agnosticevaluatorsyntheticdatasetmeta-learning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether a fixed Text2SQL model's accuracy on a brand-new, unlabeled database can be estimated before any gold SQL exists. It answers yes: by pooling the model's own embeddings on its training data and on the new workload, computing three distribution-shift descriptors (global drift, tail risk, shape change), and feeding them to a small regressor, the framework predicts execution accuracy without labels or model retraining. A reader should care because text-to-SQL systems are increasingly deployed against evolving databases where labeled test sets are expensive or privacy-blocked; a reliable label-free estimate turns deployment from a gamble into a measurable decision. The paper trains and validates the approach on a large synthetic corpus and reports mean absolute errors of about four percentage points across five benchmark transfers, with similar behavior for unseen model families via meta-learning.

What carries the argument

The load-bearing object is the shift descriptor Δ = [SD_F, SD_M, SD_SW] plus the regression head g_θ. SD_F = (∥μ_T−μ_S∥₂², variance ratios) captures global drift; SD_M whitens target embeddings with source statistics to expose tail behavior; SD_SW averages 1D Wasserstein distances between sliced projections to capture shape changes. These are computed on pooled last-layer embeddings of the model applied to its training workload and to the target workload, so the estimator needs no labels, no execution, and no access to model weights. A 3-layer MLP regressor (and a Reptile meta-learning wrapper for unseen models) maps Δ to an execution-accuracy estimate; a Hybrid SWD approximation keeps runti

What would settle it

Take two target workloads whose pooled embedding sets yield identical SD_F, SD_M, and SD_SW relative to the same source, but whose true execution accuracies differ by more than 10 points — for instance, one set of simple single-table questions and an identically phrased set over a schema with many semantically overlapping columns. If the regressor outputs the same estimate for both, the descriptor set cannot be carrying the accuracy-relevant information, and the central claim collapses.

Watch

Extended reading notes

Core claim

The paper's central claim is that the dataset-level execution accuracy of a frozen Text2SQL model on an unseen, unlabeled workload, M*, can be estimated as M̂ = g_θ(Δ_test), where Δ_test = h(ϕ_src, ϕ_tgt) is a fixed-length shift descriptor comparing pooled embeddings of the model's training workload and the target workload, and g_θ is a three-layer MLP trained on synthetic shift–accuracy pairs. The descriptor h combines three complementary statistics: a Fréchet term for global domain drift (mean and variance shifts), a Mahalanobis term for tail risk (rare, failure-prone queries), and a sliced Wasserstein term for distributional shape change (schema restructuring, collapsed query templates).

Load-bearing premise

The learned relationship between shift descriptors and accuracy transfers from the synthetic FusionDataset to real deployment targets, including shift directions not present in the synthetic corpus.

Editorial extensions

If this is right

  • Pre-deployment checks become possible without labels: teams can estimate whether a Text2SQL model is ready for a new database before any gold SQL is written.
  • Continuous monitoring of evolving schemas is feasible: as a database changes, shift descriptors can be recomputed and an accuracy drop flagged before users report failures.
  • The estimator is model-agnostic: it works with five large neural systems and with classic non-neural Text2SQL engines, so one evaluation tool can cover an organization's whole model fleet.
  • New Text2SQL model families can be handled after a few gradient steps, because the meta-learned initialization adapts rapidly without target labels.
  • Because the descriptor pipeline is a matrix multiplication plus a small MLP, evaluation is faster than LLM-as-a-judge baselines, making routine readiness checks practical.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same shift-descriptor recipe (global + tail + shape statistics on pooled embeddings) could plausibly transfer to other structured prediction tasks — code generation, semantic parsing — where accuracy is defined by execution and labels are scarce; the paper does not test this.
  • The paper's meta-adaptation for a previously unseen model requires computing the model's accuracy on known sample sets, which presupposes labels for those samples; the 'fully label-free' framing strictly holds for models whose behavior is already covered by the meta-training pool.
  • The stated objective of calibrated prediction intervals is not reported in the experiments; a testable extension is to check coverage of the reported MAE intervals on held-out transfers.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes FusionSQL, a method to estimate the execution accuracy of a fixed Text2SQL model on an unseen, unlabeled workload without gold SQL. It computes shift descriptors (Fréchet, Mahalanobis, sliced Wasserstein distances) between pooled embeddings of the model's training workload and the target workload, and trains a small MLP regressor on a large synthetic corpus (FusionDataset, 3.3M examples) to map these descriptors to accuracy. Experiments on five public benchmark transfers report MAE around 4–5 percentage points, outperforming confidence-based and LLM-judge baselines. A meta-learning extension (FusionSQL-ML) is proposed for unseen models, and ablations address scalability and non-neural systems.

Significance. If the transfer result is robust, FusionSQL addresses a real deployment gap and is one of the first dataset-level label-free evaluators for Text2SQL. Strengths include a clearly stated problem formulation, a large synthetic training corpus, a lightweight and interpretable descriptor set, and an efficiency analysis. The code release is a positive step. However, the paper's headline claims are weakened by an unspecified MAE unit, a labeled auxiliary set in the unseen-model protocol, an unevaluated uncertainty objective, and a transfer evidence base that is finite and descriptive rather than a stress test of out-of-support shifts. These issues need to be addressed before the results can be taken at face value.

major comments (4)
  1. [§V-C, Table III, Eq. (15)] The unit over which MAE is computed is not defined. For a transfer such as Spider→BIRD there is a single target workload, so Eq. (15)'s N cannot be one unless you split the target into multiple datasets (per database or per subset). The paper reports MAE with 95% CIs averaged over five seeds, but this does not explain how a dataset-level MAE is formed. Please specify what N is, how shift descriptors are computed for each unit, and report per-unit results. Without this, the central numerical claim in Table III is not interpretable.
  2. [§IV-C, Table IV] The generalization-to-unseen-models protocol is not label-free. The text says 'At deployment, for a previously unseen model m_new, we adapt the evaluator using S_i. Since S_i is known, we can compute the model's accuracy and form the loss L_{m_new,S_i}(θ).' Computing accuracy on S_i requires gold SQL and execution. Thus FusionSQL-ML uses reference labels on an auxiliary set, contradicting the abstract's 'without reference labels.' The paper should either scope the claim to 'no labels on the target workload' or provide a label-free adaptation procedure, and Table IV should be re-described accordingly.
  3. [§III-C, §V] The problem formulation promises calibrated uncertainty estimates: a prediction interval [M̂−δ_α, M̂+δ_α] with P(M* ∈ ...) ≥ 1−α. No experiment in §V reports interval half-widths, empirical coverage, or calibration plots. Since this is listed as a required property of the evaluator, the claim is unsupported. Add uncertainty-calibration experiments or remove/soften the property.
  4. [§IV-A, §V-A, Fig. 10b] The central transfer mechanism trains g_θ on synthetic shifts from FusionDataset and applies it to real benchmarks. The evidence is limited to five selected transfers, and the coverage analysis (t-SNE, radar) is descriptive. Fig. 10b shows the shift–accuracy relationship is not monotonic, so extrapolation to shift directions outside FusionDataset's support is fragile. Please add an explicit out-of-support test (e.g., a deployment workload whose schema/question style is deliberately absent from FusionDataset) or a leave-one-benchmark-out protocol, and report MAE and per-transfer errors. This would make the 'unseen workload' claim more than a finite-set demonstration.
minor comments (5)
  1. [Eq. (14)–(15)] The symbols ŷ_i and y_i are used for SQL queries in Eq. (14) and then for predicted/true accuracies in Eq. (15); use different notation to avoid ambiguity.
  2. [Table IV] Most 'unseen' models are size variants of families already in the meta-training pool (CodeLlama, StarCoder2, Phi, DeepSeek-Coder); only Mistral-7B is a genuinely new family. Please clarify how much of the reported gain is due to cross-family generalization.
  3. [§V-C, Fig. 10a] Pearson r=0.53–0.70 is described as 'strong'; this is more accurately moderate correlation. Please qualify the wording.
  4. [§IV-A] FusionDataset is not released. Since the training distribution is central to the method, please include a data-release plan or a detailed generation script in the repository.
  5. [§V-C] The sentence 'FusionSQL is trained purely from shift descriptors ... zero-label transfer setting' conflicts with the labeled adaptation described in §IV-C for unseen models. Clarify the scope.

Circularity Check

0 steps flagged · score 1.0 of 10

No construction-level circularity: the evaluator is trained on synthetic shift→accuracy pairs and applied to real unlabeled targets; the main risks are external validity and evaluation hygiene, not definitional or self-citational circularity.

full rationale

The central chain is: (1) construct FusionDataset of synthetic workloads with gold SQL; (2) for each (model, synthetic subset) pair compute shift descriptors Δ = h(φ_src, φ_syn) and the true synthetic accuracy M*_syn; (3) train g_θ by minimizing E[(g_θ(Δ)−M*)^2]; (4) at inference compute Δ_test from the real training workload and the unlabeled target workload, then output M̂_test = g_θ(Δ_test) (Eqs. 7–10). Nothing in this chain defines Δ in terms of M* on the target, nor fits any parameter to target labels. The public-benchmark transfers in Table III use target gold only to measure MAE after prediction, not as training or adaptation signal, so the reported numbers are external validation of a learned mapping rather than a forced reconstruction of the input. The adaptation protocol for unseen models (§IV-C, Eq. 13) does use labeled subsets S_i drawn from FusionDataset to adapt g_θ, but S_i is a synthetic training/adaptation set, not the unlabeled deployment target; the final target prediction remains label-free and is not statistically forced by a fit to target accuracy. Self-citations are present but not load-bearing: the sliced Wasserstein distance is cited to co-author prior work [51]–[53], but SWD is a standard, externally defined metric, and the meta-learning steps cite Reptile [54] and MAML [55], both external algorithms. The t-SNE and radar coverage evidence (§V-A) is descriptive support for the synthetic-to-real transfer assumption, not a derivation step; if the conditional Δ→accuracy relationship does not hold on genuinely new deployment workloads, that is a generalization-failure risk, not circularity. The paper does not state that early stopping or hyperparameters were selected on the Table III targets, so I do not treat that as an established circular step. Overall, the core result is an empirical regression-transfer claim with independent content; no step reduces by construction to its own inputs.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim rests on a learned mapping from embedding-distribution statistics to accuracy; the free parameters are tunable evaluation and data-construction choices, and the axioms are domain assumptions about transferability and label quality. No invented entities (particles, forces, dimensions) are needed; SD_F/SD_M/SD_SW are computed descriptors, and FusionDataset is a corpus artifact rather than a postulated entity.

free parameters (5)
  • Meta-set size n = 30,000
    Chosen by cost–error tradeoff in Fig. 6; larger n improves MLP until saturation, so 30K is a hand-selected operating point rather than derived.
  • Sample-set size cap |S_i| = ≤10,000
    Budget-limited; Fig. 5 shows MAE decreases with |S_i|, but the cap is set by the $1,000 generation budget and per-database caps.
  • Hybrid SWD slices (k,R) = k=8–10, R=16–24
    Selected from ablation in §V-D; fewer slices reduce latency while preserving MAE, so the reported accuracy depends on this configuration choice.
  • MLP architecture/hyperparameters = hidden 256/128/64, lr=1e-4, dropout=0.2, AdamW, 20 epochs
    Standard tuning choices; early stopping is done on validation MAE from the benchmark suite used for reporting, so they are fit on the same targets.
  • Generation budget costs c_gen,c_val,c_exec = 0.00012, 0.00003, 0.0004 USD/sample
    Ad hoc unit costs in Eq. 16 used to keep total under B=$1,000; they determine how many validation/execution checks are performed and thus label quality.
assumptions (5)
  • domain assumption Dataset-level accuracy of a Text2SQL model on a target workload is a learnable function of first/second-order and sliced-Wasserstein statistics of pooled embeddings of source vs target workloads.
    This is the core inductive premise of §IV-B: no proof is offered that these descriptors capture accuracy-relevant shift; the MLP g_θ is trained to approximate it.
  • domain assumption Synthetic FusionDataset shift conditions are representative of real-world Text2SQL distribution shifts, so g_θ transfers to real unseen benchmarks.
    Section V-A argues coverage via t-SNE/radar, but transferability from synthetic to real shifts is assumed; no formal guarantee or distribution-free bound is provided.
  • domain assumption Execution checks and GPT-5 judging in FusionDataset produce correct gold SQL and accuracy labels.
    Section IV-A says generated SQLs are filtered by execution and 'independent judges such as GPT-5'; if these are noisy, the training labels for g_θ are noisy.
  • domain assumption The labeled sample sets S_i used for meta-adaptation are available to the deployer for a new model.
    In §IV-C, adapting to m_new requires computing its true accuracy on labeled S_i; 'target evaluation remains label-free' only after this labeled adaptation.
  • domain assumption Last-layer pooled embeddings from different base Text2SQL models are sufficiently aligned for meta-training across model families.
    Section IV-C meta-trains across 20 models with 'shared backbone families', assuming cross-model representation alignment without quantitative alignment analysis.

how reviews work

0 comments
Cite this review

Pith. "Pith review of An Efficient and Effective Evaluator for Text2SQL Models on Unseen and Unlabeled Data." pith.science (2026). https://pith.science/paper/APYB5ZRV

@misc{pith2026260307841,
  author       = {Pith},
  title        = {Pith review of: An Efficient and Effective Evaluator for Text2SQL Models on Unseen and Unlabeled Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/APYB5ZRV}},
  note         = {Machine review of arXiv:2603.07841}
}
read the original abstract

Recent advances in large language models have strengthened Text2SQL systems that translate natural language questions into database queries. A persistent deployment challenge is to assess a newly trained Text2SQL system on an unseen and unlabeled dataset when no verified answers are available. This situation arises frequently because database content and structure evolve, privacy policies slow manual review, and carefully written SQL labels are costly and time-consuming. Without timely evaluation, organizations cannot approve releases or detect failures early. FusionSQL addresses this gap by working with any Text2SQL models and estimating accuracy without reference labels, allowing teams to measure quality on unseen and unlabeled datasets. It analyzes patterns in the system's own outputs to characterize how the target dataset differs from the material used during training. FusionSQL supports pre-release checks, continuous monitoring of new databases, and detection of quality decline. Experiments across diverse application settings and question types show that FusionSQL closely follows actual accuracy and reliably signals emerging issues. Our code is available at https://github.com/phkhanhtrinh23/FusionSQL.

Figures

Figures reproduced from arXiv: 2603.07841 by the authors.

Figure 1
Figure 1. Top: Existing Text2SQL evaluations rely on ground￾truth labels, which are often unavailable as databases evolve. Bottom: FusionSQL estimates model accuracy directly from unlabeled inputs without requiring ground-truth SQL labels. faster than annotation cycles, and manual labeling (drafting SQL, executing, debugging, and verifying) is both costly and time-consuming. Consequently, practitioners are forced into an unde… view at source ↗
Figure 2
Figure 2. FusionSQL framework. Training: A frozen Text2SQL model encodes training and FusionDataset samples into embeddings to compute shift descriptors (SDF , SDM, SDSW ) for training the FusionSQL evaluator. Inference: For unseen, unlabeled workloads, the same descriptors are computed to estimate accuracy without labels or retraining. where h(·) denotes a deterministic function that extracts a fixed-length representation of… view at source ↗
Figure 3
Figure 3. t-SNE coverage. Comparing 50K samples, Fusion￾Dataset bridges clusters of existing benchmarks in both do￾main (a) and question space (b), reflecting broader semantic and structural diversity of real-world Text2SQL variability. monotonic performance decline across tiers demonstrates that FusionDataset accurately captures a realistic gradient of rea￾soning complexity. Semantic Coverage [PITH_FULL_IMAGE:figures/full_f… view at source ↗
Figures from the paper (7 more)
Figure 5
Figure 5. Figure 5: Sample-set size. Impact of sample-set size |Si | used to compute distribution shifts for an instance (Dtrain, Si). Error decreases as |Si | grows. The per-sample costs are c gen = 0.00012, c val = 0.00003, and c exec = 0.0004, and we budget for expected to￾tals P d∈D n…
Figure 6
Figure 6. Figure 6: Meta-set size. FusionSQL’s MLP attains the lowest error and benefits most from larger meta-sets, while costs rise sharply beyond 30,000 with marginal gains. with larger meta-sets for all methods, but the MLP benefits most and continues to improve up to n=30K, whereas s…
Figure 8
Figure 8. Figure 8: Latency–accuracy trade-off. Bars report average evaluation latency, while the red curve shows MAE (lower is better). FusionSQL offers the best trade-off, whereas judge￾based methods incur high latency without much MAE gains. Estimation Calibration. MAE remains the prim…
Figure 7
Figure 7. Figure 7: EX across transfers. Across six source–target transfers and five base models, FusionSQL closely follows the ground￾truth (GT) trend, while FusionSQL-TL exhibits mild bias on harder targets due to limited labeled adaptation. Evaluator Benchmark. As this setting has not …
Figure 9
Figure 9. Figure 9: Latency–accuracy trade-off on unseen Text2SQL models. Bars denote relative latency, while lines show MAE. FusionSQL-ML attains the lowest MAE with low latency. Latency [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]
Figure 10
Figure 10. Figure 10: FusionSQL reliability and shift sensitivity. (a) FusionSQL accurately predicts EX. (b) Shift impacts EX on hard targets but only mildly on distant-but-easy transfers, demonstrating robustness. decoding and confidence statistics beyond the base Text2SQL generation. Fus…
Figure 11
Figure 11. Figure 11: Latency + MAE vs. number of slices L. Latency scales linearly with L, while MAE stabilizes beyond L≈32. Hybrid SWD (red points) achieves low MAE with reduced L, optimizing efficiency without sacrificing fidelity. Scalability. We evaluate FusionSQL’s scalability by var…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

75 extracted references · 17 linked inside Pith

  1. [1]

    The dawn of natural language to sql: Are we fully ready?

    B. Li, Y . Luo, C. Chai, G. Li, and N. Tang, “The dawn of natural language to sql: Are we fully ready?”PVLDB, vol. 17, 2024

    2024

  2. [2]

    Seq2sql: Generating structured queries from natural language using reinforcement learning,

    V . Zhong, C. Xiong, and R. Socher, “Seq2sql: Generating structured queries from natural language using reinforcement learning,”arXiv preprint arXiv:1709.00103, 2017

    arXiv 2017

  3. [3]

    Sqlnet: Generating structured queries from natural language without reinforcement learning,

    X. Xu, C. Liu, and D. Song, “Sqlnet: Generating structured queries from natural language without reinforcement learning,”arXiv preprint arXiv:1711.04436, 2017

    arXiv 2017

  4. [4]

    TypeSQL: Knowledge- based type-aware neural text-to-SQL generation,

    T. Yu, Z. Li, Z. Zhang, R. Zhang, and D. Radev, “TypeSQL: Knowledge- based type-aware neural text-to-SQL generation,” inNAACL, 2018

    2018

  5. [5]

    SyntaxSQLNet: Syntax tree networks for complex and cross-domain text-to-SQL task,

    T. Yu, M. Yasunaga, K. Yang, R. Zhang, D. Wang, Z. Li, and D. Radev, “SyntaxSQLNet: Syntax tree networks for complex and cross-domain text-to-SQL task,” inEMNLP, 2018

    2018

  6. [6]

    Editing-based SQL query generation for cross-domain context-dependent questions,

    R. Zhang, T. Yu, H. Er, S. Shim, E. Xue, X. V . Lin, T. Shi, C. Xiong, R. Socher, and D. Radev, “Editing-based SQL query generation for cross-domain context-dependent questions,” inEMNLP, 2019

    2019

  7. [7]

    To- wards complex text-to-SQL in cross-domain database with intermediate representation,

    J. Guo, Z. Zhan, Y . Gao, Y . Xiao, J.-G. Lou, T. Liu, and D. Zhang, “To- wards complex text-to-SQL in cross-domain database with intermediate representation,” inACL, 2019

    2019

  8. [8]

    RAT-SQL: Relation-aware schema encoding and linking for text-to-SQL parsers,

    B. Wang, R. Shin, X. Liu, O. Polozov, and M. Richardson, “RAT-SQL: Relation-aware schema encoding and linking for text-to-SQL parsers,” inACL, 2020

    2020

Show all 75 references
  1. [9]

    Bridging textual and tabular data for cross-domain text-to-SQL semantic parsing,

    X. V . Lin, R. Socher, and C. Xiong, “Bridging textual and tabular data for cross-domain text-to-SQL semantic parsing,” inEMNLP, 2020

    2020

  2. [10]

    SmBoP: Semi-autoregressive bottom-up se- mantic parsing,

    O. Rubin and J. Berant, “SmBoP: Semi-autoregressive bottom-up se- mantic parsing,” inNAACL, 2021

    2021

  3. [11]

    LGESQL: Line graph enhanced text-to-SQL model with mixed local and non-local relations,

    R. Cao, L. Chen, Z. Chen, Y . Zhao, S. Zhu, and K. Yu, “LGESQL: Line graph enhanced text-to-SQL model with mixed local and non-local relations,” inACL, 2021

    2021

  4. [12]

    Gra{pp}a: Grammar-augmented pre-training for table semantic parsing,

    T. Yu, C.-S. Wu, X. V . Lin, B. Wang, Y . C. Tan, X. Yang, D. Radev, R. Socher, and C. Xiong, “Gra{pp}a: Grammar-augmented pre-training for table semantic parsing,” inICLR, 2021

    2021

  5. [13]

    PICARD: Parsing in- crementally for constrained auto-regressive decoding from language models,

    T. Scholak, N. Schucher, and D. Bahdanau, “PICARD: Parsing in- crementally for constrained auto-regressive decoding from language models,” inEMNLP, 2021

    2021

  6. [14]

    DIN-SQL: Decomposed in-context learning of text-to-SQL with self-correction,

    M. Pourreza and D. Rafiei, “DIN-SQL: Decomposed in-context learning of text-to-SQL with self-correction,” inNeurIPS, 2023

    2023

  7. [15]

    Text- to-sql empowered by large language models: A benchmark evaluation,

    D. Gao, H. Wang, Y . Li, X. Sun, Y . Qian, B. Ding, and J. Zhou, “Text- to-sql empowered by large language models: A benchmark evaluation,” PVLDB, vol. 17, 2024

    2024

  8. [16]

    Chess: Contextual harnessing for efficient sql synthesis,

    S. Talaei, M. Pourreza, Y .-C. Chang, A. Mirhoseini, and A. Saberi, “Chess: Contextual harnessing for efficient sql synthesis,”arXiv preprint arXiv:2405.16755, 2024

    2024 arXiv

  9. [17]

    How to answer secure and private SQL queries?

    Q. Luo, Q. Xu, and C. Yang, “How to answer secure and private SQL queries?” inICDE, 2025

    2025

  10. [18]

    Lampat: Low-rank adaption for multilingual paraphrasing using adversarial training,

    K. M. Le, T. Pham, T. Quan, and A. T. Luu, “Lampat: Low-rank adaption for multilingual paraphrasing using adversarial training,”AAAI, 2024

    2024

  11. [19]

    UniBridge: A unified approach to cross- lingual transfer learning for low-resource languages,

    T. Pham, K. Le, and A. T. Luu, “UniBridge: A unified approach to cross- lingual transfer learning for low-resource languages,” inACL, 2024

    2024

  12. [20]

    Resdsql: Decoupling schema linking and skeleton parsing for text-to-sql,

    H. Li, J. Zhang, C. Li, and H. Chen, “Resdsql: Decoupling schema linking and skeleton parsing for text-to-sql,” inAAAI, vol. 37, 2023

    2023

  13. [21]

    CHASE-SQL: Multi-path reasoning and preference optimized candidate selection in text-to-SQL,

    M. Pourreza, H. Li, R. Sun, Y . Chung, S. Talaei, G. T. Kakkar, Y . Gan, A. Saberi, F. Ozcan, and S. O. Arik, “CHASE-SQL: Multi-path reasoning and preference optimized candidate selection in text-to-SQL,” inICLR, 2025

    2025

  14. [22]

    Csc-sql: Corrective self-consistency in text-to- sql via reinforcement learning,

    L. Sheng and S.-S. Xu, “Csc-sql: Corrective self-consistency in text-to- sql via reinforcement learning,” inIJCNLP, 2025

    2025

  15. [23]

    Alpha- SQL: Zero-shot text-to-SQL using monte carlo tree search,

    B. Li, J. Zhang, J. Fan, Y . Xu, C. Chen, N. Tang, and Y . Luo, “Alpha- SQL: Zero-shot text-to-SQL using monte carlo tree search,” inICML, 2025

    2025

  16. [24]

    Grounding natural language to SQL translation with data-based self-explanations,

    Y . Fan, T. Ren, C. Huang, Z. He, and X. S. Wang, “Grounding natural language to SQL translation with data-based self-explanations,” inICDE, 2025

    2025

  17. [25]

    Machine learning inference pipeline execution using pure SQL based on operator fusion,

    Q. Pan, J. Zhi, C. Zhang, C. Xu, Z. Zhang, A. Shao, G. Bao, Q. Cui, X. Chen, and A. Zhou, “Machine learning inference pipeline execution using pure SQL based on operator fusion,” inICDE, 2025

    2025

  18. [26]

    AID-SQL: adaptive in-context learning of text-to-sql with difficulty-aware instruction and retrieval-augmented generation,

    X. Li, Q. Cai, Y . Shu, C. Guo, and B. Yang, “AID-SQL: adaptive in-context learning of text-to-sql with difficulty-aware instruction and retrieval-augmented generation,” inICDE, 2025

    2025

  19. [27]

    Agenticdata: An agentic data analytics system for heterogeneous data,

    J. Sun, G. Li, P. Zhou, Y . Ma, J. Xu, and Y . Li, “Agenticdata: An agentic data analytics system for heterogeneous data,”arXiv preprint arXiv:2508.05002, 2025

    2025 arXiv

  20. [28]

    RefoRCE: A text-to-SQL agent with self-refinement, format restriction, and column exploration,

    M. Deng, A. Ramachandran, C. Xu, L. Hu, Z. Yao, A. Datta, and H. Zhang, “RefoRCE: A text-to-SQL agent with self-refinement, format restriction, and column exploration,” inICLR, 2025

    2025

  21. [29]

    Omnisql: Synthesizing high-quality text-to-sql data at scale,

    H. Li, S. Wu, X. Zhang, X. Huang, J. Zhang, F. Jiang, S. Wang, T. Zhang, J. Chen, R. Shi, H. Chen, and C. Li, “Omnisql: Synthesizing high-quality text-to-sql data at scale,”PVLDB, vol. 18, 2025

    2025

  22. [30]

    Multilingual text-to-sql: Benchmarking the limits of language models with collaborative language agents,

    K. T. Pham, T. H. Nguyen, J. Jo, Q. V . H. Nguyen, and T. T. Nguyen, “Multilingual text-to-sql: Benchmarking the limits of language models with collaborative language agents,” inADC, 2025

    2025

  23. [31]

    Jacobsson

    P. Jacobsson. Getting ai to write good SQL: Text-to-SQL techniques explained. [Online]. Available: https://cloud.google.com/blog/products/ databases/techniques-for-improving-text-to-sql

  24. [32]

    Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task,

    T. Yu, R. Zhang, K. Yang, M. Yasunaga, D. Wang, Z. Li, J. Ma, I. Li, Q. Yao, S. Roman, Z. Zhang, and D. Radev, “Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task,” inEMNLP, 2018

    2018

  25. [33]

    Sparc: Cross-domain semantic parsing in context,

    T. Yu, R. Zhang, M. Yasunaga, Y . C. T. Li, S. Er, E. Li, B. Li, B. Pang, X. V . Lin, Z. Y . Tanet al., “Sparc: Cross-domain semantic parsing in context,” inACL, 2019

    2019

  26. [34]

    Cosql: A conversational text- to-sql challenge towards cross-domain natural language interfaces to databases,

    T. Yu, R. Zhang, M. Yasunaga, B. P. Tan, X. V . Lin, S. Er, Y . C. Li, E. Li, B. Li, W.-t. Y . Chenet al., “Cosql: A conversational text- to-sql challenge towards cross-domain natural language interfaces to databases,” inEMNLP, 2019

    2019

  27. [35]

    Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls,

    J. Li, B. Hui, G. QU, J. Yang, B. Li, B. Li, B. Wang, B. Qin, R. Geng, N. Huo, X. Zhou, C. Ma, G. Li, K. Chang, F. Huang, R. Cheng, and Y . Li, “Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls,” inNeurIPS, 2023

    2023

  28. [36]

    Spider 2.0: Evaluating language models on real-world enterprise text-to-SQL workflows,

    F. Lei, J. Chen, Y . Ye, R. Cao, D. Shin, H. SU, Z. SUO, H. Gao, W. Hu, P. Yin, V . Zhong, C. Xiong, R. Sun, Q. Liu, S. Wang, and T. Yu, “Spider 2.0: Evaluating language models on real-world enterprise text-to-SQL workflows,” inICLR, 2025

    2025

  29. [37]

    Estimating large language model capabilities without labeled test data,

    Y . Fuet al., “Estimating large language model capabilities without labeled test data,” inEMNLP, 2023

    2023

  30. [38]

    SelfCheckGPT: Zero-resource black-box hallucination detection for generative large language models,

    P. Manakul, A. Liusie, and M. Gales, “SelfCheckGPT: Zero-resource black-box hallucination detection for generative large language models,” inEMNLP, 2023

    2023

  31. [39]

    Detecting hallucinations in large language models using semantic entropy,

    S. Farquhar, J. Kossen, L. Kuhn, and Y . Gal, “Detecting hallucinations in large language models using semantic entropy,”Nature, 2024

    2024

  32. [40]

    Autoeval done right: Using synthetic data for model evaluation,

    P. Boyeau, A. N. Angelopoulos, T. Li, N. Yosef, J. Malik, and M. I. Jordan, “Autoeval done right: Using synthetic data for model evaluation,” inICML, 2025

    2025

  33. [41]

    Prediction-powered inference,

    A. N. Angelopoulos, S. Bates, C. Fannjiang, M. I. Jordan, and T. Zrnic, “Prediction-powered inference,”Science, 2023

    2023

  34. [42]

    Leveraging unlabeled data to predict out-of-distribution performance,

    S. Garg, S. Balakrishnan, Z. C. Lipton, B. Neyshabur, and H. Sedghi, “Leveraging unlabeled data to predict out-of-distribution performance,” inNeurIPS, 2021

    2021

  35. [43]

    Predicting with confidence on unseen distributions,

    D. Guillory, V . Shankar, S. Ebrahimi, T. Darrell, and L. Schmidt, “Predicting with confidence on unseen distributions,” inICCV, 2021

    2021

  36. [44]

    Nl2sql-bugs: A benchmark for detecting semantic errors in nl2sql translation,

    X. Liu, S. Shen, B. Li, N. Tang, and Y . Luo, “Nl2sql-bugs: A benchmark for detecting semantic errors in nl2sql translation,” inKDD, 2025

    2025

  37. [45]

    Judging llm-as-a-judge with mt-bench and chatbot arena,

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica, “Judging llm-as-a-judge with mt-bench and chatbot arena,” inNeurIPS, 2023

    2023

  38. [46]

    Tablib: A dataset of 627m tables with context,

    G. Eggert, K. Huo, M. Biven, and J. Waugh, “Tablib: A dataset of 627m tables with context,”arXiv preprint arXiv:2310.07875, 2023

    2023 arXiv

  39. [47]

    KaggleDBQA: Realistic evaluation of text-to-SQL parsers,

    C.-H. Lee, O. Polozov, and M. Richardson, “KaggleDBQA: Realistic evaluation of text-to-SQL parsers,” inACL, 2021

    2021

  40. [48]

    SQLForge: Synthesizing reliable and diverse data to enhance text-to-SQL reasoning in LLMs,

    Y . Guo, D. Jin, S. Ye, S. Chen, J. Yang, and X. Tan, “SQLForge: Synthesizing reliable and diverse data to enhance text-to-SQL reasoning in LLMs,” inACL, 2025

    2025

  41. [49]

    Parsql: Enhancing text-to-sql through sql parsing and reasoning,

    Y . Dai, H. Yang, M. Hao, and P. Chao, “Parsql: Enhancing text-to-sql through sql parsing and reasoning,” inACL, 2025

    2025

  42. [50]

    Llm-based data synthesis and distillation for high-quality text-to-sql training,

    S. Cui, K. Wen, B. Sang, T. Li, Y . Zhang, and H. Gao, “Llm-based data synthesis and distillation for high-quality text-to-sql training,” inICIC, 2025

    2025

  43. [51]

    On efficient multilevel clustering via wasserstein distances,

    V . Huynh, N. Ho, N. Dam, X. Nguyen, M. Yurochkin, H. Bui, and D. Phung, “On efficient multilevel clustering via wasserstein distances,” Journal of Machine Learning Research, 2021

    2021

  44. [52]

    Probabilistic multilevel clustering via composite transportation distance

    N. Ho, V . Huynh, D. Phung, and M. Jordan, “Probabilistic multilevel clustering via composite transportation distance.” PMLR, 2019

    2019

  45. [53]

    Multilevel clustering via Wasserstein means,

    N. Ho, X. Nguyen, M. Yurochkin, H. H. Bui, V . Huynh, and D. Phung, “Multilevel clustering via Wasserstein means,” inICML, 2017

    2017

  46. [54]

    On first-order meta-learning algorithms,

    A. Nichol, J. Achiam, and J. Schulman, “On first-order meta-learning algorithms,”arXiv preprint arXiv:1803.02999, 2018

    2018 arXiv

  47. [55]

    Model-agnostic meta-learning for fast adaptation of deep networks,

    C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learning for fast adaptation of deep networks,” inICML, 2017

    2017

  48. [56]

    Sciencebenchmark: A complex real-world bench- mark for evaluating natural language to sql systems,

    Y . Zhang, J. Deriu, G. Katsogiannis-Meimarakis, C. Kosten, G. Koutrika, and K. Stockinger, “Sciencebenchmark: A complex real-world bench- mark for evaluating natural language to sql systems,”PVLDB, vol. 17, 2024

    2024

  49. [57]

    Ehrsql: A practical text-to-sql benchmark for electronic health records,

    G. Lee, H. Hwang, S. Bae, Y . Kwon, W. Shin, S. Yang, M. Seo, J.- Y . Kim, and E. Choi, “Ehrsql: A practical text-to-sql benchmark for electronic health records,” inNeurIPS, vol. 35, 2022

    2022

  50. [58]

    Gnnevaluator: Evaluating gnn performance on unseen graphs without labels,

    X. Zheng, M. Zhang, C. Chen, S. Molaei, C. Zhou, and S. Pan, “Gnnevaluator: Evaluating gnn performance on unseen graphs without labels,” inNeurIPS, vol. 36, 2023

    2023

  51. [59]

    Qwen3 technical report,

    A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lvet al., “Qwen3 technical report,”arXiv preprint arXiv:2505.09388, 2025

    2025 arXiv

  52. [60]

    The llama 3 herd of models,

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughanet al., “The llama 3 herd of models,”arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  53. [61]

    Deepseek-coder: When the large language model meets programming–the rise of code intelligence,

    D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y . Wu, Y . Liet al., “Deepseek-coder: When the large language model meets programming–the rise of code intelligence,”arXiv preprint arXiv:2401.14196, 2024

    2024 arXiv

  54. [62]

    Xiyan-sql: A novel multi-generator framework for text- to-sql,

    Y . Liu, Y . Zhu, Y . Gao, Z. Luo, X. Li, X. Shi, Y . Hong, J. Gao, Y . Li, B. Dinget al., “Xiyan-sql: A novel multi-generator framework for text- to-sql,”arXiv preprint arXiv:2507.04701, 2025

    2025 arXiv

  55. [63]

    Mixtral of experts,

    A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bam- ford, D. S. Chaplot, D. d. l. Casas, E. B. Hanna, F. Bressandet al., “Mixtral of experts,”arXiv preprint arXiv:2401.04088, 2024

    2024 arXiv

  56. [64]

    Starcoder 2 and the stack v2: The next generation,

    A. Lozhkov, R. Li, L. B. Allal, F. Cassano, J. Lamy-Poirier, N. Tazi, A. Tang, D. Pykhtar, J. Liu, Y . Weiet al., “Starcoder 2 and the stack v2: The next generation,”arXiv preprint arXiv:2402.19173, 2024

    2024 arXiv

  57. [65]

    Code llama: Open foundation models for code,

    B. Roziere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y . Adi, J. Liu, R. Sauvestre, T. Remezet al., “Code llama: Open foundation models for code,”arXiv preprint arXiv:2308.12950, 2023

    2023 arXiv

  58. [66]

    Phi- 4 technical report,

    M. Abdin, J. Aneja, H. Behl, S. Bubeck, R. Eldan, S. Gunasekar, M. Harrison, R. J. Hewett, M. Javaheripi, P. Kauffmannet al., “Phi- 4 technical report,”arXiv preprint arXiv:2412.08905, 2024

    2024 arXiv

  59. [67]

    Yi: Open foundation models by 01. ai,

    A. Young, B. Chen, C. Li, C. Huang, G. Zhang, G. Zhang, G. Wang, H. Li, J. Zhu, J. Chenet al., “Yi: Open foundation models by 01. ai,” arXiv preprint arXiv:2403.04652, 2024

    2024 arXiv

  60. [68]

    Gemma: Open models based on gemini research and technology,

    G. Team, T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre, M. Rivi `ere, M. S. Kale, J. Loveet al., “Gemma: Open models based on gemini research and technology,”arXiv preprint arXiv:2403.08295, 2024

    2024 arXiv

  61. [69]

    Internlm2 technical report,

    Z. Cai, M. Cao, H. Chen, K. Chen, K. Chen, X. Chen, X. Chen, Z. Chen, Z. Chen, P. Chuet al., “Internlm2 technical report,”arXiv preprint arXiv:2403.17297, 2024

    2024 arXiv

  62. [70]

    Athena: An ontology-driven system for natural language querying over relational data stores,

    D. Saha, A. Floratou, K. Sankaranarayanan, U. F. Minhas, A. R. Mittal, and F. Ozcan, “Athena: An ontology-driven system for natural language querying over relational data stores,”PVLDB, vol. 9, 2016

    2016

  63. [71]

    Athena++: Natural language querying for complex nested SQL queries,

    J. Sen, C. Lei, A. Quamar, F. Ozcan, V . Efthymiou, A. Dalmia, G. Stager, A. Mittal, D. Saha, and K. Sankaranarayanan, “Athena++: Natural language querying for complex nested SQL queries,”PVLDB, vol. 13, 2020

    2020

  64. [72]

    Sqlizer: Query synthesis from natural language,

    N. Yaghmazadeh, Y . Wang, I. Dillig, and T. Dillig, “Sqlizer: Query synthesis from natural language,”Proc. ACM Program. Lang., vol. 1, 2017

    2017

  65. [73]

    Don’t judge a language model by its last layer: Contrastive learning with layer-wise attention pooling,

    D. Oh, Y . Kim, H. Lee, H. H. Huang, and H. Lim, “Don’t judge a language model by its last layer: Contrastive learning with layer-wise attention pooling,” inCOLING, 2022

    2022

  66. [74]

    SimCSE: Simple contrastive learning of sentence embeddings,

    T. Gao, X. Yao, and D. Chen, “SimCSE: Simple contrastive learning of sentence embeddings,” inEMNLP, 2021

    2021

  67. [75]

    PromptBERT: Improving BERT sentence embeddings with prompts,

    T. Jiang, J. Jiao, S. Huang, Z. Zhang, D. Wang, F. Zhuang, F. Wei, H. Huang, D. Deng, and Q. Zhang, “PromptBERT: Improving BERT sentence embeddings with prompts,” inEMNLP, 2022

    2022

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.