Automatic Metadata Extraction for Text-to-SQL

Divesh Srivastava; Parisa Ghane; Theodore Johnson; Vladislav Shkapenyuk

REVIEW 5 major objections 6 minor 2 cited by

Automatic Metadata Extraction for Text-to-SQL

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that automatically extracted database profiles, summarized by an LLM into field descriptions, improve text-to-SQL accuracy more than the human-written metadata that benchmarks ship with.

desk verdict A useful engineering report on automatic metadata for text-to-SQL, but its headline claim that profiling metadata beats SME metadata rests on a single un-replicated 500-question run. read the letter →

arxiv 2505.19988 v2 pith:OUBEADN4 submitted 2025-05-26 cs.DB

Vladislav Shkapenyuk , Divesh Srivastava , Theodore Johnson , Parisa Ghane This is my paper

classification cs.DB

keywords text-to-SQLdatabaseprofilingmetadataextractionschemalinkingqueryloganalysisSQL-to-textgenerationfew-shotpromptingBIRDbenchmark

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the hardest part of text-to-SQL, understanding what is actually in a database, can be automated well enough to replace human-written documentation. The core evidence is an ablation on BIRD's MiniDev: field descriptions generated by profiling the data and having an LLM summarize the profiles raise accuracy to 61.2% without hints, above the 59.6% obtained with the benchmark's own human-supplied metadata, and fusing the two reaches 63.2%. The paper also claims that mining query logs recovers roughly 25% of the equality join paths that are used in real queries but missing from the schema, and that LLM SQL-to-text generation produces few-shot questions rated as good as or better than human annotations. If these claims hold, automatic metadata extraction could remove the expensive subject-matter-expert bottleneck for applying text-to-SQL to real databases.

What carries the argument

Two mechanisms carry the argument. First, the profile-to-description pipeline: standard database profiling statistics are converted into a textual profile, then an LLM summarizes it in the context of the table and column names, yielding a short field description for schema linking and a long one for SQL generation. Second, a SQL-first schema linking algorithm: instead of asking the LLM to name relevant fields directly, the system generates candidate SQL over several schema variants, collects the fields and literals the SQL actually references, and uses locality-sensitive hashing over sampled field values to match question literals to the fields that contain them, revising the SQL if a literal is not yet matched. This sidesteps what the paper calls task alignment: LLMs are reliable at writing SQL but unreliable at direct schema-selection tasks.

What would settle it

Run the MiniDev ablation with every LLM-generated field summary hand-checked against the actual database values and the benchmark's hints, and count summaries that contain a false or misleading statement that the generated SQL then relies on; if removing or correcting those summaries raises the profile-only accuracy enough to erase the gap over the Bird-metadata score, the central claim would collapse. A coarser check is to rerun Table 1 on the full BIRD dev set rather than MiniDev: if profile metadata does not beat Bird metadata there, the result is an artifact of the 500-question subset.

Watch

Extended reading notes

Core claim

The central discovery, on the paper's own terms, is that the data itself is a better source of field semantics than the documentation that benchmarks ship with. A mechanical profiling pass over a table, NULL counts, distinct counts, min and max, top-k values, value shapes, and minhash sketches, can be rendered as an English-language profile, and an LLM can condense that profile into short and long field descriptions that capture meaning. Feeding those descriptions to a text-to-SQL LLM improves accuracy more than feeding it the benchmark's metadata, and combining both is best. The paper further shows that query logs, when parsed and resolved through subqueries, expose join paths, multi-field joins, computed joins, and named business formulas that the schema does not document, and that LLM-generated questions from SQL can replace human-annotated few-shot examples.

Load-bearing premise

The load-bearing premise is that the LLM-generated field summaries are accurate enough that their occasional mistakes do not erase their benefit; the paper reports cases where profiling metadata caused wrong predicates but never measures how often such errors occur.

Editorial extensions

If this is right

If profile-derived metadata beats supplied metadata, text-to-SQL deployments can bootstrap from raw database contents instead of waiting for documentation.
Fusing automatically extracted metadata with any existing documentation gives the best accuracy, so adding profiling to current pipelines is worthwhile even when docs exist.
Query-log feature extraction can supply join paths and business formulas that schemas omit; systems with access to logs should mine them.
SQL-to-text can invert the annotation process, turning an existing query log into few-shot training pairs without hiring SQL-writing annotators.
Profile summaries are not harmless: they can suggest wrong predicates, so automatic metadata needs validation or monitoring.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the advantage of profiling over human metadata generalizes beyond BIRD, then documentation quality may be less important than data statistics for LLM querying; this would shift engineering effort toward profiling and index infrastructure.
The 25% undocumented-join result suggests that in older industrial databases, query logs may be the only record of how tables are actually meant to be joined; mining them could matter more as schemas grow by accretion.
A direct test of the paper's logic is to apply the same pipeline to a database with deliberately cryptic field names and no hints; if profile summaries still beat hand-written docs there, the result is about data, not about BIRD's readable schemas.
The SQL-to-text ratings imply that the human question/SQL pairs in benchmarks are themselves noisy; automated generation might produce cleaner few-shot corpora than crowd-sourced annotation.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things to know before you read it. The useful core is the schema-linking idea: instead of asking the LLM to classify relevant columns, generate SQL under several schema/profile variants and harvest the fields the SQL touches, with a literal-matching correction step. That is a sensible application of task alignment, and it is not in CHESS, Chase, or Distillery as far as I can tell. The second thing is that the paper's central claim—profiling metadata beats BIRD's SME-supplied metadata—is not actually established. On MiniDev, the no-hint gap is 1.6 points (61.2 vs 59.6 on 500 questions, roughly 8 questions), with a pipeline that is explicitly stochastic: random seeds, randomized schema order, random tie-breaks. There is no repeated run, no paired test, no error bar. The stress-test note is right: that gap could easily be noise. The fused-vs-Bird gap (63.2 vs 59.6) is larger, but still unreplicated.

What the paper does well: the query log analysis experiments on BIRD dev are concrete and interesting. The finding that 25% of equality joins actually used by queries are undocumented or need manual extraction is real, and the identification of formulas and multi-field joins is a good demonstration. The SQL-to-text study is small (83 questions) and subjective, but the authors put the full rating appendix in the paper, which is more transparency than usual; the examples show fused metadata helping substantially.

Soft spots, in proportion. No code or prompts are released, so the system is not reproducible from the paper. The accuracy of the LLM-generated field summaries is never measured; the paper's own Section 4.1.1 shows profile metadata causing wrong predicates (Q356, Q1260), so the reliability assumption is genuine. SQL-to-text grading is self-assessed with no inter-annotator agreement. The leaderboard #1 claims are time-dependent and come from a complex system that does not isolate the metadata contribution.

Overall, this is a solid, honest engineering report from people who clearly know real databases. The direction is right—under-documented schemas are the bottleneck for text-to-SQL. But the headline comparative claim needs significance testing or repeated runs before I would rely on it. It deserves peer review; a good referee should ask for those experiments. I would not cite it for the claim that profiling beats SME metadata, but I might cite it for the schema-linking-via-SQL-generation idea.

Referee Report

5 major / 6 minor

Summary. This paper argues that understanding database contents is the bottleneck in text-to-SQL and proposes automatic metadata extraction as a replacement for expensive SME-written metadata. It combines database profiling with LLM-based field summarization, develops a schema-linking algorithm that harvests fields from LLM-generated SQL, applies query-log analysis to discover undocumented join constraints, and uses LLM SQL-to-text generation to produce few-shot examples. Evaluation on BIRD MiniDev (Table 1) reports accuracy of 49.8% without metadata, 59.6% with Bird metadata, 61.2% with profiling metadata, and 63.2% with fused metadata, supporting the central claim that profiling metadata helps more than Bird-supplied metadata. The paper also reports #1 BIRD leaderboard positions and 25% undocumented equality constraints found by query-log analysis.

Significance. If substantiated, the main claim would be practically useful: automatic profiling could substitute for part of the SME metadata effort in text-to-SQL. The paper's strengths are its use of a public benchmark, its concrete worked examples of profile-to-summary generation, its explicit acknowledgment and discussion of failure cases (Q356 and Q1260 in Section 4.1.1), and its falsifiable leaderboard claim. However, the central comparison is currently supported by a single stochastic run without error bars, and the SQL-to-text evaluation is self-rated, so the strength of the evidence is below what the conclusions require. The query-log analysis results are suggestive and align with known industrial experience, but the paper does not release code or prompts, which limits reproducibility.

major comments (5)

[4.1, Table 1] The paper's central claim that profiling metadata gives a bigger accuracy boost than Bird-supplied metadata rests on a single 500-question run in which profiling scores 61.2% vs. Bird's 59.6%, a 1.6-point difference (about 8 questions), while Section 4 describes candidate generation with random seeds, randomized field order, and random tie-breaking, and Section 4.1.1 itself attributes errors to 'instability of LLM answers.' Without repeated runs, confidence intervals, or a paired significance test, the 1.6-point gap could be run-to-run noise, so this load-bearing comparison is not yet established.
[4.1.1] The paper documents two cases (Q356 and Q1260) where profile-derived summaries misled the generated SQL, but it provides no error-rate analysis for the LLM-generated field summaries. Since the claimed benefit of profiling over SME metadata depends on these summaries being reliable on average, the absence of any frequency estimate for misleading summaries is a load-bearing gap; a small error-rate analysis on a sample of fields would materially address it.
[6.1, Table 3] The SQL-to-text evaluation uses the authors' own subjective ratings ('Our grading is subjective'), with no second annotator, no blinding, and no inter-annotator reliability measure, yet the conclusion is that fused metadata yields questions 'significantly better than' human annotations. The ordinal ratings and 83-question sample also lack any statistical test; as presented, this evidence supports only a qualitative observation.
[1, 4] The BIRD leaderboard results are reported as support for the approach, but they are not a controlled comparison for the metadata claim: the leaderboard runs use different test data, the optional hints, and a different system configuration, and they lack a no-metadata baseline. They should be presented as auxiliary evidence, not as confirmation of Table 1's metadata comparison.
[3, Table 2] The schema-linking contribution is evaluated only through the same stochastic pipeline, and Table 2 shows a 5.8-point gap between the proposed linking (63.2%) and perfect schema linking (69.0%). Because the link harvesting depends on the quality of LLM-generated SQL, the method's ceiling is tied to the LLM's SQL ability; the paper should state this dependency explicitly and ideally report results with a second model.

minor comments (6)

[4.1] The text near Table 1 reads 'GTP-4o' in one place; this should be 'GPT-4o'.
[3.1] In the schema sample, 'Field from.`County Code`' appears to be a typo for 'frpm.`County Code`'.
[5.4.1] The 25% undocumented-equality-constraint claim would be easier to check if the exact numerator and denominator were stated consistently: the text alternates between 27/29 and 30/32 discovered constraints depending on whether hand-filled foreign keys are excluded.
[Appendix] The supplied question for question 937 contains the typo 'AustChineseralian Grand Prix'; if this is a faithful transcription from BIRD, it should be marked as such.
[References] Reference [CLHY+22] appears to cite a 2018 paper with a 2022 label, and [MAJM24] contains 'Jaroslawwicz' (likely 'Jaroslawicz'); these should be corrected.
[1.1 and 4] No code, profiles, or prompts are released, which limits reproducibility of the leaderboard claims; a supplementary appendix with the exact prompts and profile examples would strengthen the paper.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the accuracy claims are empirical measurements against the external BIRD benchmark, and the cited prior work by the authors is contextual rather than load-bearing.

full rationale

The paper's central claims are experimental evaluations on the external BIRD MiniDev set (Table 1) and the BIRD dev query set (Section 5.4), so they are measured outcomes rather than derivations from assumptions. The schema-linking scheme harvests fields from LLM-generated SQL (Section 3), but the evaluation compares execution accuracy against the gold SQL, so the result is not equivalent to the method's inputs by construction. The authors cite their own prior work ([DJMS02], [GJ14], [YPS09]) only for background motivation and related techniques; no uniqueness theorem or load-bearing result is imported from those citations. The SQL-to-text grading in Section 6.1 is explicitly subjective ('Our grading is subjective'), which is a methodological limitation that could inflate the reported advantage over human annotations, but it does not make the claim equivalent to its inputs by construction. There are no fitted parameters renamed as predictions and no ansatz smuggled in via citation; consequently, no circular step meeting the required evidence standard is present.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claims rest on the BIRD benchmark as a proxy, on the accuracy of LLM-generated field summaries, on the task-alignment assumption that justifies the schema linker, and on the stability of small single-model evaluations. No numerical free parameters are fit to the target result, but several hyperparameters (N=10000, retries, candidate counts) are hand-chosen. No new physical entities are introduced.

free parameters (5)

LSH sample size N = 10000
Number of distinct values sampled per field to build the string similarity index (Section 3, preprocessing step 1).
MaxRetry = not reported
Retry limit for revising SQL to use fields that contain literals (Section 3 algorithm step 2e).
Number of schema variants = 5
Five profile and schema combinations queried per question in schema linking (Section 3 step 2).
Number of candidate SQL queries = 3
Three candidates generated per question for majority voting (Section 4).
Number of few-shot examples = 8
Eight masked questions retrieved from a vector database as few-shot examples (Section 4).

assumptions (5)

domain assumption BIRD benchmark's supplied metadata and per-question hints are a fair proxy for human SME metadata and for query-guidance in practice.
The paper's comparisons treat BIRD metadata and hints as baselines, while also arguing hints are unrealistic; the evaluation rests on this proxy (Sections 1, 4.1).
domain assumption LLM-generated short and long field summaries are sufficiently accurate to improve SQL generation.
The central accuracy gains in Table 1 depend on summaries being mostly correct; the paper shows counterexamples (Q356, Q1260) without quantifying error rates (Sections 2.1, 4.1.1).
ad hoc to paper Task alignment: LLMs are poor at direct schema linking but good at generating SQL from which relevant fields can be harvested.
This empirical claim justifies the novel schema linking algorithm; it is asserted from observations, not independently benchmarked (Section 3).
domain assumption The MiniDev 500-question sample and single-LLM (GPT-4o) setting produce stable accuracy rankings.
Tables 1 and 2 report single numbers without error bars or significance tests; the 1.6-point gap between profiling and Bird metadata may be noise (Section 4.1).
domain assumption Query log features extracted from BIRD dev queries are representative of features that would appear in real query logs for unseen questions.
Section 5.4 uses dev queries as a stand-in query log; no test-set query log exists, so the benefit for deployment is inferred.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Automatic Metadata Extraction for Text-to-SQL." pith.science (2026). https://pith.science/paper/OUBEADN4

@misc{pith2026250519988,
  author       = {Pith},
  title        = {Pith review of: Automatic Metadata Extraction for Text-to-SQL},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OUBEADN4}},
  note         = {Machine review of arXiv:2505.19988}
}

read the original abstract

Large Language Models (LLMs) have recently become sophisticated enough to automate many tasks ranging from pattern finding to writing assistance to code generation. In this paper, we examine text-to-SQL generation. We have observed from decades of experience that the most difficult part of query development lies in understanding the database contents. These experiences inform the direction of our research. Text-to-SQL benchmarks such as SPIDER and Bird contain extensive metadata that is generally not available in practice. Human-generated metadata requires the use of expensive Subject Matter Experts (SMEs), who are often not fully aware of many aspects of their databases. In this paper, we explore techniques for automatic metadata extraction to enable text-to-SQL generation. We explore the use of two standard and one newer metadata extraction techniques: profiling, query log analysis, and SQL-to text generation using an LLM. We use BIRD benchmark [JHQY+23] to evaluate the effectiveness of these techniques. BIRD does not provide query logs on their test database, so we prepared a submission that uses profiling alone, and does not use any specially tuned model (we used GPT-4o). From Sept 1 to Sept 23, 2024, and Nov 11 through Nov 23, 2024 we achieved the highest score both with and without using the "oracle" information provided with the question set. We regained the number 1 spot on Mar 11, 2025, and are still at #1 at the time of the writing (May, 2025).

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Beyond Static Rules: Automated Discovery of Latent Vulnerabilities in Text-to-SQL
cs.CL 2026-07 conditional novelty 6.5 of 10

An evolving Vulnerability Codex plus hypothesis-driven perturbations exposes latent Text-to-SQL failures in LLMs far better than fixed expert rules, with transferable patterns and early remediation gains.
APEX-SQL: Talking to the data via Agentic Exploration for Text-to-SQL
cs.DB 2026-02 conditional novelty 5.0 of 10

An agentic text-to-SQL pipeline using hypothesis-verification exploration achieves 70.7% execution accuracy on BIRD-Dev and 51.0% on Spider 2.0-Snow.

Reference graph

Works this paper leans on

28 extracted references · 15 canonical work pages · cited by 2 Pith papers

[1]

Reason: not picking up on meaning of A15

Supplied question: In the branch where the second-highest number of crimes were committed in 1995 occurred, how many male clients are there? Base generated questions: Q1:: How many male clients are there in the district with the second-highest value of A15? Q2:: How many men live in the district with the second-highest A15? Base question rating: good-. Re...

work page 1995
[5]

What is the difference in the number of superheroes between DC Comics and Marvel Comics?

Finally, it subtracts the total count for 'Marvel Comics' from the total count for 'DC Comics'. The question that can be answered by the given SQL code is: "What is the difference in the number of superheroes between DC Comics and Marvel Comics?" Step 2: "How many more superheroes does DC Comics have compared to Marvel Comics?" Base question rating: good+...

work page 2013
[6]

Maamari, F

[MAJM24] K. Maamari, F. Abubaker, D. Jaroslawwicz, A. Mhedhni. The Death of Schema Linking? Text-to-SQL in the Age of Well- Reasoned Language Models. aXiv:2408.07702

arXiv
[7]

[NZZR+23] L. Nin, Y. Zhao, W. Zou, N. Ri, J. Tae, E. Zhang, A. Cohan, D. Daved. Enhancing few-shot text-to-SQL capabilities of large language models: A study on prompt design strategies. CoRR abs/2305.12586

work page Pith review arXiv
[9]

Pourreza et al

[PLSC+24b] M. Pourreza et al. CHASE-SQL: Multi-Path Reasoning and Preference Optimized Candidate Selection in Text-to-SQL. arXiv:2410.01943v1

arXiv
[10]

Parnami, M

[PL22] A. Parnami, M. Lee. Learning from few examples: a summary of approaches to few-shot learning. arXiv:2203.04291,

arXiv
[11]

October Meeting

Base generated questions: Q1:: At what distinct times did transactions occur at gas stations belonging to chain ID 11? Q2:: When did gas stations in chain 11 have transactions? Base question rating: good. Bird only generated questions: Q1:: What are the distinct times when transactions occurred at gas stations belonging to chain ID 11? Q2:: What times did...

work page 2012
[12]

Qin et al

[QHWY+22] B. Qin et al. A survey on text-to-SQL parsing: Concepts, methods, and future directions. arXiv:2208.13629,

arXiv

Show all 28 references

[13]

[TPCM+24] S

Also in arXiv:2405.15307. [TPCM+24] S. Talaei, M. Pourreza, Y-C Chang, A. Mirhoseini, A. Saberi. CHESS: Contextual Harnessing for Efficient SQL Synthesis. Arvix 2405.16755v2

arXiv
[14]

Wang at al

[WRYL24] B. Wang at al. Mac-sql: A multi-agent collaborative framework for text-to-sql. CoRR, abs/2312.11242,

arXiv
[15]

[XLS17] X. Xu, C. Liu, D. Song. SQLNet: Generating Structured Queries From Natural Language Without Reinforcement Learning. arXiv:1711.04436

arXiv
[17]

Yu, et al

[YZYY+18] T. Yu, et al. Spider: a large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. Proc. Conf. on Empirical Methods in Natural Language Processing. 2018 [ZM96] J.M. Zelle, R.J. Mooney. Learning to parse database queries usin...

2018
[19]

Question 1471: SQL: SELECT CAST(SUM(IIF(Currency = 'EUR', 1, 0)) AS FLOAT) / SUM(IIF(Currency = 'CZK', 1, 0)) AS ratio FROM customers Supplied question: What is the ratio of customers who pay in EUR against customers who pay in CZK? Base generated questions: Q1:: What is the r...

2013
[23]

= '2016-06-23' AND t2.overall_rating = 77 ORDER BY t1.birthday ASC LIMIT 1 Supplied question: For the players who had a 77 points overall rating on 2016/6/23, who was the oldest? Give the name of the player. Base generated questions: Q1:: Who is the earliest born player with a...

2016
[24]

Aaron"? Q2:: How many

BETWEEN '2007- 02-22' AND '2016-04-21' Supplied question: What was the average overall rating for Marko Arnautovic from 2007/2/22 to 2016/4/21? Base generated questions: Q1:: What is the average overall rating of the player named 'Marko Arnautovic' in the Player_Attributes tab...

2007
[28]

It then creates a subquery (alias as T) and from this subquery, it selects the 'bond_type'. Question that can be answered by the given SQL code: What is the type of bond that has the highest number of entries in the bond table? Step 2: What is the most common bond type? Base q...

1991
[55]

have high PT levels? Base question rating: good. Bird only generated questions: Step 1: What percentage of female patients over the age of 55 have a PT (Prothrombin Time) value of 14 or higher? Step 2: What is the percentage of women older than 55 with a high PT level? Bird on...

2013
[1000]

Super Strength

< 120 Supplied question: How many French drivers who obtain the laptime less than 02:00.00? Base generated questions: Q1:: How many French drivers have recorded a lap time of less than two minutes in the lapTimes database? Q2:: How many French drivers completed a lap in under ...

1980
[1996]

Zhong, C

[ZXS17] V. Zhong, C. Xiong, R. Socher. Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning. arXiv:1709.00103

arXiv
[1997]

[CLHY+22] R. Cai, B. Xu, Z. Zhang, X. Yang, Z. Li, Z. Liang. An encoder-decoder framework translating natural language to database queries. In Proc. Twenty-Seventh International Joint Conference on Artificial Intelligence, 2018 [DJMS02] T. Dasu, T. Johnson, S. Muthukrishnan, V...

2018
[2002]

[DZGM+23] X. Dong, C. Zhang, Y. Ge, Y. Mo Y. Gao, I. Chen, J. Lin, D. Lou. C#: Zero-shot text-to-SQL with chatgpt. CORR, ans/2307.07306

arXiv
[2005]

Base generated questions: Q1:: What are the names of the circuits and the corresponding race names for races that took place in September 2005, along with the locations of those circuits? Q2:: Which races and circuits occurred in September 2005, and where were they located? Ba...

2005
[2014]

Gao et al

[GLLS+24] Y. Gao et al. XiYan-SQL: A Multi-Generator Ensemble Framework for Text-to-SQL. arXiv:2411.08599

arXiv
[2015]

Base generated questions: Q1:: What are the distinct ages of players who had a sprint speed of 97 or higher recorded between the years 2013 and 2015? Q2:: What ages did the fastest players have during 2013-2015? Base question rating: good. Bird only generated questions: Q1:: W...

2013
[2017]

Yang, C.M

[YPS09] X. Yang, C.M. Procopiuc, D. Srivastava. Recommending Join Queries via Query Log Analysis. ICDE 2009: 864-975

2009
[2022]

Pourreza, D

[PR23] M. Pourreza, D. Rafiei. Din-sql: Decomposed in-context learning of text-to-sql with self-correction. CoRR, abs/2304.11015,

arXiv
[2023]

[LPKP24] D. Lee, C. Park, J. Kim, N. Tang. Mcs-sql: Leveraging multiple prompts and multiple-choice selection for text-to-sql generation. CoRR, abs/2405.07467,

arXiv
[2024]

Gao et al

[GWLS+23] D. Gao et al. Text-to-sql empowered by large language models: A benchmark evaluation. CoRR, abs/2308.15363,

arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Reason: not picking up on meaning of A15

Supplied question: In the branch where the second-highest number of crimes were committed in 1995 occurred, how many male clients are there? Base generated questions: Q1:: How many male clients are there in the district with the second-highest value of A15? Q2:: How many men live in the district with the second-highest A15? Base question rating: good-. Re...

work page 1995

[2] [5]

What is the difference in the number of superheroes between DC Comics and Marvel Comics?

Finally, it subtracts the total count for 'Marvel Comics' from the total count for 'DC Comics'. The question that can be answered by the given SQL code is: "What is the difference in the number of superheroes between DC Comics and Marvel Comics?" Step 2: "How many more superheroes does DC Comics have compared to Marvel Comics?" Base question rating: good+...

work page 2013

[3] [6]

Maamari, F

[MAJM24] K. Maamari, F. Abubaker, D. Jaroslawwicz, A. Mhedhni. The Death of Schema Linking? Text-to-SQL in the Age of Well- Reasoned Language Models. aXiv:2408.07702

arXiv

[4] [7]

[NZZR+23] L. Nin, Y. Zhao, W. Zou, N. Ri, J. Tae, E. Zhang, A. Cohan, D. Daved. Enhancing few-shot text-to-SQL capabilities of large language models: A study on prompt design strategies. CoRR abs/2305.12586

work page Pith review arXiv

[5] [9]

Pourreza et al

[PLSC+24b] M. Pourreza et al. CHASE-SQL: Multi-Path Reasoning and Preference Optimized Candidate Selection in Text-to-SQL. arXiv:2410.01943v1

arXiv

[6] [10]

Parnami, M

[PL22] A. Parnami, M. Lee. Learning from few examples: a summary of approaches to few-shot learning. arXiv:2203.04291,

arXiv

[7] [11]

October Meeting

Base generated questions: Q1:: At what distinct times did transactions occur at gas stations belonging to chain ID 11? Q2:: When did gas stations in chain 11 have transactions? Base question rating: good. Bird only generated questions: Q1:: What are the distinct times when transactions occurred at gas stations belonging to chain ID 11? Q2:: What times did...

work page 2012

[8] [12]

Qin et al

[QHWY+22] B. Qin et al. A survey on text-to-SQL parsing: Concepts, methods, and future directions. arXiv:2208.13629,

arXiv