Pith. sign in

REVIEW 13 cited by

Evaluating the Text-to-SQL Capabilities of Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2204.00498 v1 pith:SMWCGYZL submitted 2022-03-15 cs.CL cs.DBcs.LG

Nitarshan Rajkumar , Raymond Li , Dzmitry Bahdanau This is my paper · ORCID
classification cs.CLcs.DBcs.LG
keywords codexcapabilitiesexampleslanguagemodelsperformtext-to-sqlanalyze
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We perform an empirical evaluation of Text-to-SQL capabilities of the Codex language model. We find that, without any finetuning, Codex is a strong baseline on the Spider benchmark; we also analyze the failure modes of Codex in this setting. Furthermore, we demonstrate on the GeoQuery and Scholar benchmarks that a small number of in-domain examples provided in the prompt enables Codex to perform better than state-of-the-art models finetuned on such few-shot examples.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 13 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 52 citations worldwide. Full citation record

  1. Natural Language Access to Domain-Specific Metadata: A Reusable Framework for LLM Query Generation

    cs.DB 2026-07 conditional novelty 6.0 of 10

    With an OWL ontology using readable names and annotations, a local Qwen3.6-27B model generates SPARQL for 21 expert questions with 100% accuracy, versus 57% for auto-generated SQL.

  2. SDE-SQL: Enhancing Text-to-SQL Generation in Large Language Models via Self-Driven Exploration with SQL Probes

    cs.CL 2025-06 conditional novelty 6.0 of 10

    SDE-SQL improves text-to-SQL accuracy by having the model generate and execute exploratory SQL probes to learn database contents before and while writing the final query.

  3. Knowledge Base Construction for Knowledge-Augmented Text-to-SQL

    cs.CL 2025-05 conditional novelty 6.0 of 10

    KAT-SQL constructs a reusable knowledge base for text-to-SQL by expanding training data with LLM-generated knowledge and retrieving/refining the best entries for each query.

  4. DCG-SQL: Enhancing In-Context Learning for Text-to-SQL with Deep Contextual Schema Link Graph

    cs.AI 2025-05 conditional novelty 6.0 of 10

    DCG-SQL retrieves text-to-SQL demonstrations by embedding a question-to-schema link graph, improving execution accuracy on Spider by up to about 10 points over random demonstrations on small LLMs.

  5. ProgramTab: Boosting Table Reasoning of LLMs via Programmatic Paradigm

    cs.CL 2026-07 conditional novelty 5.0 of 10

    A Python-preprocessing + CoT-SQL pipeline lets LLMs reason over large, inconsistently formatted tables more accurately than prior prompting baselines.

  6. Confidence Estimation for Text-to-SQL in Large Language Models

    cs.CL 2025-08 unverdicted novelty 4.0 of 10

    Consistency-based methods are the most reliable confidence signal for text-to-SQL in black-box LLMs, and executing queries against a database adds a useful correctness signal.

  7. SQLord: A Robust Enterprise Text-to-SQL Solution via Reverse Data Generation and Workflow Decomposition

    cs.DB 2025-07 unverdicted novelty 4.0 of 10

    SQLord builds a domain-specific text-to-SQL pipeline using reverse-generated query-SQL training pairs, workflow decomposition, and an LLM-based judge, reporting consistent gains over GPT-4 and open-source baselines.

  8. Interactive Text-to-SQL via Expected Information Gain for Disambiguation

    cs.DB 2025-07 reject novelty 4.0 of 10

    An interactive text-to-SQL framework selects clarification questions by expected information gain over a distribution of candidate SQL queries.

  9. Exploring the Landscape of Text-to-SQL with Large Language Models: Progresses, Challenges and Opportunities

    cs.CL 2025-05 conditional novelty 4.0 of 10

    A systematic review organizing LLM-based text-to-SQL methods into pre-processing, in-context learning, fine-tuning, and post-processing paradigms, with a catalog of datasets, metrics, challenges, and future directions.

  10. Meta-aware Learning in text-to-SQL Large Language Model

    cs.AI 2025-05 conditional novelty 4.0 of 10

    Combining schema, chain-of-thought, metadata knowledge, and tokenized prompt structures during fine-tuning improves text-to-SQL execution accuracy on private business databases compared to schema-only fine-tuning.

  11. Chatting with your ERP: A Recipe

    cs.AI 2025-07 conditional novelty 3.0 of 10

    A dual-agent LLM system with a reasoner and critic achieves up to 10/11 expert-validated SQL accuracy on a real ERP database, but the evaluation lacks a baseline and released artifacts.

  12. THOR: Transformer Heuristics for On-Demand Retrieval

    cs.DB 2025-07 reject novelty 3.0 of 10

    THOR is an enterprise Text-to-SQL system using LLM agents, schema injection, and a five-retry self-correction loop, evaluated only on seven anecdotal prompts.

  13. LLM Inference Enhanced by External Knowledge: A Survey

    cs.CL 2025-05 conditional novelty 2.0 of 10

    A survey of methods that enhance LLM inference by integrating external structured knowledge from tables and knowledge graphs.

Pith tools