Beyond Text-to-SQL: Can LLMs Really Debug Enterprise ETL SQL?

Jing Ye; Victor Ma; Xing Chen; Yang Gao; Yiwen Duan; Yonghong Yu

REVIEW 4 major objections 5 minor 1 cited by

Beyond Text-to-SQL: Can LLMs Really Debug Enterprise ETL SQL?

T0 review · 4 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Enterprise SQL debugging is unsolved for LLMs: on a new benchmark of 985 long, real-world-style ETL repair tasks, the best model fixes 36% of syntax errors and 32% of semantic errors, and most models stay under 20%.

desk verdict A genuinely useful benchmark for enterprise SQL debugging, but the headline numbers rest on an unvalidated execution-free equivalence oracle; worth reviewing seriously, with the metric validation as the gating issue. read the letter →

arxiv 2601.18119 v2 pith:4YWVNYVD submitted 2026-01-26 cs.AI

Jing Ye , Yiwen Duan , Yonghong Yu , Victor Ma , Yang Gao , Xing Chen This is my paper · ORCID

classification cs.AI

keywords SQLdebuggingETLLLMevaluationbenchmarkconstructionsyntaxerrorssemanticgraphisomorphismHive/Spark

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Enterprise SQL debugging is more than an extension of text-to-SQL. This paper introduces Squirrel Benchmark, a set of 985 repair tasks built from production-style ETL scripts averaging more than 140 lines, with an automated pipeline that injects realistic bugs and an execution-free scoring method based on structural graph equivalence. The paper's central finding is that current LLMs largely fail at this task: the best model, Claude-4-Sonnet, repairs 36.46% of syntax-bugged queries and 32.17% of semantic-bugged queries, while most of the nearly 30 tested models fall below 20%. A sympathetic reader would take this as evidence that robust, multi-step SQL debugging in enterprise settings is an open capability gap that current benchmarks were not measuring.

What carries the argument

Two mechanisms carry the argument. First, reverse-engineering bug construction: correct SQL seeds are sampled from production logs, a taxonomy built from 268 real error cases guides minimal-change bug injection, and an attack–defense filtering pass (multiple models attempt the tasks; hard cases get human review) sets final difficulty. Second, execution-free evaluation: instead of running repaired SQL on test data, the paper compiles predicted and reference queries into canonical logical plans and scores a repair correct when the two graphs are isomorphic (Graph Match), alongside exact string match and edit-distance progress. The graph isomorphism check is the load-bearing component: it deter

What would settle it

Take a random sample of Squirrel-Semantic tasks, execute the reference and the best model's repairs on small test tables seeded with edge-case rows (NULLs, duplicate keys, no-match joins), and measure the agreement between Graph Match's correct/incorrect decision and actual output equivalence; any material disagreement rate invalidates the reported percentages.

Watch

Extended reading notes

Core claim

The paper claims that LLMs currently cannot reliably debug long enterprise ETL SQL, and that this failure is unseen by standard text-to-SQL benchmarks. It constructs a benchmark where each task is a 140+-line Hive/Spark SQL script with either an explicit syntax error or a silent semantic error, and where a repair is judged correct only if its logical query plan is isomorphic to the reference plan. Across nearly 30 models, including reasoning and SQL-specialized systems, the best score is 36.46% (syntax) and 32.17% (semantic), with most models below 20%; code-generation and text-to-SQL specialists drop to single digits. The paper also reports that tuned fine-tuning and an agentic loop improve

Load-bearing premise

The paper's headline numbers assume the execution-free Graph Match test, which compares normalized query plans by graph isomorphism, correctly decides semantic correctness without ever running the repaired SQL on data that would expose the bug.

Editorial extensions

If this is right

If the benchmark numbers hold, LLM-based SQL debugging tools cannot be trusted for unattended repair of enterprise ETL scripts; human review in the loop remains necessary.
Both syntax and semantic repair rates stay below 40% even for the strongest model, so gains from the next generation of reasoning models should show up first as higher Graph Match on these 985 tasks.
The finding that a 7B model fine-tuned on minimal-edit debugging pairs beats a 32B base model (33.18% vs 20.26% Graph Match on syntax) points to synthetic bug-injection data as a practical lever before scale.
Reasoning models' high Modify-Better scores indicate they move scripts closer to the fix but rarely land it in one attempt, suggesting iterative agent loops, rather than single-shot generation, are the right interface for SQL repair.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The construction pipeline is dialect-agnostic, so the same reverse-engineering workflow could produce sibling benchmarks for other SQL dialects; if scores stay low, the difficulty is general, not an artifact of Hive/Spark.
The single-reference Graph Match metric may penalize valid alternative repairs in the semantic tasks; building a small execution-validated subset would show whether models are failing or the tester is.
If execution-free graph scoring is validated, the same evaluation pattern could extend to other expensive-to-run code-repair domains (shell pipelines, data-prep scripts), where running code in production is costly or unsafe.
Because the benchmark's construction and much of its evaluation relied on one model family, a head-to-head with benchmarks built by other generators would reveal whether the reported ceiling is a model-family blind spot.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a real contribution with one load-bearing soft spot. The benchmark is new in scale and construction: 985 tasks averaging over 140 lines in Hive/Spark dialect, a three-level bug taxonomy from 268 production logs, a reverse-engineering pipeline for minimal bug injection, and a 30-model evaluation showing most models under 20% GM. The pipeline is described well enough to be rebuilt, and the SFT and agent baselines are a useful addition. The paper deserves a serious referee.

The central direction—LLMs are weak at long, nested enterprise ETL debugging—is plausible and survives scrutiny. The numbers are directionally consistent across model families. But the yardstick is unvalidated. The GM score is exact graph isomorphism over Calcite-optimized logical plans, scored against a single reference per task. The paper claims it is 'accurate' and 'strongly correlated with real-world debugging outcomes' without reporting a single comparison against actually executing repaired SQL on test data. That matters because Calcite's optimizer is a conservative approximation of SQL equivalence, and the benchmark is built on dialect constructs (LATERAL VIEW/explode, collect_set, INSERT OVERWRITE ... PARTITION) where canonicalization is least tested. A valid repair that changes plan shape can be scored 0, and a minimal edit that keeps plan shape but changes semantics can be scored 1. Appendix F concedes reliance on reference solutions, but the paper never quantifies the risk. This is addressable—validate on a sample against execution and report agreement—but until then the headline 36.46% and 'most models below 20%' are not fully supported.

There are also smaller inconsistencies that should have been caught: the abstract names the benchmark 'OurBench' while the body says 'Squirrel'; the introduction gives Claude 33.17% on Squirrel-Semantic while Table 2 and the abstract say 32.17%; and §5.1's claim that semantic performance is 'consistently lower' across all models is contradicted by Table 2 for Gemini-2.5-Pro (23.06 vs 21.54) and O3-mini (28.68 vs 19.83). The attack–defense filtering also removes easy instances, so reporting pre-filter numbers would make the difficulty claim more transparent.

The benchmark, code, and TQS are not public, so none of this can be independently checked today. That is a real limitation, not a fatal one.

Bottom line: the research direction is sound, the pipeline is buildable, and the paper is honest about its main risks. The metric validation is the gating issue. I'd send it to peer review and tell the authors to validate GM against execution and release the data before the numbers are treated as established.

Referee Report

4 major / 5 minor

Summary. The paper introduces Squirrel Benchmark, a set of 469 syntax-debugging and 516 semantic-debugging tasks built from synthetic Hive/Spark ETL scripts of 140+ lines, together with an execution-free evaluation framework based on Exact Match, Graph Match (graph isomorphism over Apache Calcite optimized plans), and Modify Better scores. The authors evaluate roughly 30 LLMs, report that Claude-4-Sonnet achieves only 36.46% GM on Squirrel-Syntax and 32.17% GM on Squirrel-Semantic, and present three SFT baselines plus an agentic baseline. The paper claims this is the first enterprise-level SQL debugging benchmark and that the benchmark correlates strongly with real-world debugging outcomes.

Significance. If the benchmark and metric are valid, this is a useful contribution to an underserved area: enterprise ETL SQL debugging is qualitatively different from short text-to-SQL generation, and a reproducible, complex, dialect-specific benchmark with fine-grained bug taxonomy would be valuable to the community. The automatic construction pipeline and the three-level taxonomy derived from 268 production-log errors are concrete assets, and the broad model comparison provides a starting point. However, the headline empirical claim — that even state-of-the-art models mostly fail — rests entirely on the Graph Match metric, which is not validated against execution, and on a construction loop in which the same model family generates, corrupts, and filters the tasks. Those two load-bearing supports must be strengthened before the benchmark can serve as a reliable reference.

major comments (4)

[Appendix D.1.2, Eq. (6); Appendix F] The central correctness metric GM is defined as exact graph isomorphism over Apache Calcite optimized logical plans, but no evidence is provided that this is a sound oracle for Hive/Spark ETL repairs. Calcite's rule-based optimizer is a conservative approximation of SQL equivalence, and it is not established that it canonicalizes the dialect constructs used here (LATERAL VIEW/explode, collect_set, INSERT OVERWRITE ... PARTITION, Hive functions). A valid repair that restructures a join or removes duplicates may produce a non-isomorphic plan and be scored 0, while a minimal edit that changes semantics but preserves plan shape may be scored 1. Appendix F concedes the framework 'is inherently limited by its reliance on reference solutions.' Because all headline numbers (36.46%, 32.17%, 'most models below 20%') are GM scores, the main empirical conclusion is unsupported until GM is calibrated
[§3.3, §3.4, §5.1] The benchmark construction loop is partly self-referential. Seed SQL, solution SQL, bug injection, and semantic task descriptions are all generated by Claude-4-Sonnet at temperature 0; the 'attack–defense' step in §3.4 then discards instances that a majority of evaluated models can already solve, retaining the hard tail. This does not measure a naturally occurring difficulty distribution; it measures performance on a set adversarially selected to separate models. It also creates a potential in-family advantage for Claude-4-Sonnet, since the reference answers and the task phrasing come from that model family. The paper should quantify how many instances were discarded at each stage, report results before and after filtering, and ideally validate the benchmark on an independently authored or execution-verified subset to show that the retained tasks are representative rather than synthetic
[§5.1, Table 2] The claim that 'performance on Squirrel-Semantic is consistently lower than on Squirrel-Syntax' is directly contradicted by Table 2. Gemini-2.5-Pro has GM 21.54 on Squirrel-Syntax and 23.06 on Squirrel-Semantic; O3-mini has GM 19.83 on Squirrel-Syntax and 28.68 on Squirrel-Semantic. Several other models (e.g., Qwen-2.5-Coder-32B) also have higher semantic GM than syntactic GM. The sentence should be revised to describe the observed trend rather than a universal ordering, or the discrepancy should be explained.
[§3.1, Eq. (4); Appendix D.1.3] The paper repeatedly describes the validation pipeline as 'execution-based,' but Appendix D.1.3 states explicitly that no SQL is run against a live engine and that TQS is a static-analysis tool. Equation (4) claims 'guaranteed execution correctness' from a check that only validates syntax and schema-level semantics. This overstates what the construction pipeline establishes: a query can pass TQS while returning incorrect results on data. The terminology should be corrected, and the guarantee in Eq. (4) softened to 'passes static validation,' otherwise readers will overestimate the reliability of the ground-truth references that the entire benchmark inherits.

minor comments (5)

[Abstract and §1] The abstract refers to the benchmark as 'OurBench' while the body uses 'Squirrel Benchmark'; also the abstract and §1 report Claude-4-Sonnet's Squirrel-Semantic score as both 32.17% (abstract, Table 2) and 33.17% (§1). Please reconcile.
[Table 1] Rows are mislabeled 'Squrriel-Syntax' and 'Squrriel-Semantic' (missing 'r').
[Eq. (2), §3.1] The complexity threshold τ and weights α, β are never specified. A sensitivity analysis or at least the chosen values should be reported, since the threshold controls which seed SQL enters the corpus.
[§3.4] 'Majority of models succeed' and 'only a few models succeed' are not quantified; the number of models used in the attack and the counts of discarded/retained instances should be given for reproducibility.
[Appendix D.1.3] The phrase 'strictly non-executive' is confusing; consider replacing with 'static analysis' or 'non-executing.'

Circularity Check

1 steps flagged · score 5.0 of 10

Central 'models fail to reach 20%' result is partly by construction: §3.4 discards instances where most evaluated LLMs succeed, so Table 2's low scores follow from the selection rule.

fitted input called prediction [§3.4 Validation and Analysis; §1 Introduction; Table 2]
"We first attack the benchmark by evaluating each generated instance with a diverse set of advanced LLMs (including Qwen3-Coder-32B, GPT-5, DeepSeek-V3.1, Claude-4-sonnet, and others). Instances fall into three categories: (i) If the majority of models succeed, the instance is deemed too easy and discarded; (ii) If only a few models succeed, the instance is considered an edge case and retained; ... This adversarial filtering ensures that the benchmark emphasizes cases where current models diverge, thereby sharpening its discriminatory power. ... Claude-4-Sonnet achieves only 36.46% success on S"

The headline that 'most models fail to reach 20% success' is produced by the same selection rule that keeps only instances where the evaluated model families (including Claude-4-Sonnet, GPT-5, and DeepSeek-V3.1) already fail or diverge. Since §3.4 discards instances where the majority of models succeed, the retained benchmark is, by construction, enriched for low model accuracy. Reporting those accuracy numbers as an independent empirical discovery treats the filter's output as if it were a free-standing measurement of LLM debugging ability. Table 2's low GM scores are therefore, to a first order, a consequence of the benchmark-selection criterion rather than an independent finding.

full rationale

The only genuine circular step I identified is the attack-defense filtering: the benchmark's difficulty is fitted to the same models that are later scored, so the central 'models struggle' conclusion is partially manufactured by design. The rest of the pipeline is self-contained as benchmark construction: synthesis, execution-style validation via TQS static checks, and the rule-based Graph Match metric (Eq. 6) are all well-defined. However, GM is not validated by executing repaired SQL on test data, and the paper's own Appendix F concedes the approach 'is inherently limited by its reliance on reference solutions'; the claimed 'strong correlation with real-world debugging outcomes' is asserted but not demonstrated. Those are validity limitations rather than circular reductions, so they do not add separate circular steps. The DM-SFT baseline cites the authors' own Duan et al. 2024, but that self-citation is not load-bearing for the benchmark's main claim. Overall, because the flagship empirical result—'most models below 20%'—is partly an artifact of the §3.4 filtering rule, I assign 5 rather than a lower score; the benchmark may still contain genuinely hard, human-verified tasks, but the headline discovery is not fully independent of how the test set was selected.

Assumptions & free parameters 3 free parameters · 6 assumptions · 2 invented entities

The central claims rest on three pillars unverifiable at submission time: (1) LLM-synthesized SQL and LLM-injected bugs faithfully represent real enterprise debugging (all tasks and reference answers are produced by Claude-4-Sonnet, the same model family that later tops the leaderboard); (2) the proprietary TQS/Calcite static-analysis tooling used for validation behaves correctly; and (3) the execution-free graph-match metric is a valid proxy for correctness. None has an external anchor in the paper; the practice-correlation assertion is the only place an anchor is promised but not delivered. Hand-chosen benchmark-design parameters (complexity threshold with weights α, β; top-k bug selection; attack-defense cutoffs) directly shape the reported difficulty.

free parameters (3)

Complexity threshold τ (and weights α, β in Eq. 2) = unspecified (reported corpus: ≥120 lines, AST depth >8, width >12)
Hand-chosen filtering criterion determining which seed scripts enter the corpus; it shapes the difficulty and therefore the headline results.
Top-k bug-type selection (Sec 3.3 Step 1) = k unspecified
LLM picks top-k candidate bug types per query; the value of k and the selection prompt influence which bugs get injected and thus the error distribution.
Attack-defense filtering thresholds (Sec 3.4) = unspecified ('majority', 'only a few', 'none')
Instances solved by most models are discarded; the cutoffs directly determine the benchmark's difficulty and the reported sub-20% performance band.

assumptions (6)

domain assumption Claude-4-Sonnet-synthesized SQL (domain-transferred from real seed scripts) preserves the complexity, realism, and correctness of genuine enterprise ETL code
Section 3.1 relies on LLM generation for scenario/schema/SQL synthesis with only static-style validation (Eq. 4); if synthesized scripts contain artifacts, the benchmark measures artifacts. Residual risk acknowledged in Appendix F.
domain assumption Graph isomorphism over Calcite-optimized logical plans is a valid proxy for semantic equivalence of Hive/Spark SQL (incl. explode, collect_set, LATERAL VIEW, INSERT OVERWRITE)
Eq. (6) and Appendix D.1.2 define GM via plan isomorphism; no ground-truth comparison against execution is provided, so metric validity is assumed.
domain assumption The 268 production-log error annotations produce a representative taxonomy, and minimal LLM-injected bugs drawn from it are realistic
Section 3.2; the taxonomy underpins both bug injection and the benchmark's realism claim.
domain assumption Each task has a unique repair (or all valid repairs are enumerated) matching the reference solution
EM/GM compare against the single reference; annotators only enumerate alternatives for instances flagged as unsolvable (§3.4, Limitations F).
ad hoc to paper Adversarial filtering keeps a representative sample of real-world debugging difficulty
Section 3.4 discards 'too easy' instances; this is a stated design choice that shapes reported accuracy and the 'most models below 20%' headline.
standard math AST edit distance and graph isomorphism are correctly implemented and standard operations
Used in Eqs. (6)-(7); uncontroversial in principle, but the implementation is not auditable without released code.

invented entities (2)

Squirrel Benchmark (Squirrel-Syntax 469 tasks, Squirrel-Semantic 516 tasks)
purpose: Standard testbed for enterprise ETL SQL debugging; model-selection reference for industry
New artifacts not yet released ('scheduled for public release upon acceptance'); the claimed correlation with real-world debugging outcomes is asserted without external evidence, so there is no falsifiable handle outside the paper yet.
Three-level hierarchical SQL bug taxonomy (macro/construct/atomic) from 268 production logs
purpose: Guides bug injection and enables fine-grained per-error-type evaluation
New categorization artifact embedded in the unreleased benchmark; the annotated logs are not public, so the taxonomy cannot be independently checked.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Beyond Text-to-SQL: Can LLMs Really Debug Enterprise ETL SQL?." pith.science (2026). https://pith.science/paper/4YWVNYVD

@misc{pith2026260118119,
  author       = {Pith},
  title        = {Pith review of: Beyond Text-to-SQL: Can LLMs Really Debug Enterprise ETL SQL?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4YWVNYVD}},
  note         = {Machine review of arXiv:2601.18119}
}

read the original abstract

SQL is central to enterprise data engineering, yet generating fully correct SQL code in a single attempt remains difficult, even for experienced developers and advanced text-to-SQL LLMs, often requiring multiple debugging iterations. We introduce OurBench, the first benchmark for enterprise-level SQL reasoning and debugging. Our benchmark is built on two key innovations: (1) an automated construction workflow that uses reverse engineering to systematically inject realistic bugs into large-scale SQL code, enabling scalable and diverse benchmark generation; and (2) an execution-free evaluation framework tailored to enterprise settings, providing fast, accurate, and resource-efficient assessment. OurBench comprises 469 OurBenchSyn queries featuring syntax errors with explicit error messages, and 516 OurBenchSem queries targeting semantic errors in which the code fails to meet user intent. The queries are highly complex, averaging over 140 lines and featuring deep and wide abstract syntax trees. Evaluation of nearly 30 LLMs reveals a substantial performance gap: the best-performing model, Claude-4-Sonnet, achieves only 36.46 percent accuracy on OurBenchSyn and 32.17 percent on OurBenchSem, while most models score below 20 percent. We further explore four solution strategies, identify key challenges, and outline promising directions for enterprise SQL debugging with LLMs.

Figures

Figures reproduced from arXiv: 2601.18119 by the authors.

**Figure 2.** Overview of the Squirrel Benchmark construction and evaluation pipeline. Benchmark construction [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Statistics of errors and domain distribution in Squirrel Benchmark. (a) Two-level error types in [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (10 more)

**Figure 4.** Figure 4: SFT baseline performance on Squirrel-Syntax. The horizontal axis represents the average inference speed, and the vertical axis shows the GM score. As detailed in Appendix D.3.1 and [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

**Figure 5.** Figure 5: Agent performance on SquirrelSyntax. ‘∗’ denotes agent-based methods, while others are single-model baselines. As detailed in Appendix D.3.2 and [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

**Figure 6.** Figure 6: Illustration of Graph Match Score. Although the left and right SQL snippets differ syntactically, [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]

**Figure 7.** Figure 7: Illustration of three distinct supervised fine-tuning (SFT) methods. [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]

**Figure 8.** Figure 8: Overview of the agentic method, which consists of a main agent, a code-generation sub-agent, and a [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]

**Figure 9.** Figure 9: Analysis of Qwen-2.5-Coder-7B Vanilla SFT on Squirrel Benchmark, showing corresponding training [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]

**Figure 10.** Figure 10: Model Hallucination: After modifying the code according to the error message, the model also inserted an extra “)” in similar fragments, which caused the fix to fail. Issue SQL Predict SQL Reference SQL Error Message: org.apache.calcite.runtime.CalciteContextException…

**Figure 11.** Figure 11: Long Context Reasoning Limitation: The error code uses a non-existent table b (which is usually an alias for a longer table name in SQL), but the model fail to detect this error during the repair process. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_11.png]

**Figure 12.** Figure 12: The example of Squirrel-Syntax, where an explicit error message exists. [PITH_FULL_IMAGE:figures/full_fig_p025_12.png]

**Figure 13.** Figure 13: The example of Squirrel-Semantic. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_13.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

I-Rex: An Interactive Debugger for SQL
cs.DB 2026-07 conditional novelty 6.0 of 10

I-Rex presents a scalable interactive SQL debugger with syntax-faithful canonical execution, GPL-style stepping/pinning, and query-rewrite-based teleporting; its user study claims faster bug finding but is undermined ...

Reference graph

Works this paper leans on

15 extracted references · cited by 1 Pith paper

[1]

- Treat this as a structural seed for generating an equivalent schema

Analyze Source DDL: - Examine the number of tables, fields, data types, relationships, and naming patterns. - Treat this as a structural seed for generating an equivalent schema
[2]

- Rules: - Use the databasefake_base_test

Generate Target Schema: - Create a logically equivalent schema under the target domain. - Rules: - Use the databasefake_base_test. - Format: CREATE TABLE IF NOT EXISTSfake_base_test.table_name ( ... ); - Avoid SQL reserved keywords as column names. - Reflect business meaning in the target domain. - Optionally add auxiliary fields to maintain equivalent co...
[3]

mock scenario

Validation: - Ensure DDL syntax is correct. - Ensure schema and scenario are fully adapted to the target domain, with no remnants from the source. ##Notes - Do not reuse proprietary identifiers or field names from the source domain. - Only use the user-provided target domain. - Preserve the structural pattern, complexity, and relationships of the source s...

2026
[4]

level1_error_type

Error Type Taxonomy:{SEMANTIC ERROR TYPES} ##Output Requirements: Your output must include: - The selected error type(s) at Level 1–3 granularity. ##Output Format: { 28 Under review as a conference paper at ICLR 2026 candidate_errors: { "level1_error_type": Level 1 error type, "level2_error_type": Level 2 error type, "level3_error_type": Level 3 error typ...

2026
[6]

- Preserve the overall structure, complexity, and transformation logic, but replace all table names, field names, and data types to match the target domain

Logical structure equivalence: - Analyze the ETL workflow, table relationships, and processing steps in the source SQL code. - Preserve the overall structure, complexity, and transformation logic, but replace all table names, field names, and data types to match the target domain
[7]

- Table names and field names must match the target DDL exactly

Strictly match the target DDL: - All SQL must be fully based on the provided target DDL. - Table names and field names must match the target DDL exactly. - Do not retain any original business terms, identifiers, or domain concepts from the source code
[8]

- Maintain a clear hierarchy and readability (include appropriate comments)

Output requirements: - The code must be executable, and SQL syntax must be correct. - Maintain a clear hierarchy and readability (include appropriate comments). - Naming should reflect the target business domain, ensuring a one-to-one correspondence between SQL and the target DDL. ##Input Data Source SQL:SQL Target Domain Scenario:SCENARIO Target DDL:DDL ...
[14]

level1_error_type

Error Type Taxonomy:{SEMANTIC ERROR TYPES} ##Output Requirements: Your output must include: - The selected error type(s) at Level 1–3 granularity. - The modified SQL query with the injected bug. ##Output Format: { "level1_error_type": Level 1 error type, "level2_error_type": Level 2 error type, "level3_error_type": Level 3 error type, "issue_sql": SQL que...

Show all 15 references

[15]

29 Under review as a conference paper at ICLR 2026

Introduce the error into the SQL query with the smallest possible change. 29 Under review as a conference paper at ICLR 2026

2026
[16]

##Key Guidelines: - Minimal Change: Only introduce the chosen bug

Write a realistic user-style issue report describing how the bug causes the query to behave incorrectly, and the user’s real intention. ##Key Guidelines: - Minimal Change: Only introduce the chosen bug. Do not alter the original query’s structure or intent more than necessary....
[17]

DDL (optional):{DDL}
[18]

Original Intent:{CODE INTENTION}
[19]

level1_error_type

Error Type Taxonomy:{SEMANTIC ERROR TYPES} ##Output Requirements: Your output must include: - The selected error type(s) at Level 1–3 granularity. - The modified SQL query with the injected bug. - A natural-language user bug report describing the mismatch between expected and ...

2026
[2024]

AS" at line 14, column 54. Was expecting one of:

2, and inference is performed withvLLM(Kwon et al., 2023). E ADDITIONALEXPERIMENTALRESULTS E.1 ADDITIONALANALYSIS OFSFT PERFORMANCE ONSQUIRRELBENCHMARK (a) Training loss curve. (b) Performance at different training steps. Figure 9: Analysis of Qwen-2.5-Coder-7B Vanilla SFT on ...

2023
[2025]

14 Under review as a conference paper at ICLR 2026 APPENDIX A Use of LLMs 16 B Background of ETL SQL debugging

URLhttps://openreview.net/forum?id=YrycTjllL0. 14 Under review as a conference paper at ICLR 2026 APPENDIX A Use of LLMs 16 B Background of ETL SQL debugging. 16 C Seed Data Curation 16 D Experimental Settings 17 D.1 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . ....

2026

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.

[1] [1]

- Treat this as a structural seed for generating an equivalent schema

Analyze Source DDL: - Examine the number of tables, fields, data types, relationships, and naming patterns. - Treat this as a structural seed for generating an equivalent schema

[2] [2]

- Rules: - Use the databasefake_base_test

Generate Target Schema: - Create a logically equivalent schema under the target domain. - Rules: - Use the databasefake_base_test. - Format: CREATE TABLE IF NOT EXISTSfake_base_test.table_name ( ... ); - Avoid SQL reserved keywords as column names. - Reflect business meaning in the target domain. - Optionally add auxiliary fields to maintain equivalent co...

[3] [3]

mock scenario

Validation: - Ensure DDL syntax is correct. - Ensure schema and scenario are fully adapted to the target domain, with no remnants from the source. ##Notes - Do not reuse proprietary identifiers or field names from the source domain. - Only use the user-provided target domain. - Preserve the structural pattern, complexity, and relationships of the source s...

2026

[4] [4]

level1_error_type

Error Type Taxonomy:{SEMANTIC ERROR TYPES} ##Output Requirements: Your output must include: - The selected error type(s) at Level 1–3 granularity. ##Output Format: { 28 Under review as a conference paper at ICLR 2026 candidate_errors: { "level1_error_type": Level 1 error type, "level2_error_type": Level 2 error type, "level3_error_type": Level 3 error typ...

2026

[5] [6]

- Preserve the overall structure, complexity, and transformation logic, but replace all table names, field names, and data types to match the target domain

Logical structure equivalence: - Analyze the ETL workflow, table relationships, and processing steps in the source SQL code. - Preserve the overall structure, complexity, and transformation logic, but replace all table names, field names, and data types to match the target domain

[6] [7]

- Table names and field names must match the target DDL exactly

Strictly match the target DDL: - All SQL must be fully based on the provided target DDL. - Table names and field names must match the target DDL exactly. - Do not retain any original business terms, identifiers, or domain concepts from the source code

[7] [8]

- Maintain a clear hierarchy and readability (include appropriate comments)

Output requirements: - The code must be executable, and SQL syntax must be correct. - Maintain a clear hierarchy and readability (include appropriate comments). - Naming should reflect the target business domain, ensuring a one-to-one correspondence between SQL and the target DDL. ##Input Data Source SQL:SQL Target Domain Scenario:SCENARIO Target DDL:DDL ...

[8] [14]

level1_error_type

Error Type Taxonomy:{SEMANTIC ERROR TYPES} ##Output Requirements: Your output must include: - The selected error type(s) at Level 1–3 granularity. - The modified SQL query with the injected bug. ##Output Format: { "level1_error_type": Level 1 error type, "level2_error_type": Level 2 error type, "level3_error_type": Level 3 error type, "issue_sql": SQL que...