{"id":"b60a0870-0145-4d09-8888-399632c9884b","arxiv_id":"2412.10138","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Multitask supervised fine-tuning plus collaborative prompting on schema linking, noise correction, and continuation writing lifts open-source LLM Text2SQL accuracy on SPIDER and BIRD.","lead":"This paper presents a training and prompting method, ROUTE, that makes open-source language models better at translating natural-language questions into SQL database queries. It combines four related tasks during fine-tuning and uses them together at query time, beating prior open-source methods on the standard SPIDER and BIRD benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported superiority margins over CODES/SENSE are small (1.9-2.4 EX) and lack error bars or multiple seeds; without significance testing the central claim of outperforming latest open-source methods is not established.","rationale":"I read the paper in good faith and accept that the method is coherent, thoroughly ablated, and likely yields real improvements from multitask SFT and MCP. The strongest evidence is the internal ablations: ROUTE beats plain SFT by 2-3 points on SPIDER and BIRD, the fusion schema linking in Eq. (2) outperforms either component alone, and the transferability study across 8 LLMs plus Dr.Spider results support robustness. Those are genuine strengths. However, the headline claim is not just that ROUTE improves its own ablations; it is that ROUTE 'outperforms the latest Text2SQL methods' and gives 'leading performance.' That comparative claim rests on small margins over the strongest fine-tuning baselines, and the paper provides no statistical evidence that those margins are reproducible. This is the least secure link in the central argument because a 2-point EX difference on BIRD dev could easily flip with a different seed or a stricter evaluation harness. The reader's chosen weakest assumption, the pseudo-SQL accuracy in Eq. (2), is a real robustness concern, but it is partly mitigated by the fusion with SL_sigma_s and by the paper's robustness evaluations; it affects generalization more than the reported benchmark numbers. I therefore partially agree with the reader: the no-error-bars issue is listed in the rationale, but it is not the weakest assumption, whereas I see it as the most load-bearing for the central claim. My recommendation is to keep the verdict CONDITIONAL until multiple-seed significance testing and a controlled comparison isolating inference-time feedback are provided; these are standard, concrete requirements rather than reasons to reject the method outright. I also note the abstract's 'leading' overstatement relative to Appendix A.3, but that is a wording issue that does not change the underlying verdict.","tokens_in":1042,"tokens_out":979,"duration_ms":140426,"concrete_test":"Run ROUTE (Qwen2.5-14B) and CODES-15B under an identical evaluation harness with 5 independent SFT seeds (varied data order/seed), reporting mean+/-std EX on SPIDER dev and BIRD dev. Additionally, ablate MCP down to schema-linking-plus-generation (disable NC and CW) and compare against CODES/SENSE to separate training gains from inference-time execution feedback. If the ROUTE minus baseline margin is within one standard deviation, or if the margin vanishes when NC/CW are removed, the central superiority claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claim is that ROUTE outperforms leading open-source fine-tuning methods. The margins supporting that claim are thin: on SPIDER dev, ROUTE+Qwen2.5-14B reports 87.3 EX vs 85.4 for CODES-7B (+1.9); on BIRD dev, 60.9 vs 58.5 for CODES-15B (+2.4). No standard deviations, confidence intervals, or multiple-seed runs are provided anywhere in the paper. A decoding temperature of 0.01 only reduces sampling noise; it does not control for SFT seed sensitivity, data ordering, or evaluation harness differences. If the true margin is within run-to-run noise, the 'outperforms' claim is unsupported. The comparison is also not fully controlled: ROUTE's inference pipeline includes SQL-executor feedback (NC) and continuation rewriting, while the fine-tuning baselines (CODES, SENSE) are evaluated as single-pass SQL generators. Part of the observed improvement may therefore come from extra inference-time computation rather than from the proposed multitask training. This is the most direct threat to the headline result. The pseudo-SQL fragility in Eq. (2) is a related but secondary generalization risk: Table 5 shows the fusion of SL_sigma_s and SL_sigma_t is more robust than either alone, and the paper's Dr.Spider and SPIDER-variant results provide some evidence of robustness, so it is less immediately decisive for the benchmark claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ROUTE, a two-stage method for improving Text2SQL performance of open-source LLMs. In the first stage, Multitask Supervised Fine-Tuning (MSFT) trains a single model on four SQL-related tasks: Text2SQL, schema linking, noise correction, and continuation writing, with synthetic data derived from the SPIDER and BIRD training folds. In the second stage, Multitask Collaboration Prompting (MCP) combines these capabilities at inference: schema linking plus pseudo-SQL-based database simplification, SQL-executor feedback with noise correction, and continuation writing for hard queries. The paper reports execution accuracy and test-suite accuracy on five benchmarks, with the strongest result being ROUTE with Qwen2.5-14B reaching 87.3 EX on SPIDER dev and 60.9 EX on BIRD dev, exceeding several open-source fine-tuning baselines and closing part of the gap with GPT-4-based prompting methods. The central claim is that multitask tuning plus collaborative prompting makes open-source models practical for Text2SQL without relying on closed-source APIs.","tokens_in":25145,"tokens_out":4576,"duration_ms":49335,"significance":"If the reported results hold, ROUTE is a practically valuable recipe for on-premises Text2SQL: it uses only open-source models, derives training data programmatically from public training sets, and shows strong transfer across multiple base LLMs and robustness benchmarks. The paper has several genuine strengths: evaluation on five benchmarks and eight open-source LLMs, including Dr.Spider variants; a detailed ablation study isolating the contributions of MSFT, MCP, and each prompting component; explicit prompt templates and training settings in the appendix; and released code and synthetic data. There is no circularity in the evaluation setup: training uses the public training folds and held-out dev sets are used for testing. However, the head-to-head claim of outperforming prior open-source fine-tuning methods is not yet conclusive, because the reported margins are small, no uncertainty estimates are given, and the comparison mixes different inference budgets and potentially different training settings.","major_comments":[{"comment":"The central empirical claim that ROUTE outperforms existing open-source fine-tuning methods is not supported with uncertainty quantification. The reported margins over CODES are small (1.9 EX on SPIDER dev and 2.4 EX on BIRD dev for Qwen2.5-14B), and the paper reports a single run per configuration at temperature 0.01. Fine-tuning is sensitive to seed, data ordering, and evaluation-harness differences, and temperature 0.01 does not control those sources of variance. I ask for at least three independent fine-tuning runs per main configuration, or a justified alternative such as confidence intervals or bootstrap significance testing, so that the reader can assess whether the reported margins are meaningful.","section":"Section 4.1, Table 1"},{"comment":"The comparison against CODES and SENSE is not fully controlled: ROUTE's inference pipeline includes SQL-executor exception feedback in the noise-correction step and continuation writing, whereas the fine-tuning baselines are reported as single-pass SQL generators. Part of the observed gain may therefore come from additional inference-time computation rather than from the proposed multitask training. Please provide a controlled experiment in which the same multi-step inference procedure, including executor feedback and continuation writing, is applied to the fine-tuned baselines, or report ROUTE without NC and CW to match the baseline inference budget. This attribution is load-bearing for the claim that multitask training, rather than extra compute, drives the improvement.","section":"Section 3.3 and Table 1"},{"comment":"The enhanced schema-linking strategy depends on the accuracy of the pseudo-SQL generated by the model on the full schema. Table 5 shows that relying on SL_sigma_t alone can degrade SPIDER EX from 69.3 to 64.5 on base Llama3-8B, and the paper's fusion mitigates this on the tested benchmarks. The generalization risk remains for new domains where the pseudo-SQL is inaccurate and the simplified database may omit relevant entities. I request a direct analysis of pseudo-SQL quality, for example the execution-match rate against ground-truth SQL or a qualitative study of cases where the fusion drops a needed table or column, together with a statement of the conditions under which the simplification is safe. This is a correctness-risk concern rather than a claim that the method fails on the evaluated benchmarks.","section":"Section 3.3, Eq. (2), Table 5"},{"comment":"The baseline comparison is not fully controlled in terms of training data and evaluation protocol. DTS-SQL is marked as re-evaluated with an open-source repository, but CODES and SENSE results appear to be taken from their original papers, which may use different training sets, different base-model checkpoints, and different evaluation settings. Please state explicitly for every fine-tuning baseline which training folds were used, which checkpoints were used, and whether the numbers are re-evaluated or quoted; where feasible, retrain and evaluate the baselines under identical conditions so that the margins in Table 1 are directly comparable.","section":"Section 4.1 and Table 1"}],"minor_comments":[{"comment":"The text says \"Table 5 shows the performance on the benchmarks derived from SPIDER,\" but the correct reference is Table 2.","section":"Section 4.3"},{"comment":"The BIRD dev EX score for ROUTE with Qwen2.5-14B is reported as 60.8 in Section 4.2 and in Table 11, but Table 1 reports 60.9; please make these consistent.","section":"Section 4.2 and Appendix A.3"},{"comment":"The sentence containing \"widely exist in various field ( ???? )\" has a malformed citation placeholder and should be corrected or removed.","section":"Section 3.2"},{"comment":"There are typos such as \"demonstartes\" and \"Precession\" for \"Precision.\" Also, Appendix A.3 misspells \"Gemini\" as \"Gimini\" and \"CHASE-SQL\" as \"CHASS-SQL.\"","section":"Appendix A.4 and A.6"},{"comment":"The abstract claims \"leading performance,\" but Appendix A.3 shows CHASE-SQL + Gemini 1.5 outperforming ROUTE on both SPIDER and BIRD; please soften the claim to \"competitive with closed-source prompting methods\" or clearly contextualize the comparison.","section":"Abstract and Appendix A.3"},{"comment":"The header row for Table 2 is difficult to parse because the column labels for SYN, Realistic, and DK are interleaved with the metric names; consider restructuring the table so each metric is clearly aligned with its column.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"I recommend major revision rather than rejection. The method is well specified, the training data construction is transparent, and the central mechanism is plausible, but the headline superiority claim rests on small margins without variance estimates and on a comparison that mixes inference budgets. The requested experiments are feasible within the manuscript's scope: multiple seeds, a controlled baseline with the same multi-step inference pipeline, and a pseudo-SQL quality analysis would substantially strengthen the paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: ROUTE is a solid, well-ablated open-source Text2SQL recipe, and the central claim — multitask SFT plus collaborative prompting helps 7B-14B models on SPIDER and BIRD — largely holds. What is actually new is the specific package: one model trained jointly on text-to-SQL, schema linking, noise correction, and a newly defined continuation-writing task, then prompted in a three-step pipeline. The paper earns its keep with unusually thorough ablation work: component ablations, per-hardness tables, transferability across eight LLMs, upper-bound analysis, and Dr.Spider robustness results. That is real evidence, not just a leaderboard screenshot.\n\nThe biggest soft spot is the headline comparison. The margins over CODES and SENSE are 1.9-2.4 EX points, and there are no error bars, multiple seeds, or significance tests anywhere. On top of that, the comparison is not apples-to-apples: ROUTE's inference includes schema-linked simplified databases, executor feedback, and continuation rewriting, while the fine-tuning baselines are evaluated as single-pass generators. Some of the gain may come from extra inference-time computation rather than from multitask training. I don't think that makes the paper wrong, but it does mean 'outperforms the latest methods' is not established by the reported numbers.\n\nSecond, the abstract says 'leading performance,' but Appendix A.3 shows ROUTE trailing CHASE-SQL and CHESS on BIRD by 5-12 points. That overclaim should be fixed in revision. Third, the pseudo-SQL schema linking in Eq. (2) is load-bearing and fragile in principle; Table 5 shows the fused linking is much more robust than pseudo-SQL alone, so I would call this a generalization risk rather than a fatal flaw. Minor issues: NC and CW contribute little in the ablations, yet the title features them; there is a leftover placeholder '????' in Section 3.2 and a 60.8 vs 60.9 inconsistency in the BIRD number; the code link says 'here' without verifiable release details. The citation pattern looks fair, and the self-citations to Dr.Spider and prior schema-linking work are appropriate.\n\nWho this is for: anyone building open-source Text2SQL systems who needs a practical, privacy-friendly alternative to GPT-4 prompting. It deserves a serious referee; the right outcome is conditional acceptance with requests for variance estimates, a controlled inference-cost comparison, and toned-down claims. I would bring it to reading group as a good example of how to ablate a multi-task pipeline.","headline":"A well-ablated multitask SFT + collaborative prompting recipe for open-source Text2SQL; the core result holds up, but the 'leading performance' claim outruns the significance testing and the baseline comparison is not fully controlled.","tokens_in":25756,"tokens_out":1893,"would_cite":true,"duration_ms":19606,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Multitask tuning and prompting push open LLMs to lead Text2SQL","keywords":["multitask supervised fine-tuning","schema linking","text-to-SQL","LLM prompting","noise correction","continuation writing","open-source LLMs","SPIDER and BIRD benchmarks"],"falsifier":"Take a held-out set of questions in a new domain, run ROUTE, and compare the tables and columns in the fused simplified database $\\tilde{d}_i$ with the tables and columns appearing in the ground-truth SQL. If a substantial fraction of wrong answers have ground-truth columns missing from $\\tilde{d}_i$, the load-bearing schema-linking assumption fails; if the fused schema has full recall and EX is still low, the bottleneck lies elsewhere.","tokens_in":24627,"feed_emoji":"🗄️","tokens_out":7051,"duration_ms":64360,"temperature":0.7,"pith_summary":"The paper claims that open-source LLMs can reach leading Text2SQL performance if they are fine-tuned on several SQL-related tasks at once and then asked to perform those tasks in sequence at inference. The proposed ROUTE method combines multitask supervised fine-tuning (MSFT) over four tasks — Text2SQL, schema linking, noise correction, and continuation writing — with a multitask collaboration prompting (MCP) pipeline. On SPIDER and BIRD, ROUTE with Qwen2.5-14B reports 87.3 and 60.9 execution accuracy, the best among the open-source fine-tuning baselines compared and close to prompting pipelines built on GPT-4. If the claim holds, organizations can get near-proprietary-model SQL generation while keeping data on their own hardware, because the whole pipeline runs on a single open model.","feed_headline":"Multitask tuning and prompting push open LLMs to lead Text2SQL","feed_subtitle":"Training one open model on four SQL tasks, then chaining them at inference, reaches 87.3% on SPIDER and 60.9% on BIRD.","key_machinery":"The load-bearing mechanism is the four-task MSFT objective followed by the three-step MCP loop. The central object is a fused simplified database $\\tilde{d}_i$ from Equation (2): the model's schema linking output is merged with tables and columns fuzzy-matched from a pseudo-SQL query generated on the full schema. That merged schema is fed to a Text2SQL step, whose output goes through noise correction that consults execution exceptions and, for hard or failing queries, continuation writing from a truncated prefix. The merge operation $\\uplus$ is what prevents the pseudo-SQL from silently dropping relevant entities; the paper's ablations show this schema-linking step contributes the largest share of the MCP improvement, with noise correction and continuation writing adding the rest.","core_discovery":"ROUTE's central claim is that the reason small open-source models lag behind closed ones on Text2SQL is not lack of SQL knowledge alone, but a mismatch between the single task they are trained for and the several subtasks that accurate SQL generation requires. By fine-tuning one model on Text2SQL, schema linking, noise correction, and continuation writing, and then at inference linking the schema, generating SQL, checking the generated query against execution errors, and rewriting difficult queries from a prefix, the same model can outperform single-task SFT and narrow the gap with GPT-4-based prompting. The strongest reported evidence is Table 1: ROUTE with Qwen2.5-14B scores 87.3 EX on SPIDER dev and 60.9 EX on BIRD dev, versus 84.9 and 58.5 for CODES-15B and 83.2 and 51.8 for SENSE-7B in the same table.","pith_inferences":["Editorial extension: the four-task recipe is not SQL-specific in its mechanism, so the same MSFT-plus-collaboration design is a plausible template for other schema-grounded generation tasks, such as text-to-Pandas or text-to-GraphQL, where hallucination follows from schema misalignment.","Editorial extension: the paper's Table 5 implies a cheap diagnostic: measure recall of the fused simplified database's columns against the ground-truth SQL's columns; failures that coincide with omitted columns would identify exactly which questions are at risk before execution.","Editorial extension: because the ablations give each MCP component's marginal contribution, one could build a budget-aware variant that runs schema linking and Text2SQL always, noise correction only on execution failure, and continuation writing only on hard queries, sacrificing a little accuracy for fewer LLM calls.","Editorial extension: noisy correspondence filtering helps SPIDER but slightly hurts BIRD in the ablations, suggesting the filtering threshold could be tuned per dataset or replaced by a curriculum that keeps hard examples late in training."],"forward_implications":["A single open model can carry the whole Text2SQL pipeline, so deployment does not need a separate schema-linking model the way DTS-SQL does.","Training on one task alone erodes the other SQL-related skills; the paper's MSFT is what keeps schema linking, noise correction, and continuation writing usable, and without them MCP gains shrink.","The MCP prompting stage transfers across models and sizes, improving 7B code models such as CodeLlama and Deepseek-Coder and also 70B-class Llama3 and Qwen2.5, with the largest average gains on BIRD.","Schema linking is the dominant contributor to MCP's gains, and an ideal schema-linking oracle would lift ROUTE with Llama3-8B to 87.4 EX on SPIDER and 69.6 EX on BIRD, indicating where future work should focus."],"supporting_citations":[{"why":"Provides the SPIDER benchmark, whose training and development sets are used for MSFT data construction and main evaluation.","marker":"Yu et al., 2018"},{"why":"Provides the BIRD benchmark, the harder cross-domain dataset used for training and evaluation.","marker":"Li et al., 2024c"},{"why":"DIN-SQL contributes schema linking and decomposition prompting ideas and serves as a GPT-4 prompting baseline.","marker":"Pourreza & Rafiei, 2024a"},{"why":"DTS-SQL is the single-task schema-linking SFT baseline that ROUTE improves on with one model instead of two.","marker":"Pourreza & Rafiei, 2024b"},{"why":"CODES is the open-source SFT baseline with incremental pretraining that ROUTE is compared against and outperforms on BIRD.","marker":"Li et al., 2024b"},{"why":"SENSE is the DPO-based Text2SQL baseline compared in Table 1.","marker":"Yang et al., 2024b"},{"why":"MAC-SQL supplies the multi-agent prompting baseline and comparison for open-source LLMs.","marker":"Wang et al., 2023"},{"why":"PET-SQL is the source of the pseudo-SQL based schema linking idea used in Equation (2).","marker":"Li et al., 2024d"},{"why":"Documents noisy correspondences in Text2SQL training sets, motivating the noisy correspondence filtering step.","marker":"Wretblad et al., 2024"}],"fun_headline_variants":["Multitask tuning helps open LLMs rival GPT-4 in SQL generation","Open LLMs beat closed ones with multitask SQL training and prompting","Training on four SQL tasks lifts open models to top Text2SQL scores","Multitask collaboration makes open LLMs competitive for Text2SQL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The approach relies on the first rough SQL guess being good enough that the simplified database built from it does not drop a table or column the true answer needs.","fun_headline_variants_meta":{"raw":{"variants":["Multitask tuning helps open LLMs rival GPT-4 in SQL generation","Open LLMs beat closed ones with multitask SQL training and prompting","Training on four SQL tasks lifts open models to top Text2SQL scores","Multitask collaboration makes open LLMs competitive for Text2SQL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000228,"raw_usage":{"total_tokens":1508,"prompt_tokens":1014,"completion_tokens":494,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":630,"completion_tokens_details":{"reasoning_tokens":416}},"tokens_in":630,"tokens_out":494,"duration_ms":5053,"temperature":1.0,"reasoning_tokens":416,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T16:19:27.325151+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out set of questions in a new domain, run ROUTE, and compare the tables and columns in the fused simplified database $\\tilde{d}_i$ with the tables and columns appearing in the ground-truth SQL. If a substantial fraction of wrong answers have ground-truth columns missing from $\\tilde{d}_i$, the load-bearing schema-linking assumption fails; if the fused schema has full recall and EX is still low, the bottleneck lies elsewhere.","supporting_citations":[],"review_version":1}