{"id":"b670ee9c-76be-4d08-9065-d46c151d22c8","arxiv_id":"2505.13271","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"CSC-SQL adds a merge-revision stage and RL fine-tuning to self-consistency, improving execution accuracy on BIRD and Spider text-to-SQL benchmarks.","lead":"CSC-SQL is a text-to-SQL method that samples many candidate queries, takes the two most common answers, and has a second model correct or merge them before voting again. On the BIRD benchmark it reports 73.67% execution accuracy with a 32B model, a few points above plain self-consistency.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Main results appear to use a GRPO-trained revisor for non-GRPO rows, and SC baselines differ between Table 1 and Tables 2/5, confounding the claimed CSC gains.","rationale":"I read CSC-SQL as a system paper: the central claim is that the integrated CSC-SQL pipeline outperforms plain Self-Consistency on BIRD/Spider across many open LLMs. The most load-bearing place is not whether the top-2 groups sometimes contain a correct SQL (Figure 1 and Table 5 support that), but whether Table 1's comparison isolates the proposed corrective mechanism. The paper never states which revisor (base or GRPO-trained) is used for the non-GRPO generation models, and the ablation tables show the base revisor often fails to improve over SC. If the GRPO-trained revisor is used without explicit disclosure, the headline gain is largely an RL fine-tuning effect, not a validation of corrective self-consistency. Additionally, the SC baseline for the same configuration differs between Table 1 (62.18) and Tables 2/5 (61.86), which undermines the reliability of the exact reported improvements. A single reproduction run with the released code would resolve this. The reader's conditional verdict already flags reproducibility concerns, so I keep CONDITIONAL/UNCHANGED, but the specific check is more targeted than asking for generic error bars.","tokens_in":20673,"tokens_out":18570,"duration_ms":158018,"concrete_test":"Re-run the released code for Table 1's Qwen2.5-Coder-7B-Instruct, no-GRPO, n=8, m=8 setting twice: once with the base (non-GRPO) Qwen2.5-Coder-7B-Instruct as revisor and once with the GRPO-trained revisor. Check which reproduces the reported CSC EX of 64.28 and whether SC EX is 62.18 or 61.86. If the base-revisor variant is at or below the SC baseline, the method's gain is attributable to RL fine-tuning of the revisor rather than to the top-2 corrective selection, and the paper should downgrade its claim accordingly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that CSC reliably outperforms SC across 11 models on BIRD dev (Table 1, gains 0.72–5.54%). The load-bearing assumption is that the comparison isolates the proposed selection/revision mechanism. The paper never specifies whether 'CSC' rows with a non-GRPO generation model use the base Qwen2.5-Coder-Instruct revisor or the GRPO-trained revisor. Appendix B states only two revisor models were trained (3B/7B, GRPO), and Section 2 says both a base and a GRPO variant exist. Table 2 shows that with the base revisor at m=8, Qwen2.5-Coder-7B-Instruct (no GRPO) gives 61.73 EX, below its SC baseline 61.86; with the GRPO revisor, Table 1 reports 64.28 for the same generation model and n=8. Thus the headline no-GRPO CSC rows can only reach their reported values by using the RL-finetuned revisor and by deviating from the SC baselines in Tables 2/5 (n=8: 62.18 vs 61.86; n=16: 63.35 vs 63.65). These discrepancies mean the exact numbers supporting the headline are not reproducible from the ablation tables, and the 'consistent improvement' claim conflates the corrective-selection mechanism with an additional RL-trained corrector. If the base revisor yields no gain, the specific novelty of top-2 corrective merging contributes little.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CSC-SQL, a test-time scaling method for text-to-SQL generation. Given N candidate SQL queries sampled from a generation model, the method groups them by their execution results, selects the two largest groups, and feeds the representative queries together with the schema and execution outcomes into a merge-revision model. The revision model produces M new candidates, which are then aggregated by self-consistency voting. The authors also apply GRPO reinforcement learning to fine-tune both the SQL generation and revision models, using execution accuracy and a format reward. On the BIRD development set, the method is reported to outperform plain self-consistency by 0.72%-5.54% execution accuracy across 11 open LLMs, and on the BIRD private test set it achieves 71.72% (7B) and 73.67% (32B). The paper also reports cross-dataset generalization on Spider and includes ablations on revision sample count, top-k group size, temperature, and inference cost.","tokens_in":20974,"tokens_out":9430,"duration_ms":86259,"significance":"If the reported results hold, CSC-SQL is a practically useful and inexpensive enhancement to self-consistency for text-to-SQL: it is model-agnostic, built on open models, and the code is publicly released, which is a clear strength. The breadth of the evaluation (11 models, two benchmarks, multiple ablations) and the strong private-test numbers make the contribution potentially valuable to the community. However, the current reporting leaves the central comparison ambiguous: the paper does not specify which revision checkpoint is used in each row of the main table, and several headline numbers are inconsistent with the ablation tables. Until the comparison is made clean and reproducible, the significance of the claimed consistent improvement cannot be fully assessed.","major_comments":[{"comment":"The paper does not state whether the 'CSC' rows in Table 1 are produced with the base merge-revision model or the GRPO-trained revision model, and the reported numbers do not match either column in the ablation tables. For Qwen2.5-Coder-7B-Instruct (no GRPO), Table 1 reports SC=62.18 and CSC=64.28 at n=8, while Table 2 reports SC=61.86 and, at m=8, CSC=61.73 with the base 7B revisor and CSC=63.93 with the GRPO 7B revisor; Table 5 repeats 63.93 for the top-2 configuration. No m value in Table 2 reaches 64.28 (the closest is 64.04 at m=4 or m=16). At n=16, Table 1 reports SC=63.35 and CSC=66.41, while Table 2 and Table 5 report SC=63.65 and CSC=66.78. The SC baseline itself therefore differs between Table 1 and Tables 2/5, and the CSC values in Table 1 are inconsistent with the ablation results. This means the exact numbers supporting the headline claim cannot be reproduced from the ablation tables, and the comparison conflates the corrective-selection mechanism with the effect of the GRPO-trained revisor. Please specify the revision checkpoint used for every row of Table 1 and reconcile the baselines across all tables.","section":"§3.2, Table 1 vs. §3.4, Table 2 and Appendix D.2, Table 5"},{"comment":"The claim that 'the CSC method consistently outperforms the SC method' is too broad. On the Spider development set, Table 4 shows XiYanSQL-QwenCoder-3B-2502 under GRPO at n=64 with CSC=76.11 versus SC=76.21, i.e., a negative delta of -0.1. More importantly, Table 2 shows that the base (non-GRPO) revisor degrades performance relative to SC (61.73 versus 61.86 at n=8), so the gains in Table 1 for non-GRPO generation models cannot be attributed to the top-two corrective mechanism alone. The paper should either scope the consistency claim to the BIRD development set with the exact configuration used, or provide a controlled decomposition showing the contribution of the top-two selection/revision mechanism over a single-revision baseline using the same revisor.","section":"§3.2 and Table 4"},{"comment":"The paper reports results as 'the average performance over three runs' but does not report variance, error bars, or significance tests. The smallest claimed improvement on BIRD dev is 0.72% (Table 1, n=4 for Meta-Llama-3.1-70B), which corresponds to roughly 11 questions on the 1,534-question development set. With only three runs, a difference of this magnitude can easily fall within run-to-run noise, especially given the internal inconsistency of the baselines across tables. Please report per-cell standard deviations or paired significance tests for the main comparisons in Table 1.","section":"§3.1"}],"minor_comments":[{"comment":"The paper explicitly acknowledges that CSC relies on sufficient diversity among sampled results, but it does not quantify how often the top two groups both fail. Adding a metric such as 'top2_both_fail@k' alongside major_top2_pass@k would directly measure the premise of the method.","section":"§5 and Figure 1"},{"comment":"The four panels of Figure 3 use different y-axis ranges (e.g., 0.575-0.75 for temperature 0.2 versus 0.60-0.85 for temperature 0.8), which makes visual comparison of the temperature effect misleading. A common y-axis would be clearer.","section":"Figure 3"},{"comment":"The sentence 'the EX of Qwen2.5-Coder-7B-Instruct without GRPO training improves from 1.28% to 4.25%' is misleading; 1.28% and 4.25% are the gains at n=4 and n=64, not the starting and ending EX values. Please reword.","section":"§3.2"},{"comment":"The prompt templates and output examples contain a typo 'A VG' instead of 'AVG' (e.g., the <answer> example in Appendix F.1 and the failed case in Section E.2.1).","section":"Appendix F and Appendix E"},{"comment":"Several reference names contain spurious spaces, such as 'V olvovsky' and 'V ouitsis'; also 'chose' in Section 2 should be 'choose'.","section":"References"},{"comment":"The metric major_top2_pass@k is only defined informally in the Figure 1 caption. Please provide a formal definition in the main text or appendix, including whether it is computed over the union of the top two execution-result groups.","section":"Figure 1 caption / §2"}],"recommendation":"major_revision","confidential_remarks":"The open-source code release could straightforwardly resolve the checkpoint ambiguity; I recommend requesting the exact configuration used for each Table 1 row and a reconciliation of Table 1 with Tables 2 and 5. The 'consistent improvement' language in the abstract and Section 3.2 should be scoped precisely, and the negative Spider cell in Table 4 should be acknowledged."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a practical, incremental method with a broad evaluation, but the headline comparison is muddied by an unclear revisor setting and SC baselines that don't match between tables. I'd send it to a referee, but the authors need to fix the transparency and the numbers first.\n\nWhat's actually new: the top-2 execution-result grouping plus a merge-revision step and a second self-consistency vote is a concrete extension of the authors' BASE-SQL template. The major_top2_pass@k diagnostic is a nice way to see why SC misses correct candidates. The evaluation is genuinely broad — 11 open models on BIRD dev, plus Spider for the 3B — and Table 1 shows CSC above SC in every cell, with gains that grow with n. The code is also released. That's real evidence the pipeline can buy a few points of execution accuracy, and the cost analysis in Table 7 is a useful practical touch.\n\nNow the soft spots. The biggest one: the paper never states which revisor is used for the CSC rows in Table 1. Table 2 shows the base revisor underperforms SC at n=8 (61.73 vs 61.86), and the GRPO-trained revisor is needed to reach the reported 64.28. So the main result conflates corrective selection with an RL-trained corrector. That's not fatal — GRPO training is described as part of the method — but it should be explicit, and the ablation undermines the 'the selection mechanism works' narrative.\n\nMore worrying, the SC baselines themselves don't match between tables: for the 7B model, Table 1 reports SC=62.18 at n=8 and 63.35 at n=16, while Tables 2 and 5 give 61.86 and 63.65. The exact gains in Table 1 therefore aren't reproducible from the other tables. There are no error bars or significance tests for the three-run averages, and the 'consistently outperforms' claim is contradicted by one Spider dev cell (-0.1). The hyperparameters were also tuned on the same dev set used for reporting, which is a minor additional concern.\n\nWho this is for: people working on test-time scaling for text-to-SQL, especially open-source recipes. It is not a paradigm shift, but it is a plausible technique worth knowing.\n\nRecommendation: I would send it to a serious referee — the idea is concrete, the evaluation breadth is real, and the code is released. The referee should ask for a clear statement of which revisor is used in each row, consistent baselines, variance reporting, and a softened consistency claim. If those mismatches don't resolve, I wouldn't bet on the headline numbers.","headline":"Practical, incremental text-to-SQL recipe with broad evaluation, but the main comparison is muddied by an unclear revisor setting and mismatched baselines between tables.","tokens_in":21526,"tokens_out":5955,"would_cite":true,"duration_ms":53304,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A corrective second vote on the top two SQL candidates reliably beats plain self-consistency.","keywords":["text-to-SQL","self-consistency","self-correction","merge revision","GRPO","reinforcement learning","execution accuracy","test-time scaling"],"falsifier":"On the BIRD development set, instrument the pipeline so that for every question you record whether at least one of the top-two execution-result groups contains the gold execution result and whether the final CSC-SQL output is correct when neither does. If the number of questions recovered from two wrong groups is close to zero, or if the reported accuracy gain disappears when the analysis is restricted to questions where the top-two groups exclude the gold result, the correction stage is not the source of the improvement.","tokens_in":20453,"feed_emoji":"📊","tokens_out":7297,"duration_ms":69565,"temperature":0.7,"pith_summary":"The paper proposes a test-time scaling method for text-to-SQL: it says that the usual self-consistency vote—picking the most frequent SQL among parallel samples—often selects a result that is not the one that executes correctly. Instead, CSC-SQL groups the sampled SQL queries by what they return, takes the two largest groups, and asks a merge-revision model to produce a corrected SQL from those two candidates and their execution results; a second self-consistency vote then picks the final answer. The paper further post-trains both the SQL generator and the reviser with GRPO reinforcement learning, using execution accuracy and output format as rewards. Across 11 open models on the BIRD development set, the method reports execution-accuracy gains of 0.72–5.54 percentage points over plain self-consistency, and 71.72% (7B) and 73.67% (32B) on the BIRD private test set. If the paper is right, accuracy at a given model size can be bought with extra sampling plus a small revision step instead of a larger model.","feed_headline":"Top-two vote merging lifts Text-to-SQL accuracy up to 5.54%","feed_subtitle":"CSC-SQL beats plain self-consistency across 11 open models on BIRD and reaches 73.67% on its private test set.","key_machinery":"The load-bearing object is the top-two execution-result group, together with the merge-revision template that is built from those two groups. Concretely: the generator samples N candidate SQLs; each runs against the database; the resulting query outputs are grouped by equality of execution result; the two largest groups contribute one representative SQL each. If the two representatives are identical the system stops, otherwise the schema, question, evidence, both SQLs, and both execution results are placed in a revision prompt that asks the model to produce a corrected answer. A final self-consistency vote over M revised candidates selects the output. The second component is GRPO training, whose reward combines a binary execution-accuracy term (1 if the query matches the gold execution result) with a format term (0.1 for correct answer tags), which the paper says is what turns the revision model from a degrading editor into a genuine corrector.","core_discovery":"The central claim is that the failure mode of self-consistency in Text-to-SQL—the most-voted candidate is not always the executable answer—can be repaired by a correction loop that uses execution results as the voting signal. Sampling N SQL queries, executing all of them, grouping by identical execution result, and keeping the top two groups gives the reviser a genuinely alternative candidate, not just a paraphrase of the most-voted query. Feeding those two candidates, their execution outputs, and the schema to a merge-revision model produces M revised queries, and a second execution-based vote selects the final SQL. The paper claims this two-stage scheme outperforms plain self-consistency for every model and sample-size combination tested, that GRPO post-training strengthens both halves of the pipeline, and that the resulting models match or beat larger open models while transferring from BIRD to Spider without retraining.","pith_inferences":["Editorial inference: the major_top2_pass@k diagnostic introduced for BIRD could be used at inference time as a stopping rule—keep sampling only until the top-two execution groups stop changing, saving compute without giving up the correction step.","Editorial inference: the same two-stage revision vote should transfer to other code-generation tasks with executable oracles, such as unit tests or compilers, where grouping by test outcome would supply the same corrective signal; the paper only demonstrates SQL.","Editorial inference: a direct test of the GRPO contribution would be to ablate the revision model's reward by using execution accuracy alone, isolating how much of the reported gain comes from the format component.","Editorial inference: reported gains are averaged over three runs; a run-by-run variance analysis on the BIRD development set would tell whether the 0.72% lower-end improvements are stable across models."],"forward_implications":["Replacing the plain most-frequent vote with top-two grouping plus merge revision raises execution accuracy on BIRD development by 0.72–5.54 percentage points across 11 open models and all sample sizes tested.","Because the gains grow with the number of sampled SQL candidates, users can trade more sampling for more accuracy, and the extra revision cost becomes a smaller share of total inference time.","GRPO-trained small models using CSC can beat larger same-family models using plain self-consistency, so the method offers a compute-shifted path to accuracy.","The merge-revision model trained only on BIRD transfers to Spider without retraining, showing that the correction behavior generalizes beyond the training benchmark.","The 32B configuration reaches 73.67% execution accuracy on the BIRD private test set, which the paper reports as surpassing the base model by 4.64 points.","If valid, the scheme can be dropped into other pipelines as a replacement for the self-consistency component."],"supporting_citations":[{"why":"Defines Self-Consistency, the baseline method whose majority-vote failure motivates CSC-SQL and against which all gains are measured.","marker":"Wang et al., 2023"},{"why":"Introduces the GRPO algorithm used to post-train both the SQL generation and merge-revision models.","marker":"Shao et al., 2024"},{"why":"Provides the BIRD benchmark, the main development and private test evaluation used for the headline accuracy numbers.","marker":"Li et al., 2024c"},{"why":"Supplies the merge-revision template and prompting pattern that CSC-SQL adapts for its corrective stage.","marker":"Sheng et al., 2025"},{"why":"Prior work pairing GRPO with SQL-tailored rewards, which the paper builds on and compares against as a strong open-model baseline.","marker":"Pourreza et al., 2025"},{"why":"Shows GRPO post-training on SQL with execution-based rewards, supporting the paper's choice of reinforcement training for SQL generation.","marker":"Ma et al., 2025"},{"why":"Provides the Spider dataset used to test zero-shot generalization of the BIRD-trained models.","marker":"Yu et al., 2019"},{"why":"Provides the XiYanSQL base model that yields the 73.67% private-test result and the comparison point the 32B model improves upon.","marker":"Gao et al., 2024"}],"fun_headline_variants":["Execution-based voting sharpens Text-to-SQL self-consistency","Top-two execution votes correct SQL generation by up to 5.54%","GRPO-tuned revision fixes self-consistency blind spots in SQL","CSC-SQL merges top two SQL candidates, then votes on execution","Reinforcement learning pairs self-consistency with correction for SQL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The premise the framework leans on is that the top two execution-result groups will contain at least one SQL that can answer the question often enough for the reviser to improve on the majority vote; if those groups are both wrong, or nearly identical, the correction step has nothing to recover.","fun_headline_variants_meta":{"raw":{"variants":["Execution-based voting sharpens Text-to-SQL self-consistency","Top-two execution votes correct SQL generation by up to 5.54%","GRPO-tuned revision fixes self-consistency blind spots in SQL","CSC-SQL merges top two SQL candidates, then votes on execution","Reinforcement learning pairs self-consistency with correction for SQL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000672,"raw_usage":{"total_tokens":3063,"prompt_tokens":950,"completion_tokens":2113,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":566,"completion_tokens_details":{"reasoning_tokens":2017}},"tokens_in":566,"tokens_out":2113,"duration_ms":15711,"temperature":1.0,"reasoning_tokens":2017,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:16:29.455893+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On the BIRD development set, instrument the pipeline so that for every question you record whether at least one of the top-two execution-result groups contains the gold execution result and whether the final CSC-SQL output is correct when neither does. If the number of questions recovered from two wrong groups is close to zero, or if the reported accuracy gain disappears when the analysis is restricted to questions where the top-two groups exclude the gold result, the correction stage is not the source of the improvement.","supporting_citations":[],"review_version":1}