{"id":"c90b1804-3ecf-4ee6-bc99-9f58a1f3fecc","arxiv_id":"2412.07246","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"LECSP uses LLM-generated pseudo-samples guided by SQL syntax differences between tasks, plus dual-teacher distillation, to train a continual semantic parser without replaying real historical data.","lead":"This paper presents LECSP, a method for continual semantic parsing that uses a large language model to generate synthetic SQL training data, eliminating the need to store or replay old real data. It reports large accuracy gains over prior continual learning methods on two text-to-SQL benchmarks, including a cold-start setting.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'beyond upper bound' FWT claim is an artifact of comparing across unequal resource regimes: ORACLE is not an upper bound for methods with an external LLM teacher.","rationale":"The reader's conditional verdict already flags the 'beyond upper bound' claim as an overstatement, so my concern is within scope. However, I elevate it from a phrasing issue to a load-bearing concern because the abstract and conclusion rest part of the contribution on exceeding an upper bound that is not a bound for the method's actual resource regime. The comparison with replay/ideal baselines and the Delta A mechanism are not the weakest link; the FWT upper-bound claim is. The proposed test (ORACLE + LLM) would settle whether the claim is an artifact of unequal resources. The reader's weakest_assumption focused on Delta A and calibration, not on the FWT upper-bound reasoning, hence partial agreement. Since the reader's conditional verdict already encompasses this issue and the core comparison with replay baselines is not invalidated, I recommend keeping the verdict unchanged rather than moving to accept or reject.","tokens_in":23945,"tokens_out":9833,"duration_ms":105460,"concrete_test":"Compute per-task FWT components for LECSP and ORACLE, including the random-init reference accuracies \\hat{a}_i. Then train an ORACLE variant augmented with the same LLM-generated pseudo-samples (ORACLE + LECSP memory) and recompute FWT. If this augmented ORACLE also exceeds plain ORACLE, the 'upper bound' is not an upper bound for methods with LLM access; if it exceeds LECSP, the beyond-upper-bound claim fails outright. This isolates whether the effect is due to the LLM resource rather than a true bound violation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim includes 'generalization performance beyond the upper limits' (Abstract and Conclusion). The ORACLE baseline is defined as 'the performance upper bound for CSP' (Section: Baselines), trained only on the task stream. FWT is defined as (1/(M-1)) * sum_{i=2}^M (a_{i-1,i} - \\hat{a}_i), measuring transfer to an unseen task. For ORACLE, a_{i-1,i} after training on tasks 1..i-1 should be high, but ORACLE is not a formal upper bound on FWT: (i) training on more previous-task data can induce negative transfer to a new schema, so ORACLE's FWT is not necessarily the maximum achievable; (ii) LECSP uses Mixtral-8x7B as Teacher 1, injecting knowledge from a much larger pretraining corpus, a resource not available to ORACLE. An upper bound in one resource regime does not constrain a method with additional external knowledge. Therefore Table 1's FWT values (LECSP 41.4/44.3 vs ORACLE 34.3/36.0 on Spider-stream-semi T5-large) do not demonstrate 'beyond upper limits'; they only demonstrate that the LLM provides transferable knowledge. The comparison with replay baselines is unaffected, but the headline claim needs rephrasing or a redefined upper bound that includes the LLM resource.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LECSP, a continual semantic parsing method that avoids replaying real historical data and does not rely on the ideal task-identity setting used by prompt-based continual learners. The method reconstructs pseudo-memories by (i) removing domain-specific information, (ii) clustering SQL skeletons per task via CodeT5, (iii) computing a syntax-level component bias between the current task and previous tasks, and (iv) prompting an open-source LLM (Mixtral-8x7B) to generate pseudo question-SQL pairs, which are then filtered by execution checks, LLM self-correction, and skeleton-based edit-distance sampling. A dual-teacher distillation framework trains a T5 student using the reconstructed memories, the previous student as a second teacher, and the current task's annotated data. Experiments on Spider-stream-semi and Combined-stream report that LECSP outperforms rehearsal and PET baselines in both warm-start and cold-start protocols, and the paper claims FWT performance beyond the ORACLE upper bound.","tokens_in":24238,"tokens_out":3015,"duration_ms":33757,"significance":"If the empirical results are robust, the paper makes a useful contribution: it demonstrates an effective no-replay, no-ideal-setting recipe for continual text-to-SQL by shifting the burden of memory reconstruction to an offline open-source LLM, and it provides a decomposition (inter-task skeleton completion, intra-task CFG augmentation, calibration, and distillation) with ablations showing that each component matters. The cold-start experiments and the memory-size comparison are valuable additions to the CSP evaluation landscape. However, the headline claim of surpassing an upper bound is overstated, and the absence of variance reporting makes the word 'significantly' unsupported as stated. The core comparison against replay-based baselines remains credible and is the main strength of the paper.","major_comments":[{"comment":"The claim that ORACLE represents 'the performance upper bound for CSP' is used in Section 2 and in the discussion. As noted above, this is not a theorem; it is an empirical baseline. This should be stated and the 'beyond upper bound' language removed or carefully qualified.","section":"Section 2"},{"comment":"The quality comparison in Figure 5(a) appears to compare LECSP pseudo-samples (evaluated by manual inspection) with SFNET and Mixtral samples (evaluated using their original labels). This is an asymmetric evaluation: manual judgment may use different criteria than the automatic SQL execution used for the baselines. Please state clearly whether the same execution-based metric was applied to all three methods, or provide a human-evaluation protocol applied identically to all methods.","section":"Experiments, 'Quality and Quantity of Pseudo Samples'; Figure 5(a)"}],"minor_comments":[{"comment":"The name is introduced as 'Continual Semantic Parsing' but the abstract first uses 'Continuous Semantic Parsing'; please ensure consistent terminology.","section":"Abstract and throughout"},{"comment":"The heading has inconsistent capitalization ('learning' is lowercase); please polish the formatting.","section":"Methodology, 'Task-Aware Dual-Teacher Distillation learning Framework'"},{"comment":"The phrase 'exceeds most performance upper bound on FWT' is grammatically awkward and should be reworded, especially after the upper-bound concern is addressed.","section":"Experiments, 'Comparison with ORACLE and LLMs'"},{"comment":"The cold-start task orders are given as lists of indices [3, 5, 7, 0, 2, 6, 8, 4, 9, 1] and [3, 2, 4, 0, 6, 5, 1], but the original task names are not listed in the appendix; please provide a mapping to the original task domains for reproducibility.","section":"Appendix B.1"},{"comment":"The hyperparameter ablation lists K=60, 70, 90 but not K=80, the value used in the main experiments; including K=80 would make the sensitivity curve complete.","section":"Table 7"},{"comment":"The caption says 'additional data required refers to extra historical data or unsupervised data', but the figure is not explicitly described in the text; please add a sentence in the Introduction explaining what the axes and symbols represent.","section":"Figure 1(c)"}],"recommendation":"major_revision","confidential_remarks":"The paper is squarely within the scope of a NLP venue and the central no-replay empirical claim is plausible. The main risk is the 'beyond upper bound' overclaim, which can be fixed by rewording, and the lack of variance reporting, which can be fixed by adding run-level statistics. I do not see a circularity in the method itself; the component bias is built from SQL skeletons rather than from fitted parameters, and the empirical gains are credible. I would be comfortable with acceptance after the major points are addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read on arXiv:2412.07246. The core design is solid: using SQL skeletons, clustering to find syntax gaps between tasks, prompting an LLM to generate pseudo-samples to fill those gaps, and then distilling into a small T5 student with a second teacher from the previous student. That's a genuinely replay-free approach for continual semantic parsing, and the cold-start results (which the paper adds on its own) are a useful stress test. The calibration stage—execution checking and edit-distance sampling against the skeleton—gives the generated data a quality gate, and the ablation shows it matters (without MC, ACCa drops from 47.4 to 20.3 on Spider-stream-semi). The paper credits prior work and is straightforward in its exposition.\n\nThe main empirical claim holds up: no-replay LECSP beats replay baselines like SFNET and EMAR on both benchmarks and both backbones. That's a real result, not an artifact.\n\nNow the soft spots. The headline \"beyond upper bound\" FWT claim doesn't survive contact with the setup. ORACLE is trained only on the task stream; LECSP has Mixtral-8x7B as an external teacher, so it gets knowledge from a much larger pretraining corpus. ORACLE isn't an upper bound for a method with extra external resources. The stress-test note gets this right. The right fix is to say LECSP's FWT is higher than ORACLE's because of the LLM's transferable knowledge, not \"beyond the theoretical upper limit.\"\n\nSecond, there are no error bars or significance tests anywhere—just averages over three runs. Given that the reported margins over strong baselines are often small (e.g., 0.8pp over SFNET on some metrics), readers can't tell which differences are trustworthy. That's a standard weakness in this subfield, but it's worth flagging.\n\nThird, no code or data release. The method depends on a specific LLM (Mixtral-8x7B) and a bunch of hyperparameters (K, Nske, Ncfg, lambda, M, R); without code, independent verification is hard. The Appendix covers hyperparameter sensitivity, which is good, but that's not the same as reproducibility.\n\nOne minor thing: the ablation shows an extreme reliance on the memory calibration stage. That's presented honestly, but it means the method's robustness needs more exploration—what happens with a weaker LLM or noisier generation? Table 6 shows some sensitivity, but not across the full pipeline.\n\nWho's this for? People working in continual learning for text-to-SQL, and anyone interested in replay-free continual learning with LLM-generated memory. It deserves a serious referee: the method is novel, the experiments are substantial, and the claim can be fixed with rephrasing. I'd send it to review but with a clear request to report variance and either release code or explain why not. I'd also ask the authors to reread the final paragraph and pull back the 'theoretical upper bound' language.","headline":"Smart, replay-free continual text-to-SQL with a skeleton-guided LLM memory, but the 'beyond upper bound' FWT claim is a resource-comparison artifact.","tokens_in":24777,"tokens_out":2455,"would_cite":true,"duration_ms":23478,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Continual text-to-SQL can beat replay without any historical data by using SQL syntax gaps to guide an LLM's memory reconstruction.","keywords":["continual semantic parsing","text-to-SQL","catastrophic forgetting","SQL skeleton","LLM distillation","memory reconstruction","forward transfer","data-free continual learning"],"falsifier":"Construct a task stream in which a past task's distinguishing SQL pattern (for example, COUNT(DISTINCT ...) with GROUP BY) appears in the training set but never lands close to any of the K cluster centers; if the student then forgets that pattern as badly as sequential fine-tuning, the claim that $\\Delta A^{(t)}$ preserves the syntactically complete memory gap is falsified.","tokens_in":23691,"feed_emoji":"🧩","tokens_out":9113,"duration_ms":92774,"temperature":0.7,"pith_summary":"Continual semantic parsing models forget old SQL knowledge and overfit when each task in a stream has only a few annotated examples, and the usual remedies replay historical data or assume the test task's identity is known in advance. This paper tries to establish that neither is necessary. It claims that the difference between SQL syntax used by earlier tasks and the syntax used by the current task, represented as a set of domain-erased SQL skeletons, tells an open-source LLM exactly which memories to reconstruct, and that calibrated pseudo-samples plus dual-teacher distillation transfer that knowledge into a small parser. If the claim holds, continual text-to-SQL can run without storing past data or requiring task labels at test time, and it can generalize to unseen tasks better than an oracle that trains on all tasks' data at once.","feed_headline":"Beat continual text-to-SQL forgetting without replaying old data","feed_subtitle":"The method fills missing SQL syntax from past tasks via an LLM and distills it into a small parser.","key_machinery":"The load-bearing objects are the SQL skeleton and the component bias built from it. A SQL skeleton is a domain-erased query shape such as SELECT [COL] FROM [TAB] WHERE [COL] = [VAL] GROUP BY [COL], obtained by masking schema entities; the component bias $\\Delta A^{(t)}$ selects skeletons present in the union of previous tasks' cluster-center sets but absent from the current task's set. The bias serves as the instruction to the LLM for what pseudo-sample memory to generate on the current schema, the calibration stage keeps only samples whose SQL executes and whose de-erased form is close to the target skeleton by edit distance, and the dual-teacher distillation propagates the resulting memory into the student while preserving agreement with the previous student's outputs.","core_discovery":"The paper's central discovery is that SQL syntax variance alone can serve as the memory signal in continual semantic parsing. For each task, LECSP erases database-specific entities from question-SQL pairs, clusters the resulting domain-agnostic skeletons with K-means on CodeT5 representations, and keeps K cluster-center skeletons as the task's component feature set $A^{(t)}$. The component bias $\\Delta A^{(t)} = (A^{(1)} \\cup A^{(2)} \\cup \\dots \\cup A^{(t-1)}) - A^{(t)}$ consists of skeletons seen in earlier tasks but missing from the current task; prompting an LLM with $\\Delta A^{(t)}$ and the current database schema produces pseudo question-SQL pairs that fill exactly those gaps. A calibration stage of iterative execution-checked self-correction and edit-distance sampling filters out hallucinated samples, and a task-aware dual-teacher distillation trains the student on original data, pseudo-memory, and KL agreement with the previous student. On Spider-stream-semi with T5-large, the method reaches 58.6 ACCa-EM without replay, versus 52.2 for SFNET with data replay, and its forward-transfer score 44.3 FWT-EX exceeds the ORACLE bound of 36.0.","pith_inferences":["If syntax-variance-driven memory reconstruction is the active ingredient, the same recipe should transfer to other structured generation streams, such as code generation, SPARQL, or JSON transformation, where task differences can be summarized by skeleton shapes.","The forward-transfer scores above ORACLE suggest that the LLM teacher contributes knowledge beyond what the task annotations contain, so the true ceiling for a small parser is not the oracle trained only on task data; explicitly measuring teacher-only knowledge would make this explicit.","A testable extension is to replace the cluster-center skeletons with an exhaustive syntax inventory derived from a grammar; if performance does not improve, the K-center approximation is already sufficient, and if it does, the paper's bias signal is currently under-sampling rare syntax."],"forward_implications":["A continual semantic parser can retain historical task accuracy with zero stored real data, which matters when database contents are private or memory is constrained.","Knowledge can transfer forward to unseen tasks without task identifiers at test time, because the memory is reconstructed from syntax rather than from per-task modules.","The method is robust to cold-start task orders where the first task has few annotations, while prompt-based PET methods degrade sharply in that setting.","Pseudo-memory quality, as measured by SQL execution success, is higher than self-training based pseudo-labels, so the calibration stage is essential to the gains.","Only compact skeleton sets need to be saved across tasks, so the storage cost of remembering past tasks is small and does not grow with data size."],"supporting_citations":[{"why":"Defines Spider-stream-semi and supplies the SFNET self-training replay baseline the method must outperform without replay.","marker":"Chen et al. 2023a"},{"why":"Defines Combined-stream and the PEFT/C3 ideal-setting baselines whose assumptions LECSP rejects.","marker":"Chen et al. 2023b"},{"why":"CodeT5 encodes the domain-erased question and SQL skeleton pairs used to cluster task samples into the component feature set $A^{(t)}$.","marker":"Wang et al. 2021b"},{"why":"Provides the entity-linking and context-free-grammar data synthesis approach reused for domain-information elimination and intra-task memory reinforcement.","marker":"Yu et al. 2021"},{"why":"Introduces Mixtral-8x7B-Instruct, the LLM used as Teacher 1 to generate pseudo-samples from skeletons.","marker":"Jiang et al. 2024"},{"why":"Supplies the self-correction and execution-guided revision idea that the memory calibration stage adapts.","marker":"Pourreza and Rafiei 2023"},{"why":"EMAR is the episodic-memory replay baseline compared with to demonstrate the replay-free advantage.","marker":"Han et al. 2020"},{"why":"Prompt tuning underlies the PEFT/C3 ideal-setting baselines whose dependence on known task identity LECSP avoids.","marker":"Lester, Al-Rfou, and Constant 2021"}],"fun_headline_variants":["SQL syntax gaps replay memory: LLM parser beats replayed baselines","No replay, just SQL syntax: continual parser tops replay methods","Syntax variance guides LLM to beat text-to-SQL forgetting","Beyond upper bound: LLM fills SQL memory gaps without replay"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a limited set of typical SQL query shapes extracted from each task's training examples captures every piece of SQL knowledge the current task lacks, so prompting an LLM with those shapes can reconstruct the missing memory.","fun_headline_variants_meta":{"raw":{"variants":["SQL syntax gaps replay memory: LLM parser beats replayed baselines","No replay, just SQL syntax: continual parser tops replay methods","Syntax variance guides LLM to beat text-to-SQL forgetting","Beyond upper bound: LLM fills SQL memory gaps without replay"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000194,"raw_usage":{"total_tokens":1384,"prompt_tokens":1009,"completion_tokens":375,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":625,"completion_tokens_details":{"reasoning_tokens":302}},"tokens_in":625,"tokens_out":375,"duration_ms":53217,"temperature":1.0,"reasoning_tokens":302,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:55:12.414753+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a task stream in which a past task's distinguishing SQL pattern (for example, COUNT(DISTINCT ...) with GROUP BY) appears in the training set but never lands close to any of the K cluster centers; if the student then forgets that pattern as badly as sequential fine-tuning, the claim that $\\Delta A^{(t)}$ preserves the syntactically complete memory gap is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the self-correction and execution-guided revision idea that the memory calibration stage adapts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"EMAR is the episodic-memory replay baseline compared with to demonstrate the replay-free advantage."}],"review_version":1}