{"id":"136cbdf5-70f6-4ff8-a660-e2293f0e6ad4","arxiv_id":"2505.05122","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Hard-example selection can halve Text2Cypher fine-tuning cost with modest accuracy loss, beating random sampling on the same budget.","lead":"This paper tests five ways to pick hard examples from a Text2Cypher training set, cutting fine-tuning time and cost by half while slightly lowering query-generation accuracy. It offers a practical, heuristic-based recipe for cheaply fine-tuning large language models to translate natural language into graph database queries.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No same-step control: full data at 2.5K steps vs. pruned subsets at 1K steps confounds selection with compute budget, so 'minimal impact' is unestablished.","rationale":"The reader's weakest assumption already flags the 1,000-step vs. 2,500-step mismatch and the potential test-set leakage from reference [12]. I agree with both concerns, but I would reorder them: the more decisive issue is the missing same-budget control for the full dataset. The paper's resource-savings claim is trivial if halving the step count is responsible for the performance drop; hard-example selection is only supported by the comparison against a random subset at the same budget, and that comparison is based on a single run. The Complexity-Based leakage concern is real and should be investigated, but even without leakage the headline 'minimal impact' is not supported by the reported experimental design. This is an experimental design gap, not an allegation of misconduct; the proposed rerun would settle it directly. Since the paper is already conditionally accepted, no verdict change is needed, but the revision should include the same-step full-data evaluation and multiple seeds before the claim can be taken at face value. I credit the paper for studying a practical problem and for including heuristic methods (Length-Based and Cypher-Specific) that do not depend on the prior test-set analysis; those heuristics still need the same-budget comparison to assess their standalone value.","tokens_in":8402,"tokens_out":6694,"duration_ms":67841,"concrete_test":"Retrain the full 39,554-example dataset for exactly 1,000 steps using the same hyperparameters, seed, and hardware as the subset runs, and evaluate on the same test set with the same post-processing. If the full-data-at-1K model matches or exceeds the reported Complexity-Based hard-selection numbers (Google-Bleu 0.7140, translation EM 0.2599, execution EM 0.2639), then the resource-saving/minimal-impact claim is attributable to the shorter training budget rather than to hard-example selection. Run at least three seeds for the full-at-1K, the random subset, and the best hard-selection subset to get error bars, since the current tables report single runs and the core 'hard beats random' difference (0.2599 vs. 0.2048 EM) may be within run-to-run noise.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that hard-example selection reduces resource usage by more than half while minimally affecting performance is not yet established because the headline comparison conflates two independent variables. The full dataset is trained for ~2,500 steps, while every 16,173-example subset is trained for only 1,000 steps (Section 4.2.1 and Table 1). 'Minimal impact' is therefore measured against a full-data model that received 2.5x more update steps, not against a same-compute full-data model. The translation Exact Match drop from 0.3642 to 0.2599 is a 29% relative decrease; we cannot tell how much of that drop is caused by halving the training budget rather than by pruning. The paper does not report full-data evaluation at 1,000 steps, even though it reports full-data loss at that point. The comparison to Randomly-Sampled at the same 1,000 steps (0.6971/0.2048 vs. 0.7140/0.2599) gives some evidence that hard selection beats random, but all numbers are single runs with no seed variance, so the advantage is not quantified. A second, compounding issue: Complexity-Based selection identifies 'hard' databases/data sources from a prior test-set evaluation analysis [12], so the method may be using test information to choose training data; that would inflate its apparent advantage. The decisive missing experiment is the same-step full-data control.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes five hard-example selection methods for pruning the Text2Cypher training dataset: Complexity-Based selection (guided by the author's prior analysis of where models struggle), Length-Based selection, Cypher-Specific term counting, and two combinations of Complexity-Based with each heuristic. All methods produce a 16,173-example subset from the original 39,554 training examples. The experimental protocol fine-tunes a 4-bit quantized 8B Llama 3.1 model on the full dataset for about 2,500 steps and on each subset for 1,000 steps, then compares translation-based (Google-Bleu, Exact Match) and execution-based metrics. The central claim is that hard-example selection roughly halves training time and cost while only minimally affecting Cypher generation performance, and that hard-example selection outperforms random sampling.","tokens_in":1580,"tokens_out":1549,"duration_ms":56736,"significance":"If the central claim were established, the paper would offer a practical, inexpensive data-pruning recipe for Text2Cypher and provide evidence that hardness-based selection is useful for NL-to-query fine-tuning, a relatively underexplored task compared with Text2SQL. The paper has several strengths: it uses a public dataset, it reports hyperparameters and the training seed in Appendix B so that experiments are reproducible in principle, it evaluates with both lexical and execution-based protocols, and it includes a random-sampling baseline. These strengths make the contribution potentially useful to practitioners. However, the headline result is currently confounded by the unequal training budgets between the full-data and subset runs, and by the absence of repeated runs and variance reporting; as a result, the significance of the reported gains over random sampling is not yet quantifiable.","major_comments":[{"comment":"The main comparison varies two factors simultaneously: the pruning method and the number of training steps. The full dataset is trained for roughly 2,500 steps while every 16,173-example subset is trained for 1,000 steps (Section 4.2.1, Table 1). Consequently, the drop from 0.3642 to 0.2599 in translation Exact Match, a 29% relative decrease, cannot be attributed to pruning alone; it may reflect the reduced training budget. The paper reports full-data training loss at 1K steps but does not provide full-data evaluation metrics at 1K steps, nor does it train any subset for 2.5K steps. A same-step full-data control, and ideally a same-budget subset run, is necessary before the claim of 'minimal impact' on performance can be accepted.","section":"Section 4.2.1 and Table 1"},{"comment":"The Complexity-Based selection method chooses databases and data sources where models 'struggled most' according to the author's prior analysis in reference [12]. That reference is a self-cited, non-archival blog post, and as described it is based on evaluation results over the test set. If test-set information is used to define which training examples are hard, the apparent advantage of Complexity-Based selection may be inflated by leakage. The paper should specify the data split used in [12], and either derive the difficulty signal from training/validation data only or demonstrate that the selected databases and data sources are also hard on a held-out split that did not inform the selection rule.","section":"Section 3.1"},{"comment":"All fine-tuning experiments are single runs with a fixed seed (Appendix B, seed=3407). Without multiple seeds, error bars, or significance tests, small differences such as execution-based Exact Match 0.2639 for the best hard-example method versus 0.2550 for random sampling cannot be distinguished from run-to-run noise. The qualitative ranking of methods in Sections 4.2.2 and 4.2.3 may therefore be spurious. Repeating each condition with at least three to five seeds and reporting variance or pairwise significance would make the comparisons reliable.","section":"Section 4.2 and Table 1"},{"comment":"The claim that resource usage, 'both in elapsed time and total cost,' is reduced by more than half is not directly supported by any reported measurement. The only resource metric presented is the number of training steps (Section 4.2.1, Figure 2). Step count is a proxy for, not a measurement of, elapsed time or total cost; wall-clock time and GPU cost can vary with data loading, evaluation, and infrastructure. The paper should either report actual elapsed time and cost figures or explicitly restate the claim as a reduction in training steps.","section":"Abstract, Conclusion, and Section 4.2.1"}],"minor_comments":[{"comment":"The abstract says the goal is to 'preserve or improve performance,' but in every condition the pruned datasets underperform the full dataset in Table 1. The conclusion's phrasing 'minimally affecting Cypher generation performance' is more accurate; the abstract should be revised to match the evidence.","section":"Abstract and Section 1"},{"comment":"The Randomly-Sampled baseline is not a simple uniform random sample: it first caps each data-source group at the 75th percentile (2,755) and then refines the result to 16,173 instances. The fairness of this procedure relative to the hard-selection subsets is not justified, and the paper should clarify whether the random draw is seeded and reproducible.","section":"Section 3.3"},{"comment":"The thresholds that define the subsets, such as the 4,000-instance group cap and the final size of 16,173, are introduced without sensitivity analysis. The comparison across methods could depend on these ad hoc values, so reporting at least one alternative dataset size or group cap would strengthen the conclusions.","section":"Sections 3.1 and 3.2"},{"comment":"The execution-based evaluation is said to run on 'about 50% of the dataset' because it requires an active target database. The exact number of test instances used for that protocol should be reported, because the two evaluation protocols measure different subsets and the paper itself observes that they rank methods differently.","section":"Section 4.1"},{"comment":"There are several small presentation issues: the footnote markers on 'demonstration databases of Neo4j 1 2 ' are awkward, and the data-source names such as 'functional_cypher' and 'text2cypher2023_train' are used without a definition or a pointer to where the dataset fields are described.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of the workshop and the research direction is reasonable. The main problem is experimental design rather than an unsound idea: the missing same-step control, the absence of repeated runs, and the unmeasured cost claim are all fixable with additional experiments. I would support a revised version that addresses these issues."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"My take: this is a reasonable workshop-level first look at pruning Text2Cypher fine-tuning data by selecting hard examples. The application is new as far as I know, and the paper is transparent about his heuristics, reports both lexical and execution-based metrics, and includes a random-sampling baseline. I also give credit for comparing hard selection to random at the same 1K-step budget, where hard selection clearly wins (0.7140 vs 0.6971 Google-Bleu, 0.2599 vs 0.2048 Exact Match). That is a real signal, even if the magnitudes are modest. The soft spots are not minor. First, the headline claim that hard-example selection 'halves cost with minimal impact' is not established because the full dataset is trained for ~2.5K steps while every subset is trained for only 1K steps. The performance gap between full data and subsets could be mostly explained by the extra training budget. The stress-test note is correct: there is no same-step full-data control. Second, the 'Hard Example Selection (best)' row in Table 1 appears to pick the best score per metric across different methods, not a single method's result. That inflates the apparent performance. Report each method separately. Third, the Complexity-Based selection relies on the author's prior test-set analysis (a self-cited Medium post) to identify 'hard' databases and data sources. Using test-set performance to choose training data leaks test information into the selection process, so the method's advantage over random is untrustworthy. Fourth, every number comes from a single run with no error bars, so differences like 0.26 vs 0.25 Exact Match are essentially noise. The paper is honest about some limitations and the writing is clear, so I don't think the authors are trying to hide anything. But the experimental design needs fixing before the central claim can be taken seriously: add a full-data same-step baseline, run multiple seeds, report per-method results, and either drop or properly validate the test-informed selection. Who gets value from this? People working on data pruning for text-to-query tasks, and anyone interested in how evaluation leakage can creep into data selection. It deserves a serious referee rather than a desk reject, because the core question is sensible and the fix is an experimental redesign, not a conceptual dead end. I would not cite the results as evidence until those controls are in place.","headline":"Useful first study of hard-example selection for Text2Cypher, but the cost-saving claim is confounded by unequal training steps and the 'best' row cherry-picks across methods.","tokens_in":747,"tokens_out":711,"would_cite":false,"duration_ms":34134,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Hard-example selection halves Text2Cypher fine-tuning cost with little accuracy loss.","keywords":["hard-example selection","data pruning","Text2Cypher","fine-tuning","large language models","dataset selection","Cypher generation","cost reduction"],"falsifier":"Train the 16,173-example hard subset for 2,500 steps with the same learning-rate schedule and seed as the full-data run, and compare it against a random subset trained identically; if the hard subset no longer beats random sampling or no longer closes the gap to the full dataset, the reported advantage is an artifact of the shorter training budget rather than of example difficulty.","tokens_in":8224,"feed_emoji":"🎯","tokens_out":8587,"duration_ms":69771,"temperature":0.7,"pith_summary":"The paper argues that pruning a Text2Cypher fine-tuning dataset down to fewer than half its examples—by keeping only the hardest queries—cuts training time and cost by more than half with only a modest drop in Cypher-generation accuracy. This matters because fine-tuning LLMs on large synthetic datasets is expensive, and if smaller curated subsets can nearly match full-data models, then training Text2SQL and Text2Cypher systems becomes much cheaper. The paper compares three difficulty heuristics and two combinations against random sampling and the full 39,554-example dataset, and shows that hard-example selection beats random sampling on every reported metric. The strongest subset keeps execution-based Exact Match close to the full-data level (0.2639 versus 0.2740) while using 1,000 training steps instead of 2,500, at the cost of a larger drop in translation-based Exact Match (0.2599 versus 0.3642).","feed_headline":"Hard-example selection halves fine-tuning cost for Text2Cypher","feed_subtitle":"The hardest 16K examples keep most accuracy while cutting steps from 2.5K to 1K.","key_machinery":"The load-bearing mechanism is the selection function that scores each training instance as hard. Three variants are used: Complexity-Based selection, which relies on a prior error analysis to flag specific database and data-source groups; Length-Based selection, which ranks ground-truth Cypher queries by their length; and Cypher-Specific selection, which counts Cypher keywords such as MATCH, WHERE, and RETURN. Two further methods first apply the Complexity-Based filter and then sort by query length or keyword count. All subsets are fixed at 16,173 instances so that the comparison isolates selection quality from dataset size; the fine-tuned model is a 4-bit quantized Llama-3.1-8B-Instruct, trained for 1,000 steps versus 2,500 steps for the full data.","core_discovery":"On the paper's own terms, the central claim is that for the Text2Cypher task, deliberately keeping the most difficult training instances—rather than randomly subsampling—preserves most of the Cypher-generation quality of the full dataset while roughly halving resource use. The Complexity-Based selector identifies databases and data sources where the model struggled (the recommendations, companies, and neoflix demonstration databases; the functional_cypher, synthetic_gemini, and text2cypher2023_train data sources) and keeps 16,173 such instances. This subset reaches Google-Bleu 0.7140 and Exact Match 0.2599 in translation-based evaluation, closing most of the gap to the full dataset (0.7585 and 0.3642) while sharply beating random sampling (0.6971 and 0.2048). Execution-based Exact Match stays at 0.2639 versus 0.2550 for random sampling and 0.2740 for the full dataset. The paper reads this as evidence that cost-effective fine-tuning can be built on hard-example selection, with the length- and keyword-based combinations adding only marginal gains.","pith_inferences":["The same difficulty-focused pruning recipe could transfer to Text2SQL, where datasets such as Spider and IndDB already carry difficulty annotations, making a direct comparison straightforward.","The divergence between translation-based and execution-based rankings suggests future selectors could treat execution success itself as the difficulty signal instead of query length or keyword counts.","Because the Complexity-Based selector is anchored in the author's earlier blog-post analysis, re-deriving the hardest database and data-source groups on an independent model and a held-out split would test whether the difficulty ranking generalizes.","Training the same 16,173-example subset for more than 1,000 steps might close much of the remaining gap to the full dataset; this is implied but not tested in the paper."],"forward_implications":["Training on 16,173 hard examples takes about 1,000 steps instead of 2,500, so elapsed time and GPU cost drop by more than half.","Hard-example selection dominates random sampling: on every reported metric the best hard subset scores above the same-size random subset.","The best hard subset nearly preserves execution-based Exact Match (0.2639 vs 0.2740 for the full dataset), meaning generated Cypher queries still run correctly on the demonstration databases.","The combined heuristics do not hurt performance but add only marginal gains, so the simple Complexity-Based filter carries most of the benefit.","The full 39,554-example dataset still gives the highest absolute scores, so the paper positions hard-example selection as a cost-efficiency tool rather than a free accuracy improvement."],"supporting_citations":[{"why":"provides the prior error analysis of databases and data sources that defines the Complexity-Based selection.","marker":"[12]"},{"why":"is the Text2Cypher dataset used for all training and evaluation splits.","marker":"[13]"},{"why":"supplies the Google-Bleu and Exact Match evaluation metrics.","marker":"[6]"},{"why":"shows that selecting harder instances via cross-entropy improves model performance, motivating difficulty-based pruning.","marker":"[2]"},{"why":"uses difficulty scores such as IFD to select instruction-tuning data, a precedent for the selected subset approach.","marker":"[7]"},{"why":"selects difficult examples alongside diversity for instruction tuning, reinforcing the difficulty-focused subsetting premise.","marker":"[22]"},{"why":"supports the premise that small high-quality datasets can match larger ones in LLM fine-tuning.","marker":"[24]"}],"fun_headline_variants":["Keep the hard stuff: pruning Text2Cypher data cuts cost in half","Hard examples only: half the data, same Cypher quality","Data pruning picks hard cases to halve Text2Cypher training cost","Text2Cypher: hard-example pruning halves training time, holds accuracy","Cypher training on hard examples only: half the cost, same accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The Complexity-Based method depends on the author's earlier analysis of which database and data-source groups are hardest, and that analysis was published only as a blog post and may have been shaped by the same test data or model used here; if that difficulty ranking is biased, the selected 'hard' examples may not be the most informative ones.","fun_headline_variants_meta":{"raw":{"variants":["Keep the hard stuff: pruning Text2Cypher data cuts cost in half","Hard examples only: half the data, same Cypher quality","Data pruning picks hard cases to halve Text2Cypher training cost","Text2Cypher: hard-example pruning halves training time, holds accuracy","Cypher training on hard examples only: half the cost, same accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000772,"raw_usage":{"total_tokens":3407,"prompt_tokens":920,"completion_tokens":2487,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":536,"completion_tokens_details":{"reasoning_tokens":2387}},"tokens_in":536,"tokens_out":2487,"duration_ms":16704,"temperature":1.0,"reasoning_tokens":2387,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:11:55.385921+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the 16,173-example hard subset for 2,500 steps with the same learning-rate schedule and seed as the full-data run, and compare it against a random subset trained identically; if the hard subset no longer beats random sampling or no longer closes the gap to the full dataset, the reported advantage is an artifact of the shorter training budget rather than of example difficulty.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"provides the prior error analysis of databases and data sources that defines the Complexity-Based selection."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"is the Text2Cypher dataset used for all training and evaluation splits."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the Google-Bleu and Exact Match evaluation metrics."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"shows that selecting harder instances via cross-entropy improves model performance, motivating difficulty-based pruning."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supports the premise that small high-quality datasets can match larger ones in LLM fine-tuning."}],"review_version":1}