{"id":"799c133a-7340-48d0-b318-d9b02dbbe3ed","arxiv_id":"2412.12541","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An LLM-scored easy-to-hard curriculum for training grammatical error correction models yields small but consistent F0.5 gains over one-shot and length-based training.","lead":"This paper uses a large language model to grade how hard it is to fix each sentence in grammar-correction training data, then trains smaller models on easy sentences before hard ones. The method improves English grammar correction scores by about one point on standard benchmarks and shows that the ordering of training data, not just the data itself, matters.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 3E+2M+1H control in Table 3 may not match the CL schedule's total gradient updates, so the claimed ordering benefit could be confounded with extra training; the paper reports neither update counts nor a threshold sweep.","rationale":"The paper's strongest evidence is the Table 3 comparison between the same-data mixed baseline (3E+2M+1H) and the staged CL schedule, since this is designed to control for data repetition. The load-bearing assumption is that this baseline actually sees each sample the same number of times as the CL schedule. The paper never specifies the number of epochs per stage for CL or the training length for the baseline. The hyperparameter table lists 'Epoches 3' for LLaMA2 SFT but does not say whether that applies per stage or to the whole run. If each CL stage trains to convergence (as stated in §2.2), the easy subset is replayed across all three stages, and the total update count can be several times larger than the 3E+2M+1H baseline. Under that scenario, the F0.5 improvement over the baseline could reflect additional training on the easiest examples rather than the benefit of an easy-to-hard ordering. The hard-to-easy ablation in Table 4 does show that the order matters, but it only compares against the default baseline, not against a matched-update hard-first run, so it does not disentangle ordering from total budget. The paper has other strengths: the effect reproduces across T5 and LLaMA2 families, the length-based curriculum baseline is a reasonable comparison, and the hard-to-easy result is a useful internal control. However, without update counts or a matched-budget retraining, the central quantitative claim that curriculum ordering, not extra training on easy data, produces the gains is not fully established. This is exactly the gap the reader identified, so the CONDITIONAL verdict remains appropriate; a single reporting or re-training check would settle it.","tokens_in":17927,"tokens_out":5997,"duration_ms":48234,"concrete_test":"Obtain the training logs (or rerun with logging) for T5-xl and LLaMA2-13b. For each run, record (a) the number of gradient updates in each CL stage and in total, and (b) the number of gradient updates for the 3E+2M+1H baseline. If the totals differ, retrain the 3E+2M+1H baseline with the same total number of gradient updates (e.g., by increasing epochs on the mixed dataset) and recompute F0.5 on CoNLL14 and BEA19-test. If the gap between LLM-based CL and this update-matched baseline shrinks to within run-to-run variance, the curriculum-ordering claim is not supported. Also report the validation F0.5 at each CL stage to confirm the stopping criterion.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central comparison in Table 3 between LLM-based CL (E->EM->EMH) and the same-data baseline (3E+2M+1H) is intended to show that the curriculum ordering, not just repeated exposure to easy data, drives the F0.5 gains. But the paper never states the number of epochs or gradient updates for either the CL stages or the baseline. The CL schedule is described as training 'until the model converges' at each stage (§2.2), while the 3E+2M+1H baseline is only described as 'trained with a mix of 3 times easy, 2 times medium, and 1 time hard data' (§4.1). If each CL stage runs for multiple epochs (e.g., the LLaMA2 hyperparameters in Table 5 list 'Epoches 3'), an easy sample is seen many more than three times over the three stages, and the total number of gradient updates for the CL run substantially exceeds that of the mixed baseline trained for one or even three epochs. In that case the improvements attributed to curriculum ordering could be partly or wholly due to more total training on the easy subset. The hard-to-easy ablation in Table 4 does control order, but it does not compare against a matched total-update baseline, so it does not resolve the budget confound. Without reported update counts (and ideally a threshold sweep on the 1-3/4-7/8-10 cutoffs), the claim that ordering alone accounts for the gains remains unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LLMCL-GEC, a curriculum-learning method for grammatical error correction. A frozen LLaMA2-70b model scores the correction difficulty of each source sentence in the cLang8 training set on a 1-10 scale; the scored dataset is split into easy (1-3), medium (4-7), and hard (8-10) buckets; and T5-large/xl and LLaMA2-7b/13b are fine-tuned sequentially on easy, then easy+medium, then easy+medium+hard. The authors evaluate on CoNLL14 test, BEA19 test/dev using F0.5 and compare against standard fine-tuning, length-based curriculum learning, a 3E+2M+1H rebalanced baseline, and a hard-to-easy sequence. They report consistent F0.5 gains for the easy-to-hard LLM-based curriculum and conclude that ordering examples from easy to hard is what drives the improvement.","tokens_in":18151,"tokens_out":9188,"duration_ms":77531,"significance":"The paper attacks a practical and timely question: how to order GEC training data for fine-tuned models. Its strengths are the breadth of the empirical matrix (two model families, four model sizes, three benchmarks), the explicit attempt to control for data repetition through the 3E+2M+1H baseline in Table 3, the hard-to-easy ablation in Table 4, and the human-agreement sanity check in Figure 1. If the ordering effect is real, the method is simple and transferable: a strong LLM scores the data once, and smaller models are trained on the resulting curriculum. However, the manuscript provides no code or checkpoints, no multi-seed error bars, and no training-budget accounting, so the central claim that ordering rather than extra updates or arbitrary thresholds drives the gains is not yet fully established. The contribution is currently a promising empirical result conditional on those controls.","major_comments":[{"comment":"The comparison between LLM-based CL (E->EM->EMH) and the 3E+2M+1H baseline is meant to show that curriculum ordering, not repeated exposure, produces the F0.5 gain, but the paper never reports the total number of gradient updates for either condition. The CL procedure in §2.2 trains \"until the model converges\" at each stage, and the only epoch count given (Appendix A.1, Table 5, \"Epoches 3\") is ambiguous as to whether it applies per stage. If each CL stage runs several epochs, an easy example is seen many more times than in the baseline, and the gap in Table 3 (e.g., 69.3 vs 68.6 for T5-xl on CoNLL14; 75.6 vs 74.8 on BEA19-test) could be partly a training-budget effect. The Introduction also describes the baseline as \"summing up the data iteratively trained in curriculum learning,\" which does not exactly match the 3E+2M+1H description in §4.1. Please report epoch/step counts for every condition and clarify the baseline construction; ideally, train the mixed baseline for the same total number of updates and the same data-exposure counts as the CL schedule.","section":"§4.1, Table 3; §2.2; Appendix A.1 Table 5"},{"comment":"The LLaMA2-70b difficulty scores are validated against human judgment on only 200 sentences, with no details on the number or background of the human raters and no inter-annotator agreement. More importantly, there is no evidence that the LLM scores predict the student models' learning difficulty, e.g., through correlation with per-example loss or convergence speed. The 1-3/4-7/8-10 cutoffs are fixed without any sensitivity analysis. Because the entire curriculum is defined by these scores and thresholds, the paper should add a threshold sweep (or at least report bucket sizes and stability under adjacent cutoffs) and a direct correlation between LLM difficulty scores and student-model learning dynamics.","section":"§2.1, Figure 1"},{"comment":"The claim of a \"significant performance boost\" is supported only by a terse \"p<0.01\" notation with no number of seeds, standard deviations, test procedure, or clarification that the BEA19 test set was excluded from significance testing, as stated in §3.1. The reported differences are small (0.4-1.2 F0.5), so single-run numbers are insufficient to establish stability. Please provide means and variances over at least three runs, or paired bootstrap confidence intervals, and state exactly which comparisons and datasets the significance tests cover.","section":"§3.4, Table 2; §3.1"},{"comment":"The conclusion that \"learning from hard to easy is almost ineffective\" rests on a single model (T5-large), a single run, and no significance test. The hard-to-easy and easy-to-hard schedules are also not matched for total gradient updates. This ablation is the only direct evidence for the ordering claim, so it should be replicated on at least one more model and reported with error bars and a matched training budget.","section":"§4.2, Table 4"}],"minor_comments":[{"comment":"There are typos in the abstract (\"pretrianed\") and in the Limitations section (\"aome limitations\"); the paper should be proofread.","section":"Abstract, Limitations"},{"comment":"The notation \"T5-xx/LLaMA2-xx\" is used without defining what \"xx\" ranges over; please spell out the model sizes or define the shorthand.","section":"Table 2 and throughout"},{"comment":"The prompt text appears with whitespace stripped (\"Asagrammarcorrectionexpert...\"), making it hard to read; additionally, the model output in Figure 5 includes a corrected sentence and prose explanation, and the post-editing procedure used to extract the numeric score is not described.","section":"Figure 2, Appendix A.2, Figure 5"},{"comment":"The column labeled \"SAME\" (0.89M) is not explained in the text, and the sizes of easy/medium/hard sum to 1.31M rather than 2.2M; this presumably reflects exclusion of correct-correct pairs, but it should be stated explicitly.","section":"Table 1"},{"comment":"The phrase \"This is the first time we have implemented these as comparable baselines\" is unclear; it should say that the length-based CL baselines are newly implemented in this paper.","section":"§3.3"},{"comment":"The y-axis is not labeled in the caption (it is F0.5), and the error-type names on the x-axis are too compressed to read; a larger figure or rotated labels would help.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the paper is within scope for the venue and the main idea is reasonable, but the verification is currently incomplete. The most important request in revision should be the training-budget-matched comparison and the threshold/difficulty-score sensitivity analysis; without them, the central ordering claim is not fully supported. I see no ethical concerns. The related-work claim that this is the first CL method for GEC may be too strong and should be checked carefully during revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know upfront. First, this is a genuinely new application: curriculum learning had not been tried on GEC before, and using LLaMA2-70b as a difficulty scorer instead of sentence length is a reasonable, clean idea. Second, the central empirical claim is plausible but not fully established, because the control for training budget is under-specified.\n\nWhat the paper does well: the gains are consistent across two model families (T5 and LLaMA2), three benchmarks, and the authors add a useful hard-to-easy ablation that goes in the expected direction. The 3E+2M+1H baseline in Table 3 is a thoughtful attempt to control for data repetition, and the paper is honestly written with adequate literature coverage. No circularity here: the difficulty scores come from a fixed LLaMA2-70b prompt and are independent of the student models and test labels.\n\nThe soft spots are real. The main one is the update-count confound flagged by the stress-test. The CL stages are trained “until convergence,” while the 3E+2M+1H baseline is described as a single mixed training run; the paper never reports epochs or gradient updates for either side. If each CL stage runs multiple epochs (the LLaMA2 hyperparameter table lists “Epoches 3,” ambiguous as to per-stage), easy samples are seen far more than three times, and part of the gain could simply be more training. The hard-to-easy ablation controls order but not total budget, since convergence times may differ between directions. Second, the p<0.01 significance claims come without error bars, number of runs, or a description of the test procedure — a t-test on a single number is meaningless. Third, no code or data is released, and the difficulty thresholds (1-3/4-7/8-10) are arbitrary, validated on only 200 sentences with no sweep. These are all fixable in revision.\n\nThis paper is for GEC practitioners and anyone working on data-ordering methods. The method is simple, cheap to test, and the claim is consequential for training practice; it deserves a serious referee, but not a quick accept. I'd send it to review and ask the authors to report epochs per stage, give update counts or a matched-budget baseline, add error bars or multiple runs, and ideally release the scored data. Without that, it stays a plausible but unverified preprint.","headline":"A plausible, well-executed first application of LLM-scored curriculum learning to GEC, with a real budget-matchedness confound in the main comparison that needs to be resolved before the central claim is trusted.","tokens_in":18786,"tokens_out":3682,"would_cite":false,"duration_ms":33141,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training grammatical error correction models on an LLM-designed easy-to-hard curriculum improves F0.5 over one-shot fine-tuning.","keywords":["grammatical error correction","curriculum learning","LLM-based difficulty scoring","easy-to-hard training","supervised fine-tuning","cLang8","T5","LLaMA"],"falsifier":"Train the same student model on the same data in hard-to-easy order with the same total gradient updates as the easy-to-hard schedule; if the hard-to-easy run matches or beats the easy-to-hard run in $F_{0.5}$ on CoNLL14 and BEA19, the ordering claim fails.","tokens_in":17645,"feed_emoji":"📝","tokens_out":9377,"duration_ms":75035,"temperature":0.7,"pith_summary":"The paper claims that the order in which a grammatical error correction (GEC) model sees its training data is itself a source of accuracy. It introduces LLM-based curriculum learning: a large language model scores each source sentence's correction difficulty on a 1–10 scale, the data are split into easy (1–3), medium (4–7), and hard (8–10) courses, and the student model is fine-tuned first on easy data, then on easy plus medium data, then on all three. On the CoNLL14 and BEA19 benchmarks this schedule beats both one-shot training on all data and a sentence-length curriculum for T5-xl and LLaMA2-13b, adding roughly one point of $F_{0.5}$ in the best configurations. The paper also reports that training hard-to-easy ends near the one-shot baseline, so the direction of the curriculum, not just the selected data, is what carries the benefit.","feed_headline":"Easy-to-hard training order improves grammar correction","feed_subtitle":"An LLM scores sentence difficulty, and fine-tuning on that order beats one-shot training by over a point of F0.5.","key_machinery":"The engine of the method is a difficulty-scoring prompt run by a large LLM on every source sentence in the training set, producing an integer rating from 1 to 10 that partitions the data into easy, medium, and hard courses. The training schedule is the second half of the mechanism: at each stage the model maximizes log-likelihood on the union of all previously seen courses plus the newly introduced one, so easier data are repeatedly rehearsed while harder data are gradually added. This rehearsal design is what distinguishes the staged curriculum from a one-shot mixture of the same examples.","core_discovery":"The central discovery is that an easy-to-hard curriculum built from LLM-assigned difficulty scores makes GEC fine-tuning more effective than training on the same data all at once. The scoring stage uses a large model as a human-expert stand-in: it rates how hard it is to correct each source sentence, and fixed thresholds turn those ratings into three courses. The training stage then optimizes the student model sequentially on easy, easy+medium, and easy+medium+hard data, always retaining earlier data so that adding harder examples does not erase what was learned. With this schedule, T5-xl reaches $F_{0.5}$ of 69.3 on CoNLL14 and 75.6 on BEA19-test, up from 68.1 and 74.5 for the one-shot baseline; LLaMA2-13b reaches 69.6 and 74.3, up from 68.4 and 73.3. The same-data ablation, which mixes three passes over easy data with two over medium and one over hard, stays below the staged schedule, and the hard-to-easy version ends near baseline, which is why the authors attribute the gain to easy-to-hard ordering itself.","pith_inferences":["If the ordering effect transfers, other correction and rewriting tasks, such as text simplification, style transfer, and sentence-level paraphrase, could adopt the same LLM-scored easy-to-hard schedule without any new task-specific data.","The fixed cutoffs at scores 3 and 7 and the 'until convergence' stage lengths are untested choices; sweeping them could either enlarge the reported gains or show that only a brief easy-data warm-up does the work.","The expensive 70b-model scoring step could likely be distilled into a small classifier, making the curriculum pipeline practical at scale if the difficulty scores prove reusable across student models.","A stricter test of the ordering claim would match total gradient updates between staged and one-shot training exactly, since the paper's stage lengths are convergence-based."],"forward_implications":["GEC fine-tuning can be improved without new training data or architecture changes, purely by reordering existing examples according to LLM-scored difficulty.","A general-purpose large language model can stand in for human expert curriculum design, since its difficulty ratings agreed with human ratings on the 200-sentence sample the paper inspected.","Sentence length is an incomplete difficulty signal: length-based curricula improve over the baseline but lag the LLM-scored curriculum under the same staged schedule.","Curriculum direction matters: hard-to-easy training finishes near the one-shot baseline, so future GEC training recipes should start from easier corrections and add harder ones."],"supporting_citations":[{"why":"Introduces curriculum learning, the principle this paper applies to GEC.","marker":"Bengio et al. (2009)"},{"why":"Supplies the T5-based GEC training recipe and cLang8 data used by every student model.","marker":"Rothe et al. (2021)"},{"why":"Provides the LLaMA2 model family, including the 70b model that assigns the difficulty scores.","marker":"Touvron et al. (2023b)"},{"why":"Defines the BEA19 test and development sets used for evaluation.","marker":"Bryant et al. (2019)"},{"why":"Defines the CoNLL14 test set used for evaluation.","marker":"Ng et al. (2014)"},{"why":"Motivates adding previous-stage data back into training to prevent forgetting across curriculum stages.","marker":"Hui et al. (2022)"},{"why":"Provides the sentence-length curriculum baseline that the LLM-scored curriculum is compared against.","marker":"Kocmi and Bojar (2017)"}],"fun_headline_variants":["LLM-scored easy-to-hard order lifts GEC accuracy","Training on easy-to-hard data beats one-shot GEC","LLM rates sentence difficulty for better GEC","Easy-to-hard fine-tuning improves grammar correction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes the large model's scores really rank sentences from easier to harder for the smaller model being trained, and that the staged schedule's advantage is not merely extra training on easy sentences.","fun_headline_variants_meta":{"raw":{"variants":["LLM-scored easy-to-hard order lifts GEC accuracy","Training on easy-to-hard data beats one-shot GEC","LLM rates sentence difficulty for better GEC","Easy-to-hard fine-tuning improves grammar correction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000173,"raw_usage":{"total_tokens":1307,"prompt_tokens":1002,"completion_tokens":305,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":618,"completion_tokens_details":{"reasoning_tokens":242}},"tokens_in":618,"tokens_out":305,"duration_ms":3437,"temperature":1.0,"reasoning_tokens":242,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:58:25.831456+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same student model on the same data in hard-to-easy order with the same total gradient updates as the easy-to-hard schedule; if the hard-to-easy run matches or beats the easy-to-hard run in $F_{0.5}$ on CoNLL14 and BEA19, the ordering claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates adding previous-stage data back into training to prevent forgetting across curriculum stages."}],"review_version":1}