REVIEW 3 major objections 4 minor 32 references
Adapting LLMs for Minimal-edit Grammatical Error Correction
T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read The paper claims that a simple fine-tuning schedule with unedited-pair augmentation makes a 9B LLM reach F0.5 77.41 on BEA-test, a new single-model state of the art, with 78.70 for the 27B variant.
desk verdict Solid empirical recipe, but the SOTA claim is entangled with an LLM-relabeled training set whose validation is thin. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is a training schedule that moves the precision-recall trade-off from inference into training. It works by ordering the data: after an initial pass over FCE-train, the BEA-train data is split into erroneous and correct halves; the model trains first on erroneous examples and then on correct examples with a much lower learning rate, so the final gradient steps only teach restraint. A second mechanism, unedited-pair augmentation, lowers the data's error rate by creating a correct-to-correct example from every corrected sentence. A third piece is LLM-based detokenization, which aligns model input and output with naturally written text and, as a by-product, corrects a small number of annotation errors. The schedule sets the precision-recall operating point, the augmentation supplies the 'do nothing' behavior, and detokenization changes the data distribution.
What would settle it
Retrain the exact training-schedule recipe on the original tokenized BEA-train and FCE-train data (no LLM modifications, no filtered-out examples) and evaluate on BEA-test; if the resulting F0.5 does not stay above the 76.50 of FLAN-T5 XXL, then the claimed state of the art depends on the detokenization and annotation changes rather than the schedule itself.
Extended reading notes
Core claim
The central claim is that the overcorrection tendency of LLMs in minimal-edit GEC can be controlled during fine-tuning with two simple tools. The first is unedited-pair augmentation: every corrected example is split into an error-correction pair and an unedited pair where the corrected text is both source and target, lowering the dataset's error rate. The second is a training schedule that begins with a general GEC corpus (FCE-train) and then moves through the target corpus (BEA-train) in two halves, first erroneous sentences and then correct sentences at a reduced learning rate; the final low-learning-rate pass teaches the model not to alter already-correct text while preserving its ability to fix errors. On the BEA-test set this yields a new single-model state of the art: F0.5 of 77.41 for Gemma 2 9B and 78.70 for Gemma 2 27B, compared with 76.50 for FLAN-T5 XXL, the best system previously listed. The paper also reports that detokenizing the standard GEC datasets with an LLM exposes annotation errors in roughly 6-8% of checked examples, and that models trained on the corrected detokenized data perform comparably to those on tokenized data while outputting text in a more natural form.
Load-bearing premise
The headline result rests on the assumption that the Llama-3.1-70b model's detokenization of the training targets, plus the manual review of only a 284-example sample per dataset, produced corrected training examples that faithfully represent the minimal-edit task rather than introducing artifacts.
Editorial extensions
If this is right
- Greedy decoding is enough for the reported results; the precision-recall balance is controlled by a learning-rate choice in the final correct-sentence stage rather than by beam search or external tools.
- Keeping (and even adding) unedited pairs is beneficial when fine-tuning LLMs for minimal-edit GEC, the opposite of the older practice of removing unedited pairs for from-scratch neural models.
- The released detokenized and corrected versions of FCE-train, BEA-train/dev, CoNLL-2014-test, and JFLEG let other groups train on the same data and replicate or extend the results.
- The final-stage learning rate is highly sensitive: small changes in value produce large swings in precision and recall, so applying the schedule to a new model or dataset requires tuning.
- Larger is not uniformly better under this schedule: the 27B model scores higher on BEA-test but worse on CoNLL-2014-test than the 9B model, suggesting the same learning rate overtunes the larger model for precision.
Reading between the lines
- Beyond the paper: because the schedule only reorders data and changes one learning rate, it could plausibly transfer to other precision-sensitive generation tasks, such as factuality filtering or citation correction, though the paper does not test this.
- Beyond the paper: the detokenization error analysis is based on manual review of only 284 examples per dataset; a full re-annotation of BEA-train could show whether the claimed state of the art depends on the specific corrected subset used.
- Beyond the paper: the final pass on correct sentences with a low learning rate is a simple form of 'do-nothing' alignment; one could compare it directly with preference-optimization alternatives to see which gives finer control over the precision-recall frontier.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper presents an empirical study of fine-tuning decoder-only LLMs for minimal-edit grammatical error correction (GEC). The authors detokenize the FCE-train, BEA-train/dev, CoNLL-2014-test, and JFLEG datasets using a detokenizer plus Llama-3.1-70b, manually annotate a subset of modified examples, and report finding annotation errors in the original datasets. They then study error-rate adaptation: adding unedited pairs and a data-augmentation method that creates correct-to-correct pairs improves precision-oriented F0.5 on BEA-dev. They propose a training schedule in which the final BEA-train stage first trains on erroneous examples and then on correct examples with a lower learning rate. The best Gemma 2 9B model trained with this schedule achieves 77.41 F0.5 on BEA-test, and a 27B variant achieves 78.70, which the paper claims is a new single-model state of the art compared with the listed prior best of 76.50.
Significance. The practical contribution is potentially valuable: the proposed heuristics are simple, the code and detokenized datasets are released, and the observation that modern LLMs benefit from more unedited pairs rather than fewer is an interesting reversal of earlier practice. The learning-rate sensitivity in Table 6 is clearly demonstrated. However, the headline SOTA claim is not yet fully supported because the training-schedule experiments appear to use the Llama-modified detokenized data without a control on the original tokenized data, and only a small fraction of the modified training targets were manually validated. If these concerns are addressed, the artifacts and findings could be useful to the GEC community.
major comments (3)
- [§6.2–6.3, Table 7] The state-of-the-art claim is confounded with the dataset preprocessing variant. The paper never states which of the four setups from Section 2.2 (tokenized-filtered, detokenized-filtered, tokenized-full, detokenized-full) was used for the training-schedule runs. Section 2.3 says subsequent experiments use the detokenized version "as they contain corrected erroneous annotations," implying the SOTA models were trained on detokenized-full, i.e., with Llama-3.1-70b-modified targets. Table 3 shows that without the schedule, tokenized-full is the best Gemma 2 setup (66.47 F0.5 on BEA-dev), so the schedule's contribution cannot be isolated without a tokenized-full training-schedule control. Please report the exact data version for every model in Table 7 and add a tokenized-full (or at least tokenized-filtered) run with the same training schedule.
- [§2.1, Table 2] The validation of the LLM-modified training targets is too thin to support training the final model on them. For BEA-train and FCE-train only 284 random modified examples were checked; Table 2 reports 9.80% and 12.24% of those as "erroneous" and 6.64% and 3.85% as "not assessable," respectively. Extrapolated to the full modified subsets, this leaves thousands of unvalidated training targets that could introduce non-minimal or incorrect edits. Please validate the remaining modified examples, or at minimum run the training-schedule experiment after excluding all modified examples and compare with the version that includes them, to show that the relabeled data is not driving the BEA-test gain.
- [Tables 5–7] All results are from single training runs with no error bars or repeated seeds. Given that the final learning rate is highly sensitive (Table 6 shows F0.5 swinging from 55.29 to 66.80 across 1e-7 to 5e-7) and the claimed SOTA margin is 0.91 F0.5 over the best listed baseline, the result should be confirmed with multiple seeds and reported with mean and variance, or with a significance test.
minor comments (4)
- [§6.1] The text contains a typo: "Mistal-7b-EPO" should be "Mistral-7b-EPO."
- [Appendix B, Table 1] The detokenization prompt instructs the model to change only spaces, but Table 1 and Table 4 document deletions, insertions, and replacements; please clarify that those operations were unintended modifications later classified as annotation corrections.
- [Figure 1] Figure 1 is referenced in Section 5 and Section 6.2 but appears to be missing from the manuscript; please ensure the figure is included and legible.
- [Table 7] Capitalization is inconsistent: "LLama-2-13b Training-Schedule" should be "Llama-2-13b Training-Schedule" to match the rest of the paper.
Circularity Check
No circularity found: all results are empirical training heuristics with external test evaluation; model selection on BEA-dev is standard practice.
full rationale
The derivation chain is not circular. The paper's contributions are training heuristics (detokenization, augmentation, schedule) evaluated on held-out BEA-test; no claim reduces to its input by construction. Hyperparameters are tuned on BEA-dev and then reported on BEA-test, which is standard model selection rather than fitted-input prediction. The Llama-3.1-70b detokenization modifies only training targets, and the paper explicitly states that test source texts are unchanged and BEA-test targets are hidden, so the SOTA evaluation is external. There are no load-bearing self-citations; the prior works cited (e.g., Junczys-Dowmunt et al., Bout et al., Liang et al.) are external and only frame motivations. The limitations section's admission that BEA-dev LR tuning may overfit the metric is a generalization risk, not circularity. Data-quality concerns about unvalidated relabeled training examples are correctness risks and do not make the reported result equivalent to the input.
Assumptions & free parameters
free parameters (1)
- Final-stage learning rate for correct examples =
3e-7 (chosen from 1e-7 to 5e-7 on BEA-dev)
assumptions (3)
- domain assumption Training first on erroneous examples, then on correct examples with a lower learning rate, increases precision without destroying recall.
- domain assumption The detokenization process with Llama-3.1-70b and manual adjustments faithfully restores natural tokenization and corrects genuine annotation errors without introducing new ones at scale.
- domain assumption LLMs benefit from additional unedited sentence pairs (lower error rate) for minimal-edit GEC, contrary to earlier findings for smaller neural models.
Cite this review
Pith. "Pith review of Adapting LLMs for Minimal-edit Grammatical Error Correction." pith.science (2026). https://pith.science/paper/M2B4GTN6
@misc{pith2026250613148,
author = {Pith},
title = {Pith review of: Adapting LLMs for Minimal-edit Grammatical Error Correction},
year = {2026},
howpublished = {\url{https://pith.science/paper/M2B4GTN6}},
note = {Machine review of arXiv:2506.13148}
}
read the original abstract
Decoder-only large language models have shown superior performance in the fluency-edit English Grammatical Error Correction, but their adaptation for minimal-edit English GEC is still underexplored. To improve their effectiveness in the minimal-edit approach, we explore the error rate adaptation topic and propose a novel training schedule method. Our experiments set a new state-of-the-art result for a single-model system on the BEA-test set. We also detokenize the most common English GEC datasets to match the natural way of writing text. During the process, we find that there are errors in them. Our experiments analyze whether training on detokenized datasets impacts the results and measure the impact of the usage of the datasets with corrected erroneous examples. To facilitate reproducibility, we have released the source code used to train our models.
Figures
Reference graph
Works this paper leans on
-
[1]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Andrey Bout, Alexander Podolskiy, Sergey Nikolenko, and Irina Piontkovskaya. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.355 Efficient grammatical error correction via multi-task training and optimized training schedule . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5800--5816, Singapore. Associati...
-
[4]
Christopher Bryant, Mariano Felice, istein E. Andersen, and Ted Briscoe. 2019. https://doi.org/10.18653/v1/W19-4406 The BEA -2019 shared task on grammatical error correction . In Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 52--75, Florence, Italy. Association for Computational Linguistics
-
[5]
Christopher Bryant, Mariano Felice, and Ted Briscoe. 2017. https://doi.org/10.18653/v1/P17-1074 Automatic annotation and evaluation of error types for grammatical error correction . In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 793--805, Vancouver, Canada. Association for Computat...
-
[6]
Christopher Bryant, Zheng Yuan, Muhammad Reza Qorib, Hannan Cao, Hwee Tou Ng, and Ted Briscoe. 2023. https://doi.org/10.1162/coli_a_00478 Grammatical error correction: A survey of the state of the art . Computational Linguistics, pages 643--701
-
[7]
Hannan Cao, Liping Yuan, Yuchen Zhang, and Hwee Tou Ng. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.185 Unsupervised grammatical error correction rivaling supervised methods . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3072--3088, Singapore. Association for Computational Linguistics
-
[8]
Shamil Chollampatt and Hwee Tou Ng. 2018. A multilayer convolutional encoder-decoder neural network for grammatical error correction. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Inte...
work page 2018
Show all 32 references
-
[9]
Steven Coyne, Keisuke Sakaguchi, Diana Galvan-Sosa, Michael Zock, and Kentaro Inui. 2023. https://arxiv.org/abs/2303.14342 Analyzing the performance of gpt-3.5 and gpt-4 in grammatical error correction . Preprint, arXiv:2303.14342
2023 arXiv
-
[10]
Daniel Dahlmeier and Hwee Tou Ng. 2012. https://aclanthology.org/N12-1067 Better evaluation for grammatical error correction . In Proceedings of the 2012 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies , p...
2012
-
[11]
Christopher Davis, Andrew Caines, O Andersen, Shiva Taslimipoor, Helen Yannakoudakis, Zheng Yuan, Christopher Bryant, Marek Rei, and Paula Buttery. 2024. https://doi.org/10.18653/v1/2024.findings-acl.711 Prompting open-source and commercial language models for grammatical erro...
2024 doi
-
[12]
Wong, Jinpeng Hu, Lidia S
Tao Fang, Shu Yang, Kaixin Lan, Derek F. Wong, Jinpeng Hu, Lidia S. Chao, and Yue Zhang. 2023. https://arxiv.org/abs/2304.01746 Is chatgpt a highly fluent grammatical error correction system? a comprehensive evaluation . Preprint, arXiv:2304.01746
2023 arXiv
-
[13]
Tao Ge, Furu Wei, and Ming Zhou. 2018. https://arxiv.org/abs/1807.01270 Reaching human-level performance in automatic grammatical error correction: An empirical study . Preprint, arXiv:1807.01270
2018 arXiv
-
[14]
Md Asadul Islam and Enrico Magnani. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.239 Is this the end of the gold standard? a straightforward reference-less grammatical error correction metric . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language...
2021 doi
-
[15]
Marcin Junczys-Dowmunt, Roman Grundkiewicz, Shubha Guha, and Kenneth Heafield. 2018. https://doi.org/10.18653/v1/N18-1055 Approaching neural grammatical error correction as a low-resource machine translation task . In Proceedings of the 2018 Conference of the North A merican C...
2018 doi
-
[16]
Shun Kiyono, Jun Suzuki, Masato Mita, Tomoya Mizumoto, and Kentaro Inui. 2019. https://doi.org/10.18653/v1/D19-1119 An empirical study of incorporating pseudo data into grammatical error correction . In Proceedings of the 2019 Conference on Empirical Methods in Natural Languag...
2019 doi
-
[17]
Wei Li and Houfeng Wang. 2024. https://doi.org/10.18653/v1/2024.acl-long.96 Detection-correction structure via general language model for grammatical error correction . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long P...
2024 doi
-
[18]
Wong, Yang Gao, Heyan Huang, and Min Zhang
Yinghao Li, Xuebo Liu, Shuo Wang, Peiyuan Gong, Derek F. Wong, Yang Gao, Heyan Huang, and Min Zhang. 2023. https://doi.org/10.18653/v1/2023.acl-long.380 T emplate GEC : Improving grammatical error correction with detection template . In Proceedings of the 61st Annual Meeting o...
2023 doi
-
[19]
Jiehao Liang, Haihui Yang, Shiping Gao, and Xiaojun Quan. 2025. https://aclanthology.org/2025.coling-main.229/ Edit-wise preference optimization for grammatical error correction . In Proceedings of the 31st International Conference on Computational Linguistics, pages 3401--341...
2025
-
[20]
Mengsay Loem, Masahiro Kaneko, Sho Takase, and Naoaki Okazaki. 2023. https://doi.org/10.18653/v1/2023.bea-1.18 Exploring effectiveness of GPT -3 in grammatical error correction: A study on performance and controllability in prompt-based methods . In Proceedings of the 18th Wor...
2023 doi
-
[21]
Arianna Masciolini, Andrew Caines, Orph \'e e De Clercq, Joni Kruijsbergen, Murathan Kurfal , Ricardo Mu \ n oz S \'a nchez, Elena Volodina, and Robert \"O stling. 2025. https://aclanthology.org/2025.nlp4call-1.1/ The M ulti GEC -2025 shared task on multilingual grammatical er...
2025
-
[22]
Courtney Napoles, Keisuke Sakaguchi, and Joel Tetreault. 2017. https://aclanthology.org/E17-2037/ JFLEG : A fluency corpus and benchmark for grammatical error correction . In Proceedings of the 15th Conference of the E uropean Chapter of the Association for Computational Lingu...
2017
-
[23]
Hwee Tou Ng, Siew Mei Wu, Ted Briscoe, Christian Hadiwinoto, Raymond Hendy Susanto, and Christopher Bryant. 2014. https://doi.org/10.3115/v1/W14-1701 The C o NLL -2014 shared task on grammatical error correction . In Proceedings of the Eighteenth Conference on Computational Na...
2014 doi
-
[24]
Kostiantyn Omelianchuk, Vitaliy Atrasevych, Artem Chernodub, and Oleksandr Skurzhanskyi. 2020. https://doi.org/10.18653/v1/2020.bea-1.16 GECT o R -- grammatical error correction: Tag, not rewrite . In Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building ...
2020 doi
-
[25]
Kostiantyn Omelianchuk, Andrii Liubonko, Oleksandr Skurzhanskyi, Artem Chernodub, Oleksandr Korniienko, and Igor Samokhin. 2024. https://aclanthology.org/2024.bea-1.3 Pillars of grammatical error correction: Comprehensive inspection of contemporary approaches in the era of lar...
2024
-
[26]
Sascha Rothe, Jonathan Mallinson, Eric Malmi, Sebastian Krause, and Aliaksei Severyn. 2021. https://doi.org/10.18653/v1/2021.acl-short.89 A simple recipe for multilingual grammatical error correction . In Proceedings of the 59th Annual Meeting of the Association for Computatio...
2021 doi
-
[27]
Felix Stahlberg and Shankar Kumar. 2021. https://aclanthology.org/2021.bea-1.4/ Synthetic data generation for grammatical error correction with tagged corruption models . In Proceedings of the 16th Workshop on Innovative Use of NLP for Building Educational Applications, pages ...
2021
-
[28]
Xin Sun and Houfeng Wang. 2022. https://doi.org/10.18653/v1/2022.acl-short.77 Adjusting the precision-recall trade-off with align-and-predict decoding for grammatical error correction . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ...
2022 doi
-
[29]
Maksym Tarnavskyi, Artem Chernodub, and Kostiantyn Omelianchuk. 2022. https://doi.org/10.18653/v1/2022.acl-long.266 Ensembling and knowledge distilling of large sequence taggers for grammatical error correction . In Proceedings of the 60th Annual Meeting of the Association for...
2022 doi
-
[30]
Yixuan Wang, Baoxin Wang, Yijun Liu, Qingfu Zhu, Dayong Wu, and Wanxiang Che. 2024. https://doi.org/10.18653/v1/2024.findings-acl.647 Improving grammatical error correction via contextual data augmentation . In Findings of the Association for Computational Linguistics ACL 2024...
2024 doi
-
[31]
Haihui Yang and Xiaojun Quan. 2024. https://doi.org/10.18653/v1/2024.findings-acl.148 Alirector: Alignment-enhanced C hinese grammatical error corrector . In Findings of the Association for Computational Linguistics: ACL 2024, pages 2531--2546, Bangkok, Thailand. Association f...
2024 doi
-
[32]
Helen Yannakoudakis, Ted Briscoe, and Ben Medlock. 2011. https://aclanthology.org/P11-1019 A new dataset and method for automatically grading ESOL texts . In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, p...
2011
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.