{"id":"eb4eb24a-75bb-4802-9a75-d438e5768035","arxiv_id":"2506.02899","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A reference-free GEC evaluation method that combines grammatical error detection pre-training with IMPARA's quality estimator achieves the highest sentence-level correlation with human judgments on SEEDA-S.","lead":"IMPARA-GED is a new way to automatically score grammar-correction systems without needing human-written reference sentences. It first trains a language model to detect grammatical errors, then uses that model to judge how good a correction is, and this beat existing evaluation methods on a standard benchmark.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"GED gain may be confounded: IMPARA-GED changes both the QE base model and the edit-impact scorer at once, so the reported SEEDA improvement cannot be attributed to GED training alone.","rationale":"The reader's weakest_assumption points to the same sentence in Section 3.2, and I agree that the 'same GED model' impact calculation is the most load-bearing issue. However, I would sharpen the reader's framing: the problem is not necessarily circularity or self-alignment, but a confounded comparison. The with-GED and without-GED rows differ in two ways: the base model receives GED pretraining, and the edit-impact scores used to construct the pairwise training data are computed by the GED model rather than by the original IMPARA impact scorer. Either change could explain the improvement, so the attribution 'GED is boosting' is not identified by the current experiment. This is a correctness risk for the central claim, not merely a stylistic concern. The paper has independent strengths: the comparisons are reported across multiple PLMs, the window and pairwise analyses are informative, and the released model weights on Hugging Face provide some reproducibility. The absence of code is a secondary issue, but the impact-scorer confound is the primary blocker for interpreting the headline result. A four-condition control experiment would settle the attribution, and the reader's CONDITIONAL verdict remains appropriate. I therefore recommend no change to the verdict, with the concrete control test as the explicit condition for confirmation.","tokens_in":10403,"tokens_out":4613,"duration_ms":52268,"concrete_test":"Run four conditions on ModernBERTLarge: (A) current IMPARA-GED with GED pretraining and GED-model-based impact scoring; (B) no GED pretraining but GED-model-based impact scoring; (C) GED pretraining with the original IMPARA impact scorer; and (D) no GED pretraining with the original IMPARA impact scorer. If (C) approximates (D), the GED-specific gain disappears and the improvement is attributable to the impact scorer; if (A) approximates (C) and both exceed (B) and (D), GED pretraining is responsible. Report SEEDA-S and SEEDA-E sentence-level Accuracy and Kendall's tau for all four conditions using identical random seeds and the same devtest-based model selection.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that additional GED training boosts reference-free GEC quality estimation. The comparison 'without GED' versus 'with GED' in Table 2 changes two variables simultaneously. Section 3.2 states: 'The impact calculation is also done using the same GED model.' Section 4 says that after building the GED model, 'the GED model is used to build the quality estimator following the procedure described in §2.' Thus the pairwise training labels (S+, S−) for Equation 1 are generated by the very model that is then fine-tuned as the quality estimator. The w/o-GED rows instead use the original IMPARA pipeline, which relies on a different, presumably similarity-based impact scorer. Consequently, the Table 2 gains, such as ModernBERTLarge sentence-level SEEDA-S Accuracy rising from 0.767 to 0.829, could be caused by replacing the impact scorer with a stronger GED model rather than by the GED task itself. The paper's title and attribution claim, 'Grammatical Error Detection is Boosting', requires isolating the effect of GED pretraining from the effect of the impact-scoring change. A secondary but separate risk is the model-selection procedure in Section 4, where the best of five random seeds is chosen on the CoNLL-2013 devtest set, which may inflate results through selection on a held-out split. A control experiment is therefore needed: keep the impact scoring fixed and vary only GED pretraining, or use an independent model for impact scores while keeping GED pretraining.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes IMPARA-GED, a reference-free grammatical error correction (GEC) evaluation method. Building on IMPARA's quality estimator, it first fine-tunes a PLM on a token-level grammatical error detection (GED) task (2/4/25/55-class labels), then uses that model to construct the pairwise ranking training data and to initialize the quality estimator, and finally removes IMPARA's similarity filter so the evaluation score is just the sigmoid output of the quality estimator. On the SEEDA meta-evaluation benchmark, GED-fine-tuned variants improve sentence-level correlation over their non-GED counterparts across BERT, DeBERTa-v3, and ModernBERT; the best variant, ModernBERTLarge with binary GED, achieves the highest sentence-level SEEDA-S result (Accuracy 0.829, Kendall 0.658) and is second only to GPT-4-S variants on sentence-level SEEDA-E. The paper releases model weights on Hugging Face.","tokens_in":10744,"tokens_out":9310,"duration_ms":101354,"significance":"If the attribution were clean, this would be a practically useful result: a lightweight token-level objective, applied before IMPARA's unsupervised pairwise ranking training, improves sentence-level correlations for all three PLM families tested, and binary labels are sufficient. The paper also contributes a useful negative result about the unreliability of IMPARA's similarity filter with modern PLMs, and the window and pairwise analyses localize the gains to top-ranked system pairs. The public release of the final weights supports reproducibility. However, the current experiments do not isolate the effect of GED training from a simultaneous change in the edit-impact scorer, and the seed-selection protocol may inflate the headline numbers. These issues are load-bearing for the paper's central attribution claim and need to be resolved before the 'GED is boosting' conclusion is accepted.","major_comments":[{"comment":"The comparison between 'without GED' and 'with GED' changes two independent components at once. The paper states that 'The impact calculation is also done using the same GED model' (Section 3.2) and that after GED training 'the GED model is used to build the quality estimator following the procedure described in §2' (Section 4). In the w/o-GED rows of Table 2, the edit-impact scores used to generate the (S+, S−) training pairs necessarily come from a different, non-GED scorer (the vanilla-PLM-based process of the original IMPARA pipeline). The reported gains, e.g., ModernBERTLarge sentence-level SEEDA-S Accuracy rising from 0.767 to 0.829, can therefore be caused either by the GED pre-training objective or by replacing the impact scorer with a stronger error-aware model. To support the title claim, the authors need a controlled experiment that keeps the edit-impact scoring method fixed and varies only whether the quality estimator is initialized from a GED-fine-tuned model; alternatively, they can keep GED pre-training fixed and vary the impact scorer independently.","section":"§3.2 and Table 2"},{"comment":"The final model is selected as 'the one that performs best on the CoNLL-2013 devtest set' among five random seeds. Using the devtest split for model selection, even if only for choosing among seeds after early stopping on the dev set, can overestimate generalization to SEEDA, particularly with only five seeds. The headline ModernBERTLarge results are single-seed numbers chosen on this held-out split. The authors should select the seed on the dev set, report all five seeds with variance, or use a nested validation scheme; without this, the size of the reported improvements is not reliable.","section":"§4, model selection"},{"comment":"The quality estimator is fine-tuned to rank (S+, S−) pairs whose labels are produced by the same GED model that is then used to initialize the quality estimator. If the GED model's edit-impact scores are idiosyncratic, the fine-tuned QE will reproduce those idiosyncrasies, and SEEDA gains could reflect self-alignment with the GED model's preferences rather than a genuinely more human-like grammaticality judgment. This is not fully circular because the GED model is supervised with human error labels, but it is a robustness risk that the current design cannot rule out. An independent impact scorer, or an analysis where impact labels are derived from human annotations directly, would break the loop and strengthen the claim.","section":"§3.2, self-referential training data"}],"minor_comments":[{"comment":"The abstract's claim of 'the highest correlation with human sentence-level evaluations' is too broad. Table 2 shows that on sentence-level SEEDA-E, GPT-4-S +Fluency (Acc 0.831, τ 0.662) and GPT-4-S +Meaning Preservation (0.813, 0.626) outperform the best IMPARA-GED (0.797, 0.594). The claim should be qualified as sentence-level SEEDA-S.","section":"Abstract / §5"},{"comment":"The conclusion that the similarity estimator 'fails' is based on a fixed threshold θ=0.9 inherited from IMPARA. The two illustrative examples show that vanilla PLMs are unreliable similarity scorers at that threshold, but they do not rule out a better similarity estimator or a tuned threshold. Removing the SE is a design choice that should be explicitly justified against such alternatives, or the claim should be narrowed.","section":"§3.1 / Table 1"},{"comment":"The window analysis caption does not specify how the reported Pearson/Spearman values are aggregated over the window or whether the differences are statistically significant; please add this information.","section":"Figure 1"},{"comment":"The sentence 'we did not determine which class type contributes the most to this improvement' is confusing because Table 2 directly compares 2-, 4-, 25-, and 55-class settings. The intended point seems to be that the paper does not analyze the causal mechanism or optimize the label taxonomy; please rephrase.","section":"§7 Limitations"}],"recommendation":"major_revision","confidential_remarks":"The empirical pattern is plausible and the resource release is a plus, but the attribution of the gains to GED training is not supported by the current design because the impact scorer changes at the same time. I would not reject the paper; a controlled experiment and a fix to the devtest-based model selection are sufficient for a revision. If the control experiment is infeasible, the authors should weaken the title claim and present the result as a combined method rather than as evidence that GED training is the cause."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper delivers a genuinely useful result: IMPARA-GED, built by fine-tuning a PLM on GED and then applying IMPARA's quality-estimator construction, lifts sentence-level Kendall on SEEDA-S from 0.506 to 0.658 (ModernBERTLarge, 2-class), and stays competitive with GPT-4-S on SEEDA-E at a fraction of the cost. That is a meaningful step for reference-free GEC evaluation, and the failure analysis of the similarity estimator is the most insightful part—Table 1 shows that for many PLMs the similarity filter is vacuous, and the \"I like cats\"/\"I dislike cats\" example makes the point crisply. The sweep across three PLMs and four label granularities is also thorough, and the window and pairwise analyses give useful detail on where the gains occur.\n\nNow the soft spots, and the main one is real. The w/ vs w/o GED comparison in Table 2 changes at least three things at once: the QE initialization (GED fine-tuning), the edit-impact scorer used to generate pairwise training data (Section 3.2 says the impact calculation uses the same GED model), and the removal of the similarity estimator. So the reported improvement cannot be attributed to GED training alone. It could come largely from a better impact scorer, or from dropping the similarity filter. The title and abstract claim that GED is boosting performance, but that attribution is not isolated. A control experiment—keeping the impact scoring fixed while varying GED pre-training, or using an independent GED model for impact scores—would settle it. This is not a fatal flaw for the method as a recipe, but it does undercut the paper's central explanatory claim.\n\nTwo smaller issues. First, model selection uses the best of five seeds on the CoNLL-2013 devtest set; that can inflate results and should at least be justified or replaced with a fixed-seed protocol. Second, the abstract says \"highest correlation\" on SEEDA, but on SEEDA-E sentence-level, GPT-4-S is marginally better (0.798/0.595 vs 0.797/0.594). That overclaim should be qualified.\n\nCode is not released yet, though model weights are public; the authors say code will follow acceptance, which is acceptable but should be flagged.\n\nOverall, this is a solid empirical paper that deserves a serious referee. The confound is addressable in revision, and the method is likely to be useful even if the causal story needs to be narrowed. I would recommend conditional acceptance, asking for the ablation and a tightened abstract.","headline":"Real gains on SEEDA, but the GED attribution is confounded by simultaneous changes to the impact scorer and the similarity filter, so the causal claim needs an ablation.","tokens_in":11245,"tokens_out":3771,"would_cite":true,"duration_ms":42244,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A grammar-error-detection pretraining step makes reference-free GEC quality estimation track human sentence judgments better than prior metrics.","keywords":["grammatical error correction","reference-free evaluation","grammatical error detection","quality estimation","meta-evaluation","SEEDA","IMPARA","ModernBERT"],"falsifier":"Feed the same GED model human-rated edit impacts instead of its own computed impacts when building the training pairs and meta-evaluate on SEEDA-S; if sentence-level accuracy and Kendall return to the levels of the non-GED baseline, the reported gains are an artifact of self-alignment in the training-signal construction.","tokens_in":10117,"feed_emoji":"🎯","tokens_out":5031,"duration_ms":47155,"temperature":0.7,"pith_summary":"This paper proposes IMPARA-GED, a reference-free automatic evaluation method for grammatical error correction (GEC), and argues that training a quality estimator on the auxiliary task of grammatical error detection (GED) markedly improves how well its scores match human judgments of correction quality. The method takes the existing IMPARA quality estimator and prepends a token-level GED fine-tuning stage, then removes IMPARA's similarity-filtering component entirely because the paper finds that off-the-shelf similarity models either fail to filter or wrongly filter corrections. On the SEEDA meta-evaluation benchmark, the best variant—ModernBERTLarge trained with binary error labels—attains the highest sentence-level accuracy and Kendall correlation on SEEDA-S among all compared methods, and is second only to GPT-4-S on SEEDA-E by a negligible margin. The authors interpret this as evidence that reliable token-level error detection, rather than complex error-type taxonomies or similarity scoring, is what a reference-free GEC quality estimator needs.","feed_headline":"Error-detection training lifts GEC scores toward human judgment","feed_subtitle":"With no reference corrections, IMPARA-GED beats all baselines on SEEDA-S and nearly ties GPT-4-S on edit-level.","key_machinery":"The load-bearing mechanism is the GED-then-QE training pipeline together with the simplified scoring rule $S(I,O)=\\sigma(R(O))$, with no similarity threshold. An encoder-based PLM is first fine-tuned as a token-level GED classifier (using 2-, 4-, 25-, or 55-class labels assigned by ERRANT alignment), then that same GED model is used to compute edit impacts for building pairwise training data, and finally it is fine-tuned on IMPARA's pairwise ranking loss with mean pooling over token embeddings instead of the first-token embedding. The binary-label GED setting carries the reported result.","core_discovery":"The paper's central claim is that a reference-free GEC quality estimator built after token-level GED fine-tuning achieves the highest correlation with human sentence-level evaluations among compared methods. Concretely, ModernBERTLarge with binary GED labels reaches SEEDA-S sentence-level Accuracy 0.829 and Kendall $\\tau$ 0.658, the best of all methods, and SEEDA-E Accuracy 0.797 / Kendall 0.594, a hair behind GPT-4-S (0.798/0.595). The paper argues that this shows GED capability—not similarity filtering or fine-grained error taxonomy—is what boosts quality estimation.","pith_inferences":["Because edit impacts for the training pairs and the final quality estimator come from the same GED model, an external or human-rated impact signal could reveal whether the gain reflects true error sensitivity or self-agreement between the two training stages.","The consistent gains across BERT, DeBERTa, and ModernBERT suggest GED pretraining is a model-agnostic repair for weak grammaticality signals; trying multilingual encoders would test the scope.","Re-adding a genuinely meaning-aware filter to IMPARA-GED could recover protection against adversarial outputs while preserving sentence-level accuracy, provided the filter is trained on grammaticality, not surface similarity.","The strong binary-label result implies that cheap, reliable annotation may beat rich but noisy taxonomy; building GED data by automatic alignment on large parallel corpora might push the method further."],"forward_implications":["Reference-free GEC evaluation can safely drop similarity filtering for outputs of modern GEC systems, at least when adversarial corrections are rare.","Binary token-level error labels deliver better quality estimation than 4-, 25-, and 55-class error taxonomies, suggesting label reliability outweighs label informativeness.","GED training improves system-level correlation mainly for the top-ranked systems and sharpens sentence-level pairwise discrimination between systems far apart in human rankings.","The released IMPARA-GED weights let practitioners score GEC outputs without any human reference sentences or similarity threshold."],"supporting_citations":[{"why":"Supplies IMPARA's quality-estimator construction and pairwise ranking loss that IMPARA-GED reuses.","marker":"Maeda et al. (2022)"},{"why":"Supplies the multi-class GED training setup and the finding that binary labels are reliable, which the method follows.","marker":"Yuan et al. (2021)"},{"why":"Provides SEEDA, the meta-evaluation benchmark and TrueSkill-based settings used for all headline numbers.","marker":"Kobayashi et al. (2024b)"},{"why":"Supplies the reported GPT-4-S and derivative evaluation scores that the paper compares against.","marker":"Kobayashi et al. (2024a)"},{"why":"Provides BERT, the PLM family extended by the method and the base model for the comparison.","marker":"Devlin et al. (2019)"},{"why":"Provides ModernBERTLarge, the model that yields the best reported results.","marker":"Warner et al. (2024)"},{"why":"Provides ERRANT's error-type alignment and label categories used to assign token-level GED labels from parallel data.","marker":"Bryant et al. (2017)"}],"fun_headline_variants":["GED fine-tuning makes reference-free GEC evaluation rival GPT-4","Reference-free GEC score: token-level error detection boosts human agreement","ModernBERT with GED labels tops human-correlation for GEC quality estimates","No references needed: GED pre-training lifts GEC evaluation to peak SEEDA-S","Reference-free GEC evaluation: GED fine-tuning nearly matches GPT-4 on edits"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results stand on the assumption that the edit-impact scores used to create the pairwise training data, computed by the same GED model that is then fine-tuned as the quality estimator, reflect genuine human-judged correction quality.","fun_headline_variants_meta":{"raw":{"variants":["GED fine-tuning makes reference-free GEC evaluation rival GPT-4","Reference-free GEC score: token-level error detection boosts human agreement","ModernBERT with GED labels tops human-correlation for GEC quality estimates","No references needed: GED pre-training lifts GEC evaluation to peak SEEDA-S","Reference-free GEC evaluation: GED fine-tuning nearly matches GPT-4 on edits"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000286,"raw_usage":{"total_tokens":1590,"prompt_tokens":758,"completion_tokens":832,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":374,"completion_tokens_details":{"reasoning_tokens":730}},"tokens_in":374,"tokens_out":832,"duration_ms":7666,"temperature":1.0,"reasoning_tokens":730,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:12:38.172827+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Feed the same GED model human-rated edit impacts instead of its own computed impacts when building the training pairs and meta-evaluate on SEEDA-S; if sentence-level accuracy and Kendall return to the levels of the non-GED baseline, the reported gains are an artifact of self-alignment in the training-signal construction.","supporting_citations":[],"review_version":1}