{"id":"45c2b8ea-f6fb-4129-bdde-46c3fe0f650b","arxiv_id":"2505.19388","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new open-source library unifies ten GEC evaluation metrics and meta-evaluation frameworks, plus new empirical results including an ensemble that reaches 0.984 Spearman on SEEDA-E.","lead":"This paper presents gec-metrics, a unified Python library for evaluating grammatical error correction systems with ten metrics and meta-evaluation tools. It also reports new meta-evaluation results, including the first application of LLM-based metrics to the GJG15 dataset.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"GPT-4-E 'fails to reproduce' claim is unsupported because Appendix C alters model, prompt, context, and hypothesis sampling; no ablation isolates the cause of the discrepancy.","rationale":"The reader's weakest_assumption identifies exactly the load-bearing concern, and I agree. The paper's central contribution is a unified library with a consistent API; that contribution is real and is supported by the presence of code, documentation, and verification for non-LLM metrics. The specific claim that matters most for the experiments is that GEC-METRICS reproduces official metric behavior. For ERRANT, PT-ERRANT, GoToScorer, GLEU, GREEN, SOME, and Scribendi, the implementations are standard and the verification claim is credible. The problem is confined to LLM-based metrics, for which no official implementation exists. The paper's own footnote 6 says the authors received 'some codes and prompts' from Kobayashi et al. and then modified the procedure, confirming that the equivalence claim is not established. Because the abstract's 'fair system comparisons' and Section 5's 'fails to reproduce' statement both rely on that equivalence, the empirical claims about LLM metrics are not yet supported. A favorable reading is that the authors made reasonable adaptations to make LLM metrics applicable to arbitrary datasets; the fix is to relabel those results as a variant, add a controlled comparison on SEEDA with the original configuration, and soften the reproduction-failure language. This does not require rejecting the library or the non-LLM experimental results, so the reader's CONDITIONAL verdict remains appropriate.","tokens_in":14079,"tokens_out":3094,"duration_ms":54126,"concrete_test":"Re-run the SEEDA meta-evaluation for GPT-4-E using the library with the original Kobayashi et al. (2024a) configuration: gpt-4-1106-preview, the original instruction with contextual sentences, and the original pre-sampled hypothesis sets. Compute SEEDA-S and SEEDA-E system-level Pearson and Spearman correlations. If these match the originally reported values, Appendix C's modifications are the cause of the discrepancy, and the 'fails to reproduce' sentence must be revised to 'produces different results under modified settings' (or the default configuration should be changed). If they still do not match, the library's LLM-E implementation is at fault and should be debugged before any GJG15 LLM scores are presented. Additionally, report GJG15 correlations under both configurations; if they differ materially, the first-time GJG15 results should be labeled as coming from a modified metric.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that GEC-METRICS is consistent with official implementations (Section 4) and that the new LLM metric results are informative (Section 5). Appendix C explicitly lists four modifications to Kobayashi et al. (2024a): removing surrounding context, changing the instruction text, selecting up to five hypotheses by frequency rather than using the original pre-sampled hypotheses, and using gpt-4o-mini-2024-07-18 instead of gpt-4-1106-preview. For a stochastic prompted evaluator, any one of these can alter sentence-level scores, and the paper provides no ablation isolating their effects. Consequently, the first-time GJG15 scores for GPT-4-S and GPT-4-E describe a modified metric, not the original LLM-S and LLM-E. Likewise, the claim in Section 5 that GPT-4-E 'fails to reproduce' Kobayashi et al.'s results is not a valid reproduction failure; it compares a modified metric under different data-handling to original reported numbers. The library may still be internally consistent, and the modifications may be reasonable engineering choices, but the 'consistent with official implementations' verification claim and the headline empirical finding about LLM metrics both rest on an equivalence that is never demonstrated. The non-LLM metrics and the ensemble result (which excludes LLM metrics) are less affected, but the paper should not claim to have tested the original LLM evaluators.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces gec-metrics, a unified Python library for grammatical error correction (GEC) evaluation. It implements ten metrics (ERRANT, PT-ERRANT, GoToScorer, GLEU, GREEN, SOME, Scribendi, IMPARA, LLM-S, LLM-E) and two meta-evaluation frameworks (GJG15 and SEEDA), with CLI, Python API, and GUI interfaces, plus analysis and visualization tools. The authors claim that results from gec-metrics are consistent with official implementations for all publicly available metrics (Section 4). They present new meta-evaluation results on GJG15 and SEEDA, including first-time GJG15 scores for LLM-based metrics, an ensemble result, and pairwise analyses. The paper also describes modifications to the LLM-based metrics in Appendix C.","tokens_in":14333,"tokens_out":5809,"duration_ms":38537,"significance":"If the implementation is verified, the library fills a genuine need for a unified, reproducible evaluation framework in GEC, analogous to Hugging Face Evaluate in other fields. The open-source release, API design, meta-evaluation interface, and analysis scripts are valuable contributions. The first public implementations of LLM-based metrics and the new experimental results could be informative, but the strength of these contributions depends on the verification evidence and the careful interpretation of the LLM metric modifications. The paper ships a working library, but the key claim of consistency with official implementations is currently asserted rather than demonstrated.","major_comments":[{"comment":"The sentence \"We have verified that the results obtained using GEC-METRICS are consistent with those from official implementations for all publicly available metrics\" is a central claim of the paper, but no verification details are provided. There are no comparison tables, test cases, or scripts, and it is unclear which metrics have official implementations and what quantitative tolerance defines \"consistent.\" Please provide a concrete verification procedure and results, such as a table comparing corpus-level and sentence-level scores from gec-metrics with the original implementations on a fixed set of inputs for each metric.","section":"Section 4"},{"comment":"The modifications to LLM-S/LLM-E described in Appendix C -- removing contextual sentences, changing the instruction prompt, selecting up to five hypotheses by frequency instead of using the original pre-sampled set, and replacing gpt-4-1106-preview with gpt-4o-mini-2024-07-18 -- mean that the results for GPT-4-E, GPT-4-S, Gemini-S, and Qwen2.5-S in Table 2 are for modified variants, not for the original metrics. Consequently, the statement in Section 5 that \"GPT-4-E fails to reproduce the results reported by (Kobayashi et al., 2024a)\" is not a valid reproduction claim, because it compares a different model, prompt, context, and sampling setup to the original reported numbers. This conclusion should be removed or supported by an ablation that varies one factor at a time. Table 2 should also label these rows as \"modified implementations\" to avoid presenting them as the original LLM-S and LLM-E.","section":"Section 5 and Appendix C"},{"comment":"Because IMPARA's original fine-tuned weights are not publicly available, the authors retrained the model, as stated in Appendix B, and Table 2 reports results from this reproduction. The paper should explicitly disclose this in the experimental setup and in the Table 2 caption, and the Section 4 consistency claim should be qualified: it can only apply to metrics with official implementations, not to IMPARA or the modified LLM metrics. This distinction is important given that Table 1 shows substantial variation in previously reported IMPARA results across studies.","section":"Section 5 and Appendix B"}],"minor_comments":[{"comment":"The ensemble result reporting Spearman 0.984 on SEEDA-E does not specify which metrics were included, how ties in average ranks were handled, or whether the correlation is system-level or sentence-level; please provide these details to make the experiment reproducible.","section":"Section 5, Metric Ensemble"},{"comment":"Table 2 reports correlations without confidence intervals or significance tests; given the small number of systems and the noisiness of system-level correlation, please add bootstrap intervals or otherwise discuss the uncertainty of the reported values.","section":"Table 2"},{"comment":"The claim that the library enables \"fair system comparisons\" by ensuring a consistent implementation is overstated if some metrics are reimplementations (e.g., IMPARA) or modified from the original (LLM-based); consider softening the phrasing to acknowledge the provenance of each implementation.","section":"Section 4.1"},{"comment":"In the list of n-gram groups, \"Over Delete (OD), Over Insert (OD)\" contains a typo; the second abbreviation should be \"OI\" (Over Insert).","section":"Figure 3 and Section 2.1"},{"comment":"There is a typo \"exoerimental settings\" in the first line, and the paper should specify exact versions of all dependencies (not just errant) to fully support reproducibility.","section":"Appendix B"}],"recommendation":"major_revision","confidential_remarks":"The paper is a system demonstration, and the most important issue is the unsubstantiated verification claim in Section 4. If the authors can provide a rigorous comparison with official implementations (including a statement about which metrics are reimplemented), and if they either remove or rework the reproduction-failure claim for LLM metrics, the paper could become acceptable. The LLM metric modifications are not necessarily problematic as engineering choices, but they must be transparently labeled and their implications for the conclusions discussed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read: the library part is solid and worth having. The meta-evaluation experiments are more fragile than the prose suggests, mostly because of the changes to the LLM-based metrics described in Appendix C.\n\nWhat's actually new: gec-metrics is the first unified implementation of the main GEC metrics and meta-evaluation frameworks. That fills a real hole. Several of these metrics had no public code or broken code, so having one installable package with a consistent API, CLI, and YAML configs directly helps reproducibility. The pairwise analysis is a small but reasonable idea. The ensemble result on SEEDA-E is a nice demonstration of what the unified interface makes easy.\n\nCredit where due: the authors contacted the original LLM-S/E authors and received code and prompts; they made their own implementation public; they also released retrained IMPARA weights. That is genuine contribution.\n\nThe soft spots are in Sections 4-5. The claim that results are consistent with official implementations is asserted but never shown. There are no comparison tables or test outputs, only a sentence saying it was verified. That needs evidence, even in a demo paper.\n\nMore important: the LLM-based experiments are described as if they are evaluating the original LLM-S and LLM-E. They are not. Appendix C lists four modifications: no surrounding context, different instruction, frequency-based selection of up to five hypotheses instead of the original pre-sampled set, and a different model (gpt-4o-mini instead of gpt-4-1106-preview). For a stochastic prompt-based evaluator, any of these could change sentence-level scores. The conclusion that 'GPT-4-E fails to reproduce' Kobayashi et al.'s results is therefore not a legitimate reproduction claim. It is a comparison of a modified metric under different data handling to original reported numbers. There is no ablation isolating the cause. This should be fixed by either running the original setup on the original data or rephrasing the claims as results for a new variant.\n\nThe non-LLM results, including the ensemble, are less affected. The IMPARA results are based on a retrained model, so they are approximately comparable but not identical to previous papers; that should be stated more prominently. The meta-evaluation datasets are small and no confidence intervals are given, which makes the correlations noisy. That is a minor issue for a demo, but worth a caveat.\n\nBottom line: this paper deserves peer review, but the version after a referee request should temper the empirical claims. The library is useful now; the experimental section needs either ablations or more careful wording. I would use the library and would probably cite it when reporting GEC results.","headline":"A genuinely useful GEC metrics library, but the new LLM meta-evaluation results are about a modified metric, not the original one.","tokens_in":14859,"tokens_out":2000,"would_cite":true,"duration_ms":17328,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GEC-METRICS unifies ten grammatical error correction metrics and two meta-evaluation frameworks under one interface, and reports new results: first GJG15 scores for LLM-based metrics and an ensemble Spearman of 0.984 on SEEDA-E.","keywords":["grammatical error correction","evaluation metrics","meta-evaluation","unified library","reproducibility","LLM-based evaluation","ERRANT","SEEDA"],"falsifier":"Run the original authors' LLM-S and LLM-E implementations (with gpt-4-1106-preview, original prompt, and context) on the SEEDA or GJG15 datasets and compare the resulting system rankings to those produced by GEC-METRICS; a substantial divergence in rankings or scores would show the modifications changed the metric.","tokens_in":13847,"feed_emoji":"📊","tokens_out":9519,"duration_ms":53949,"temperature":0.7,"pith_summary":"The paper argues that the fragmentation of grammatical error correction (GEC) evaluation software—inconsistent interfaces, missing official implementations, and broken legacy code—has made fair system comparisons and meta-evaluation needlessly hard. It introduces GEC-METRICS, a library that reimplements ten public GEC metrics (ERRANT, PT-ERRANT, GoToScorer, GLEU, GREEN, SOME, Scribendi, IMPARA, LLM-S, LLM-E) and two meta-evaluation suites (GJG15 and SEEDA) behind a single API, CLI, and GUI, with verified agreement to original implementations where available. The library also adds analysis tools such as pairwise analysis, and the paper's own meta-evaluations yield the first GJG15 results for LLM-based metrics, showing they do not generalize as well as on SEEDA, and an ensemble of non-LLM metrics reaching Spearman 0.984 on SEEDA-E. The central promise is that a unified implementation makes evaluation transparent, reproducible, and extensible, so that new metrics can be developed and compared without re-deriving baselines.","feed_headline":"Unified library reproduces all public GEC metrics","feed_subtitle":"One Python package yields fair, reproducible grammar-correction scoring and new meta-evaluation results on LLM metrics.","key_machinery":"The load-bearing machinery is the library's interface design: an abstract Metric class and an abstract MetaEval class that define minimal required methods (e.g., score_sentence()), with every metric and meta-evaluation framework inheriting from them. A Config dataclass separates parameters (e.g., beta, language, model choice) from logic, YAML files make experiment settings reproducible, and the same scoring functions are callable programmatically (Python API), from the command line, or via a GUI. For sentence-level meta-evaluation, the paper also introduces pairwise analysis, which groups system pairs by the difference in human rankings and plots metric agreement, giving a more interpretable view than accuracy or Kendall's tau alone. A second key piece is the modified LLM-based metric pipeline: to make LLM-S and LLM-E applicable beyond SEEDA, the authors exclude surrounding context, change the instruction prompt, select up to five unique hypotheses by frequency among the systems, and use gpt-4o-mini-2024-07-18 instead of the original gpt-4-1106-preview.","core_discovery":"On the paper's own terms, the central discovery is that a single carefully designed library can faithfully reproduce all publicly available GEC evaluation metrics—verified to be consistent with official implementations—and that using this consistent implementation reveals new empirical facts about the metrics: LLM-based evaluators (GPT-4-S, GPT-4-E, Gemini-S, Qwen2.5-S) correlate well with human judgments on SEEDA but poorly on GJG15, GPT-4-E fails to reproduce the results reported in the original LLM-S/E paper, and averaging the rankings of seven non-LLM metrics yields the best system-level correlation (Spearman 0.984 on SEEDA-E). The library's claim is that these results are trustworthy because all metrics were run under the same interface, with shared configurations, rather than citing scores from disparate sources.","pith_inferences":["The modification to select hypotheses by frequency (rather than the original pre-sampled set) could bias LLM-based scores toward majority outputs, possibly masking rare but meaningful corrections; a testable extension would compare frequency-based selection with random selection on SEEDA to measure sensitivity.","Because the LLM-based metrics were evaluated with gpt-4o-mini instead of the original gpt-4-1106-preview, the reported GJG15 failure of GPT-4-E might reflect the weaker model rather than the approach; running the original model with the original prompt on GJG15 would separate these factors.","The pairwise analysis method is not limited to GEC; it could be applied to evaluate any sentence-level metric against human pairwise judgments in other generation tasks such as machine translation or summarization."],"forward_implications":["Researchers and system builders can compare GEC models by running any of the ten metrics on their own hypotheses rather than citing scores from prior papers, eliminating a major source of unfair and irreproducible evaluation.","The first GJG15 results for LLM-based metrics, if confirmed, indicate that strong correlation on SEEDA does not guarantee strong correlation on other meta-evaluation datasets, so LLM evaluators need dataset-specific validation before deployment.","The ensemble result (Spearman 0.984 on SEEDA-E) suggests that combining several complementary metrics—readily done in one library—can approach human agreement, pointing toward ensembling as a cheap way to strengthen evaluation.","Standardizing metric implementations through one package may accelerate research that uses metrics as components, such as reinforcement learning rewards, minimum Bayes risk decoding, and quality-estimation-based system combination."],"supporting_citations":[{"why":"Supplies the ERRANT edit extraction method and F-score baseline that the library reimplements and verifies against.","marker":"Bryant et al. (2017)"},{"why":"Defines PT-ERRANT, the edit-level metric with BERTScore-based weights, which the library implements and checks against its official code.","marker":"Gong et al. (2022)"},{"why":"Introduces GREEN, the n-gram F-score metric, one of the ten metrics the library supports and evaluates.","marker":"Koyama et al. (2024)"},{"why":"Provides GLEU, a precision-based n-gram metric whose original Python 2 code is broken; the library offers a working reimplementation.","marker":"Napoles et al. (2016)"},{"why":"Defines GoToScorer, the correction-difficulty-weighted edit metric that the library reimplements and tests.","marker":"Gotou et al. (2020)"},{"why":"Presents SOME, a reference-free metric fine-tuned on human judgments, which the library uses with the official pre-trained weights.","marker":"Yoshimura et al. (2020)"},{"why":"Defines IMPARA, a reference-free metric based on quality estimation; the library reproduces its training because no public weights existed.","marker":"Maeda et al. (2022)"},{"why":"Introduces LLM-S and LLM-E, the LLM-based evaluators that the library modifies and extends; the paper's new GJG15 results hinge on these implementations.","marker":"Kobayashi et al. (2024a)"},{"why":"Provides the GJG15 meta-evaluation dataset with human rankings, which the library uses to produce its first LLM-based metric scores.","marker":"Grundkiewicz et al. (2015)"},{"why":"Supplies the SEEDA meta-evaluation dataset and the window-analysis method, the primary benchmark for the library's meta-evaluation results.","marker":"Kobayashi et al. (2024b)"}],"fun_headline_variants":["Unified GEC library reproduces metrics, reveals LLM gaps","GEC library unifies scoring, exposes LLM evaluator inconsistencies","All public GEC metrics in one library, with new LLM findings","Averaging non-LLM metrics wins: GEC library insights","GEC library: consistent implementation, surprising LLM results"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the changes made to the LLM-based metrics—removing surrounding context, rewording the prompt, picking the five most common corrections, and using a cheaper model—do not change what the metric measures; if they do, the new GJG15 scores and the failure of GPT-4-E to reproduce earlier results would not tell us about the original metric.","fun_headline_variants_meta":{"raw":{"variants":["Unified GEC library reproduces metrics, reveals LLM gaps","GEC library unifies scoring, exposes LLM evaluator inconsistencies","All public GEC metrics in one library, with new LLM findings","Averaging non-LLM metrics wins: GEC library insights","GEC library: consistent implementation, surprising LLM results"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000545,"raw_usage":{"total_tokens":2531,"prompt_tokens":791,"completion_tokens":1740,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":407,"completion_tokens_details":{"reasoning_tokens":1649}},"tokens_in":407,"tokens_out":1740,"duration_ms":11355,"temperature":1.0,"reasoning_tokens":1649,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:14:04.874189+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the original authors' LLM-S and LLM-E implementations (with gpt-4-1106-preview, original prompt, and context) on the SEEDA or GJG15 datasets and compare the resulting system rankings to those produced by GEC-METRICS; a substantial divergence in rankings or scores would show the modifications changed the metric.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines IMPARA, a reference-free metric based on quality estimation; the library reproduces its training because no public weights existed."}],"review_version":1}