{"id":"3fdcedcc-ebb0-4e68-858e-c3b6fe3c48cd","arxiv_id":"2502.09416","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Using TrueSkill aggregation instead of averaging raises the agreement of existing automatic GEC metrics with human system rankings on SEEDA.","lead":"This paper proposes ranking grammar correction systems the way human evaluators rank them, by converting sentence-level metric scores into pairwise comparisons and aggregating with TrueSkill. On the SEEDA benchmark, this procedural alignment improves agreement with human rankings for most existing automatic metrics.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Pairwise conversion of metric scores is the load-bearing step; without sentence-level validation, TrueSkill gains may reflect algorithmic artifacts rather than human alignment.","rationale":"I agree with the reader's identification of the weakest assumption: the conversion of sentence-level metric scores into pairwise comparisons in Section 2.2 is the pivot on which the central claim rests. If numeric score ordering does not track human pairwise preference, then the proposed method is not genuinely 'the same evaluation process as human,' and the observed correlation gains could be produced by TrueSkill's smoothing and tie-handling rather than by resolving the procedure gap. The paper's evidence for the conversion is indirect: improvements on a single benchmark with 14 systems, no significance tests, and no per-sentence comparison against human pairwise labels. A direct pairwise-level validation would settle whether the assumption holds. Because the reader already assigned a CONDITIONAL verdict on essentially this basis, my read does not change the verdict; it reinforces the need for the stated condition. I also credit the paper for a clear articulation of the aggregation mismatch, a simple and reproducible method integrated into gec-metrics, and a consistent pattern of improvements across several metrics, which makes the hypothesis plausible even though the central assumption remains unvalidated.","tokens_in":8893,"tokens_out":5818,"duration_ms":59774,"concrete_test":"On the SEEDA human-annotation data, compute each metric's sentence-level pairwise preference accuracy: for every (source sentence, system pair) with a human pairwise judgment, check whether the metric's score ordering matches the human win/loss/tie. Report accuracy and Cohen's kappa per metric, and compare against the TrueSkill-based ranking correlations in Table 1. If metrics whose ranking correlation improves most (e.g., IMPARA, ERRANT) show near-chance pairwise accuracy, the improvement is an artifact of TrueSkill rather than support for the aggregation-alignment claim. As a control, run TrueSkill on pairwise comparisons obtained by permuting sentence scores within each source sentence; if similar improvements appear, the specific score ordering is not the driver.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.2 converts each metric's sentence-level absolute scores into pairwise comparison results by numeric comparison: 'the evaluation scores of 0.8, 0.7, and 0.9 for corrected sentences corresponding to the first sentence can be compared to produce pairwise comparison results similar to those in human evaluation.' This is the load-bearing step, because the proposed procedure is only 'the same as human evaluation' if the metric's sentence-level ordering is a faithful proxy for human pairwise preference. The paper offers no direct evidence for this assumption. The reported correlation improvements on SEEDA are indirect and confounded: switching from mean aggregation to TrueSkill changes how ties, outliers, and per-system variance are handled, so improved correlation could arise from the rating algorithm's statistical properties rather than from resolving the procedure gap. The issue is especially visible for edit-based metrics, where Appendix A selects the reference with the highest F0.5 per sentence, meaning different systems can be scored against different reference sentences; a higher score then need not correspond to a better correction. Without sentence-level pairwise validation, the central conclusion that 'existing automatic evaluation metrics were underestimated' is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes that automatic grammatical error correction (GEC) evaluation should aggregate sentence-level metric scores using the same procedure as human evaluation, specifically by converting metric scores into pairwise comparisons and applying TrueSkill, rather than averaging absolute scores across the corpus. The authors evaluate seven existing metrics on the SEEDA benchmark under both traditional averaging/summing and their proposed TrueSkill-based aggregation, reporting improved correlation with human rankings for most metrics, and finding that some BERT-based metrics (e.g., IMPARA) outperform reported GPT-4 evaluator results. The method is implemented in the open-source gec-metrics library.","tokens_in":1530,"tokens_out":1834,"duration_ms":48244,"significance":"The central observation—that human and automatic evaluation differ not only in the score source but also in the aggregation procedure, and that aligning this procedure can change system rankings—is an important and largely overlooked point in GEC evaluation. If the result holds, it would improve the practical utility of existing metrics without requiring new metric design, and would provide a concrete recommendation for future meta-evaluations. The paper is also useful as a reproducibility-oriented contribution: Appendix A gives detailed experimental settings, and the integration into gec-metrics lowers the barrier for adoption. The main weakness is that the load-bearing assumption—that numeric comparison of metric sentence-level scores faithfully represents human pairwise preference—is not directly validated; the reported correlation improvements are therefore suggestive but not conclusive.","major_comments":[{"comment":"The proposed procedure rests on converting sentence-level metric scores into pairwise comparison results by simple numeric comparison, as illustrated with the sentence scores 0.8, 0.7, and 0.9. This assumption is not validated at the sentence level. For edit-based metrics (ERRANT, PT-ERRANT), Appendix A states that when multiple references are available, the reference yielding the highest F0.5 is selected per sentence. Consequently, different systems may be scored against different references, and a higher score does not necessarily correspond to a better correction from a human perspective. The paper should provide direct per-sentence evidence that metric score differences align with human pairwise preferences, e.g., by computing pairwise agreement between metric decisions and SEEDA-S human judgments before any aggregation. Without such validation, the reported improvements could arise from the aggregator rather than from resolving a genuine procedural gap.","section":"Section 2.2, Figure 1, Appendix A"},{"comment":"Both the gold-standard human ranking and the proposed automatic ranking use TrueSkill. Because TrueSkill includes priors, tie handling, and rating dynamics, the shared aggregator can inflate rank correlations independently of the quality of the pairwise signals derived from metric scores. The paper does not control for this potential confound. A control experiment that applies the same pairwise-conversion to permuted or random sentence scores, or that swaps TrueSkill for another rating algorithm (e.g., Expected Wins or Bradley-Terry) for both human and automatic rankings, would clarify whether the improvement is due to procedural alignment or to the statistical properties of TrueSkill itself.","section":"Section 2.2, Section 3.3, Table 1"},{"comment":"The meta-evaluations are based on a small number of systems (14 in the +Fluency configuration, fewer in Base), and no significance tests, confidence intervals, or bootstrap analyses are reported. Many of the observed differences, particularly for n-gram metrics (GLEU+, GREEN) where changes are small or even negative, and for edit-based metrics in the +Fluency column, are likely within noise. For example, GLEU+ Spearman drops from 0.902 to 0.846 in SEEDA-S Base, while GREEN improves slightly in one setting and worsens in others. The claim that the proposed method improves results for most metrics should be supported by error bars or a significance test, and the discussion should more honestly reflect the n-gram and +Fluency cases.","section":"Table 1, Section 3.2"},{"comment":"The reported GPT-4 evaluation results are taken from Kobayashi et al. (2024a) and may not be directly comparable because of differences in prompts, decoding settings, or the exact TrueSkill implementation. The paper compares IMPARA and ERRANT against these externally reported numbers to support the claim that BERT-based metrics can outperform GPT-4. To make this comparison fair, the authors should either run GPT-4 under the exact same pairwise-conversion and TrueSkill pipeline used for the other metrics, or clearly state the limitations of comparing numbers from potentially different protocols.","section":"Section 3.3, Table 1, GPT-4 rows"}],"minor_comments":[{"comment":"The phrase 'improves results for the most of metrics' should be corrected to 'for most of the metrics' or 'for the majority of metrics'.","section":"Abstract"},{"comment":"The text 'IMAPARA' is a typo and should be 'IMPARA'.","section":"Section 4"},{"comment":"'integratedGEC-METRICS' is missing a space; it should read 'integrated into GEC-METRICS' or similar.","section":"Section 1"},{"comment":"The labels 'w/o TrueSkill' and 'w/ TrueSkill' are ambiguous; consider using 'conventional (average/sum)' and 'proposed (TrueSkill)' for clarity.","section":"Table 1"},{"comment":"The sentence 'GREEN uses the reference that yields the highest score is selected for each sentence' is ungrammatical; it should be 'GREEN selects the reference that yields the highest score for each sentence.'","section":"Section 3.1, GREEN description"},{"comment":"The figure captions and legend are small and difficult to read, and the window-analysis method is only briefly described in Section 3.2; consider expanding the caption and using larger fonts.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The paper tackles a real and underappreciated question in GEC evaluation, and the proposed change is simple enough to be adopted widely. The central claim is plausible but not yet established: the pairwise-conversion step needs direct validation, and the shared TrueSkill aggregator is a potential confound that should be addressed with control experiments. The lack of significance testing also weakens the quantitative conclusions. These issues are fixable within the scope of the manuscript, so I recommend major revision rather than rejection. The paper fits the journal's scope well."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, the central observation is correct and useful: human GEC evaluation derives rankings by applying TrueSkill to sentence-level pairwise comparisons, while automatic metrics average sentence-level scores and sort. The authors show that applying TrueSkill uniformly to automatic metrics improves correlation with human rankings for most metrics on SEEDA, with IMPARA beating GPT-4-based evaluation in some settings. Second, the paper is honest about its scope and limitations, and it ships code via gec-metrics.\n\nWhat is actually new: the paper is the first to apply the TrueSkill aggregation procedure uniformly across edit-based, n-gram, and sentence-level metric families. Kobayashi et al. (2024a) used it only for their LLM evaluator; this paper turns it into a general prescription and backs it with an empirical map of how each metric family responds. That empirical map, plus the window analysis, is the substantive contribution.\n\nWhere it falls short: the load-bearing step is the conversion of sentence-level metric scores into pairwise comparisons by simple numeric comparison. The paper assumes a higher score for the same source sentence always indicates the better correction. That is not directly validated, and for edit-based metrics the per-sentence reference selection in Appendix A means a higher score can reflect a different reference rather than a better output. The improvement on SEEDA is real but it is one benchmark with 14 systems and no significance tests. TrueSkill changes how ties and outliers are handled, so some of the gain could come from the rating algorithm's properties rather than from resolving the procedural gap. The mixed results on the +Fluency setting for edit-based metrics and the absence of gains for n-gram metrics are consistent with the authors' explanation that sentence-level scores are poor for those families, but they also limit the generality of the claim that existing metrics were underestimated. The limitations section correctly notes that the method is for ranking only, which is a real constraint.\n\nOverall, the paper is worth taking seriously. The central argument is coherent and the empirical trend supports it, but the headline conclusion needs more evidence. A careful referee should ask for significance tests, a second benchmark, and per-sentence validation of the pairwise-conversion assumption. I would accept it for review.","headline":"Clear procedural fix for GEC evaluation with real empirical gains on SEEDA, though statistical evidence is thin and the core pairwise-conversion assumption needs direct validation.","tokens_in":9595,"tokens_out":2957,"would_cite":true,"duration_ms":28332,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that automatic grammatical error correction (GEC) metrics should be ranked using the same aggregation procedure humans use—TrueSkill on sentence-level pairwise comparisons—rather than by averaging sentence-level scores…","keywords":["grammatical error correction","automatic evaluation metrics","TrueSkill","pairwise comparison","system ranking","meta-evaluation","SEEDA","sentence-level scores"],"falsifier":"Take a meta-evaluation benchmark that records human pairwise judgments at the sentence level, and check how often the metric's higher-scored output loses the human pairwise comparison; if the metric's sentence-level score order disagrees with human pairwise choices on a substantial share of sentences, the proposed TrueSkill aggregation cannot be trusted to reveal human preferences.","tokens_in":8715,"feed_emoji":"📊","tokens_out":5080,"duration_ms":41844,"temperature":0.7,"pith_summary":"This paper claims that a simple procedural mismatch explains why automatic grammatical error correction (GEC) metrics appear to disagree with human judgments. Humans rank GEC systems by collecting sentence-level pairwise comparisons and aggregating them with a rating algorithm such as TrueSkill, while automatic metrics average sentence-level scores into corpus-level numbers and sort those. The paper proposes converting each metric's sentence-level scores into pairwise comparisons and running the same TrueSkill aggregation, and shows on the SEEDA benchmark that this improves correlation with human rankings for most of the tested metrics, sometimes substantially. If right, the result means existing metrics such as ERRANT and IMPARA were underestimated in prior meta-evaluations, and that future metrics should be designed to support accurate sentence-level relative comparisons.","feed_headline":"TrueSkill lifts grammar-correction metric rankings closer to human","feed_subtitle":"Switching from score averaging to pairwise TrueSkill aggregation sharpens rankings; some BERT-based metrics beat GPT-4.","key_machinery":"The central object is the aggregation step: instead of averaging per-sentence absolute scores and sorting, the method converts each metric's sentence-level scores into pairwise comparison results for every pair of systems on the same source sentence, then aggregates those comparisons with TrueSkill, the same Bayesian rating algorithm used to produce the human rankings in SEEDA. For N systems this yields N(N−1) comparisons per sentence, including ties. The mechanism works only to the extent that sentence-level scores are reliable enough that 'higher score on the same source' is a proxy for 'better correction,' which is why metrics trained for relative comparison, such as IMPARA, benefit most.","core_discovery":"On its own terms, the paper's central claim is that the evaluation procedure, not just the metric, determines how well automatic GEC evaluation matches human preference. Using the SEEDA benchmark, whose human rankings were produced by applying TrueSkill to pairwise comparisons, the paper derives pairwise comparison outcomes from sentence-level metric scores and then applies TrueSkill to those outcomes. Across edit-based, n-gram-based, and sentence-level metrics, most show higher Pearson and Spearman correlations with human rankings under this procedure than under the conventional average-and-sort procedure. IMPARA in the SEEDA-S +Fluency setting exceeds the reported GPT-4 evaluator, and ERRANT improves by more than 0.2 in several configurations; n-gram metrics such as GLEU+ and GREEN do not improve, which the paper traces to unreliable sentence-level scores for short sentences. The paper's core statement is not that TrueSkill is the right aggregator, but that automatic evaluation should mirror whatever aggregation method the human evaluation used.","pith_inferences":["A testable extension is to train or fine-tune GEC metrics directly on pairwise preference data and compare their TrueSkill-based rankings against those of metrics trained on absolute scores; the paper's reasoning predicts the pairwise-trained metric will win.","The pairwise conversion could amplify a metric's systematic sentence-level biases instead of removing them; checking agreement between metric score order and human pairwise judgment per sentence would tell whether the improvement generalizes beyond SEEDA.","The implicit recommendation for GEC researchers is to publish all compared system outputs, since the relative ranking method requires re-running all systems rather than citing previously reported corpus scores; this is a reproducibility-friendly side effect the paper notes explicitly.","If the same aggregation logic applies elsewhere, multilingual GEC and other sentence-level generation tasks with pairwise human evaluation could adopt the same procedure."],"forward_implications":["Meta-evaluations that keep the average-and-sort procedure understate how well existing edit-based and sentence-level metrics align with human preference.","Metrics such as ERRANT and IMPARA become more competitive with, and in some settings surpass, LLM-based evaluators such as GPT-4 when evaluated the same way humans are ranked.","Developing sentence-level relative metrics, trained or tuned on pairwise comparisons, becomes a priority over corpus-level aggregate design.","The same principle transfers to other meta-evaluation datasets: follow whatever aggregation method the human evaluation used, such as Expected Wins, if that is what the dataset relies on.","n-gram metrics need sentence-level calibration before they can benefit from relative aggregation."],"supporting_citations":[{"why":"Supplies the SEEDA benchmark and the human TrueSkill rankings used as the meta-evaluation target; its observation that evaluation granularity matters motivates the paper.","marker":"Kobayashi et al. (2024b)"},{"why":"Defines TrueSkill, the rating algorithm the paper applies to pairwise comparisons derived from metric scores.","marker":"Herbrich et al. (2006)"},{"why":"Documents that human GEC evaluation has used sentence-level pairwise comparisons aggregated by a rating algorithm, establishing the procedural baseline the paper aligns with.","marker":"Grundkiewicz et al. (2015)"},{"why":"Provides the GPT-4 evaluator results and window-analysis methodology the paper compares against and extends.","marker":"Kobayashi et al. (2024a)"},{"why":"Defines IMPARA, a sentence-level metric trained on pairwise comparison data whose TrueSkill-based results exceed GPT-4 in one SEEDA setting.","marker":"Maeda et al. (2022)"},{"why":"Defines ERRANT, the edit-based metric whose correlation with human rankings improves by more than 0.2 in several configurations under TrueSkill aggregation.","marker":"Felice et al. (2016); Bryant et al. (2017)"},{"why":"Defines GLEU+, an n-gram metric whose lack of improvement illustrates that sentence-level score quality limits the proposed aggregation.","marker":"Napoles et al. (2015, 2016)"}],"fun_headline_variants":["TrueSkill aggregation improves GEC metric rankings","Match human eval: TrueSkill beats score averaging for GEC","GEC metrics rank humans better with TrueSkill, not averaging","TrueSkill ranking aligns GEC metrics with human preferences","Pairwise TrueSkill lifts GEC evaluation correlation with humans"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that comparing numerical metric scores for the same source sentence is equivalent to a human pairwise judgment, so when a metric gives a higher score to one correction than another, the higher-scored correction really is the better one; the paper offers only indirect evidence for this from one benchmark, with no per-sentence validation.","fun_headline_variants_meta":{"raw":{"variants":["TrueSkill aggregation improves GEC metric rankings","Match human eval: TrueSkill beats score averaging for GEC","GEC metrics rank humans better with TrueSkill, not averaging","TrueSkill ranking aligns GEC metrics with human preferences","Pairwise TrueSkill lifts GEC evaluation correlation with humans"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000218,"raw_usage":{"total_tokens":1420,"prompt_tokens":909,"completion_tokens":511,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":525,"completion_tokens_details":{"reasoning_tokens":432}},"tokens_in":525,"tokens_out":511,"duration_ms":4911,"temperature":1.0,"reasoning_tokens":432,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T21:34:53.441750+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a meta-evaluation benchmark that records human pairwise judgments at the sentence level, and check how often the metric's higher-scored output loses the human pairwise comparison; if the metric's sentence-level score order disagrees with human pairwise choices on a substantial share of sentences, the proposed TrueSkill aggregation cannot be trusted to reveal human preferences.","supporting_citations":[],"review_version":1}