{"id":"577bbf08-a57b-4766-ab69-38b8a2a4847a","arxiv_id":"2412.13110","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Sentence-level GEC metric scores are decomposed into per-edit Shapley attributions, enabling edit-level explanation and error-type analysis.","lead":"This paper shows how to break a sentence-level grammar correction score into per-edit contributions using Shapley values, so you can see which fixes helped or hurt. It tests this on several GEC metrics and reports around 70% agreement with human judgments while exposing metric biases.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 4.3's merge-consistency hypothesis is an unproven and generally false property of Shapley values, so Figure 3 does not establish granularity-consistency as claimed.","rationale":"The reader's weakest_assumption identifies exactly the same load-bearing issue: Section 4.3 relies on an unproven merge-consistency property of Shapley values. I agree this is the central soft spot because the paper's headline 'high consistency across edit granularities' is substantiated only by this evaluation. The property is known to be false for Shapley values in general, and the paper gives no argument for why metrics like SOME, IMPARA, or GPT-2 PPL should satisfy it. A direct empirical or counterexample-based test would settle the matter. I also considered two other concerns. First, the human-evaluation protocol in Section 4.4 selects, per sentence, the reference that maximizes agreement with the attribution scores; this can inflate the reported ~70% alignment and should be reported per-reference or as an upper bound. This is a real weakness but is secondary because the paper itself says human alignment is not the primary goal. Second, dependent edits can make the value function in Eq. (2) ill-defined for some edit sets; the authors acknowledge this in the Limitations and argue Shapley partially captures dependencies, so it is a recognized limitation rather than an unacknowledged flaw. On the positive side, the method is standard Shapley attribution, the code is available, and the paper is transparent about computational trade-offs and limitations. The conditional verdict remains appropriate: the merge-consistency and reference-selection issues should be addressed or the claims softened, but the core attribution framework and case studies still have value.","tokens_in":12356,"tokens_out":5067,"duration_ms":52689,"concrete_test":"Using the released code, sample a few hundred JFLEG sentences with 4-8 edits. For each sentence, compute exact Shapley values for the original edit set, then merge edits with the same sign into one grouped edit and compute exact Shapley values for this reduced game. Report the signed difference and absolute error for each group: phi_grouped - sum(phi_component). If the mean absolute error is material (e.g., changes the sign or exceeds 10% of the group's sum), Section 4.3's consistency claim is refuted and the paper must soften it to 'sign agreement under a heuristic check' or provide a proof of merge-invariance for these metrics.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The faithfulness evaluation in Section 4.3 assumes that if same-sign edits are merged into a single grouped edit, the Shapley value of the grouped edit equals the sum of the Shapley values of its components. This is not a general property of Shapley values, and the paper provides no proof for the specific value functions in Eq. (2). A concrete counterexample with three players and value v(S)=1 if |S|>=2 else 0 gives component Shapley values 1/3 each, while the merged two-player game gives the merged player value 1, not 2/3. Since the value functions here are determined by arbitrary sentence-level metrics, nothing in the setup enforces the needed merge-invariance. The paper's consistency metrics (sign match ratio and Pearson/Spearman correlations) are weaker than the stated equality hypothesis; they can be high even when the equality fails substantially. Therefore the claim 'Shapley shows higher consistency' and the broader claim that attribution is reliable across edit granularities are not supported as faithfulness statements. The same issue also affects the comparison to Add/Sub baselines, since all three methods are evaluated under the same unvalidated grouping assumption. This is the most load-bearing weakness because it is the quantitative evidence for the central 'high consistency across edit granularities' claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes to make sentence-level reference-free GEC metrics explainable by attributing the score difference ΔM(H|S) = M(H|S) − M(S|S) to individual edits via Shapley values. The method computes Shapley values over the game induced by Eq. (2), uses Shapley sampling to reduce cost, and applies L1 normalization to make scores comparable across metrics. The evaluation has two parts: faithfulness, assessed by a consistency test across edit granularities (§4.3), and explainability, assessed by agreement with reference-based human labels (§4.4). The paper reports roughly 70% human alignment, higher consistency for Shapley than Add/Sub baselines, and an application to error-type precision analysis that reveals, e.g., downplaying of orthographic edits. Experiments use SOME, IMPARA, and GPT-2 perplexity on CoNLL-2014 and JFLEG with five GEC systems.","tokens_in":12620,"tokens_out":3774,"duration_ms":37033,"significance":"If the central claims were fully validated, the paper would provide a practical tool for diagnosing the behavior of reference-free GEC metrics and for giving edit-level feedback, which is a useful contribution to GEC evaluation and explainability. The paper has clear strengths: a public implementation, use of a principled game-theoretic attribution framework, a concrete efficiency analysis (§4.5), and application to multiple metrics, datasets, and systems. However, the faithfulness evidence in §4.3 rests on an unproven merge-consistency property that is false for general Shapley games, and the human evaluation in §4.4 uses a reference-selection procedure that can inflate agreement. These issues bear directly on the paper's two headline claims, so the contribution is currently promising but not yet adequately supported.","major_comments":[{"comment":"The consistency evaluation assumes that if same-sign edits are grouped into a single edit, the Shapley value of the grouped edit equals the sum of the Shapley values of the component edits. This property is not true for general Shapley values, and the paper provides no proof for the specific value functions in Eq. (2). A concrete counterexample: for three players with v(S)=1 if |S|≥2 and 0 otherwise, each Shapley value is 1/3; merging two players into one gives a two-player game with w({A,3})=1, so the merged player's Shapley value is 1/2, not 2/3. Since the value functions here are defined by arbitrary sentence-level metrics, nothing in the setup enforces merge-invariance. The consistency metrics (sign match ratio, Pearson/Spearman correlation) are weaker than the stated equality hypothesis; they can be high even when the equality fails substantially. Therefore the claim that 'Shapley shows higher consistency' and the broader claim that attribution is reliable across edit granularities are not supported as faithfulness statements. This is load-bearing because it is the quantitative evidence for the central consistency claim. I recommend either proving the merge-invariance property for the metrics considered, or replacing this evaluation with a direct behavioral test of attribution quality (e.g., comparing attribution signs against the metric's sensitivity to individual edits or against deletion/insertion curves).","section":"§4.3, Eq. (2)"},{"comment":"The human evaluation selects, among multiple references, 'the reference that results in the highest agreement with the attribution scores' before computing the agreement ratio. This is a form of selection bias: the same attribution scores are used both to choose the reference and to measure agreement, which can inflate the reported agreement numbers and explains part of the 'approximately 70% alignment'. The paper should report agreement separately for each reference, as well as an average or a principled aggregation (e.g., an edit is correct if it appears in any reference, or majority voting across references). Without this, the human-alignment claim is not a fair estimate of explainability.","section":"§4.4"},{"comment":"The Shapley sampling evaluation reports an average absolute error of 0.066 for IMPARA against a distribution of absolute Shapley values of 0.052 ± 0.071. The paper notes that IMPARA 'exhibits higher errors' but does not quantify what this means for the downstream claims: an error larger than the average attribution magnitude can flip signs for a substantial fraction of edits, undermining the reliability of the attribution scores that are later used in §5.1 and §5.2. I recommend reporting the sign-flip rate under sampling and, if necessary, increasing T or flagging metrics for which the sampling approximation is too coarse.","section":"§4.5, Table 2"}],"minor_comments":[{"comment":"The abstract states 'approximately 70% alignment with human evaluations', but Figure 4 shows agreement values that vary with threshold and metric, and the paper does not report a single overall number with a confidence interval. Please state the exact range and how it was computed.","section":"Abstract / §4.4"},{"comment":"The axis label 'Peason' should be 'Pearson'.","section":"Figure 3"},{"comment":"The notation for the empty edit set appears as 'ϕ' or '{ϕ}', which is confusing; use ∅ consistently. Also, Eq. (3) writes ∆M(S{e1,e2}|S) while Eq. (2) uses ∆M(Se′|S); please unify the notation.","section":"Eq. (3) and surrounding text"},{"comment":"The system is called 'GTP-4o mini' in the text but 'GPT-4o mini' elsewhere; please fix the typo.","section":"§4.1.2"},{"comment":"The sentence 'It is important to not that the attribution scores reflect the internal decisions of the metric' contains a typo: 'not' should be 'note'.","section":"§5.1"},{"comment":"The L1 normalization in Eq. (5) is undefined when all attribution scores are zero; please state how such cases are handled.","section":"§3, Eq. (5)"}],"recommendation":"major_revision","confidential_remarks":"The paper is clearly written and the code release is commendable. The main risk is the unfounded merge-consistency assumption underpinning §4.3; a reviewer will likely reject the consistency claim without either a proof for this metric class or a replacement evaluation. The human evaluation's reference-selection procedure is also likely to be seen as optimistic. If the authors can rework these two evaluation components, the paper could become a solid contribution. The scope fits the journal well as an explainability/evaluation paper for NLP."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper brings something new to GEC evaluation: it treats the difference between source and corrected sentence scores as a cooperative game over edits and distributes it with Shapley values. That framing is not in the earlier feature-attribution work, and it gives researchers and teachers a way to see which specific corrections drive a sentence-level metric. The implementation is linked, the experiments cover several metrics, systems, and datasets, and the authors are unusually candid in the Limitations section. The efficiency discussion is honest, and the error-type bias analysis is genuinely useful.\n\nThe soft spots are real. Section 4.3's consistency evaluation assumes that if you group edits with the same sign, the Shapley value of the grouped edit equals the sum of the components. That is not a general property of Shapley values, and the paper supplies no proof for the specific value functions here. A three-player threshold game shows the equality can fail outright, and the sign-match and correlation metrics used in Figure 3 do not test the equality. So the 'high consistency across edit granularities' claim is not established as a faithfulness statement. It is a claim about a convenient property, not a demonstrated property.\n\nThe human evaluation also has a selection effect: when multiple references exist, the paper picks the reference that maximizes agreement with the attribution scores. That can inflate the reported 70% alignment, and there are no error bars or significance tests anywhere. These two issues together mean the headline numbers should be treated as upper estimates.\n\nThe central method, however, is a forward computation on a fixed metric, and the case studies and corpus-level precision analysis are compelling. The paper is not trying to replace reference-based metrics; it offers a diagnostic for GEC evaluation and educational feedback. As such, it deserves peer review, but a referee should ask for a proof or a proper empirical justification of the merge-consistency assumption, reference-averaged human agreement instead of oracle selection, and variance estimates. Send it to review, but expect revision.","headline":"Useful Shapley-based edit attribution for GEC, but the consistency and human-alignment evidence is weaker than claimed.","tokens_in":13093,"tokens_out":2815,"would_cite":true,"duration_ms":25284,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Attributing sentence-level GEC metric scores to individual edits via Shapley values yields edit-level explanations that reveal which corrections helped and which hurt.","keywords":["Grammatical Error Correction","Shapley values","edit-level attribution","reference-free evaluation metrics","explainability","metric bias","human evaluation agreement","Shapley sampling values"],"falsifier":"Take GEC sentences with three or four edits, compute exact Shapley values for each edit and for the edit formed by merging two same-sign edits, and check whether the merged value equals the sum of the two individual values; a systematic gap across many sentences would falsify the claim that consistency across granularities supports reliability.","tokens_in":12141,"feed_emoji":"📝","tokens_out":7004,"duration_ms":55081,"temperature":0.7,"pith_summary":"Reference-free grammatical error correction (GEC) metrics give a single sentence-level score with no indication of which edits drove the score. This paper proposes attributing the score difference between the source and corrected sentence to individual edits using Shapley values, so each edit receives a signed contribution: positive for edits that help the metric, negative for edits that hurt it, with magnitude reflecting impact. Applied to the reference-free metrics SOME, IMPARA, and GPT-2 perplexity, the method reports consistent attributions across edit granularities and about 70% agreement with human judgments. The authors use these attributions to expose metric biases, such as a tendency to downplay orthographic edits, and to decompose corpus-level scores into precision per error type.","feed_headline":"Shapley values turn GEC metric scores into per-edit explanations","feed_subtitle":"Each edit gets a sign and magnitude, exposing metric bias and reaching about 70 percent agreement with humans.","key_machinery":"The central object is the Shapley value of each edit under the game whose payoff is the sentence-score difference $\\Delta M(H|S) = M(H|S) - M(S|S)$. To keep the computation tractable the paper uses Shapley sampling values, which average the marginal contribution of an edit over $T$ uniformly sampled permutations of all edits; L1 normalization of the resulting scores makes attributions comparable across sentences and metrics. This machinery distributes the entire score difference exactly across edits, which is what makes sign and magnitude read as benefit and impact.","core_discovery":"The paper's central claim is that the opaque sentence-level output of a reference-free metric can be made explainable at the edit level by treating the edits between source and correction as players in a cooperative game whose total payoff is $\\Delta M(H|S) = M(H|S) - M(S|S)$. The Shapley value of each edit distributes this payoff fairly over all subsets and orders of edits, and the sign of the resulting attribution indicates whether the metric considers the edit helpful, while the absolute value gives its influence on the final score. The authors demonstrate this on three metrics and five GEC systems, finding that Shapley-based attributions are more consistent across edit granularities than simple add/remove baselines and that higher-magnitude attributions align better with reference-based human judgments. They further show the attributions can be averaged over error types to reveal metric biases and used to compute per-error-type precision at the corpus level.","pith_inferences":["If the grouping-sum property used in the consistency evaluation fails for Shapley values, the faithfulness claim weakens, but the human-alignment evidence for explainability would still stand on its own.","The same attribution scheme should transfer to text simplification or any edit-based generation task where a sentence-level quality metric exists, since the method only needs a source, a hypothesis, and the metric.","Because Shapley values exactly decompose the score difference, they could be used to debug reference-free metrics by tracing surprising sentence-level scores back to the specific edits that caused them.","A direct human study rating the attributions themselves, rather than comparing them to reference-based labels, would test whether the signs are useful as learner-facing feedback."],"forward_implications":["Researchers can now ask which specific edits a reference-free metric rewards or penalizes, turning a single score into a per-edit diagnosis of a GEC model's output.","Corpus-level scores decompose into precision per error type, making it possible to see that a system's high overall score may hide low precision on certain error classes.","Shapley sampling brings the attribution cost down to about one second per sentence for outputs with up to roughly ten edits, making the analysis practical on standard benchmarks.","The observed metric biases, such as downplaying orthography, imply that users should interpret sentence-level scores with the metric's weighting preferences in mind.","Higher absolute attribution scores agree more often with human judgments, so the magnitude can serve as a confidence signal when using the attribution as feedback."],"supporting_citations":[{"why":"Defines the Shapley value, the cooperative-game-theory mechanism used to distribute the sentence-score difference across edits.","marker":"Shapley et al., 1953"},{"why":"Supplies the sampling approximation that reduces attribution cost from exponential to roughly one second per sentence.","marker":"Strumbelj and Kononenko, 2010"},{"why":"Provides the SOME reference-free metric, one of the three sentence-level metrics the attribution method is applied to and analyzed on.","marker":"Yoshimura et al., 2020"},{"why":"Provides the IMPARA reference-free metric, another target metric whose attribution behavior is measured and compared.","marker":"Maeda et al., 2022"},{"why":"Provides the GPT-2 language model used to build the perplexity-based fluency metric, the third target metric.","marker":"Radford et al., 2019"},{"why":"Defines the edit extraction and error-type categories used to group edits and to compute per-error-type attributions.","marker":"Bryant et al., 2017"},{"why":"Provides the CoNLL-2014 test set, one of the two datasets on which consistency and human-alignment are evaluated.","marker":"Ng et al., 2014"},{"why":"Provides the JFLEG validation set, the fluency-oriented dataset used for the other evaluation.","marker":"Napoles et al., 2017"}],"fun_headline_variants":["Shapley values explain GEC scores edit by edit","Attributing GEC metric scores to individual edits via Shapley","Edit-level explanations for sentence-level GEC metrics","Shapley attributions reveal biases in GEC evaluation metrics","Breaking down GEC scores: Shapley attribution per edit"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The consistency evaluation assumes that grouping same-sign edits into a single edit yields a Shapley value equal to the sum of the individual Shapley values, an equality that does not hold for Shapley values in general.","fun_headline_variants_meta":{"raw":{"variants":["Shapley values explain GEC scores edit by edit","Attributing GEC metric scores to individual edits via Shapley","Edit-level explanations for sentence-level GEC metrics","Shapley attributions reveal biases in GEC evaluation metrics","Breaking down GEC scores: Shapley attribution per edit"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000163,"raw_usage":{"total_tokens":1217,"prompt_tokens":892,"completion_tokens":325,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":508,"completion_tokens_details":{"reasoning_tokens":241}},"tokens_in":508,"tokens_out":325,"duration_ms":3519,"temperature":1.0,"reasoning_tokens":241,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:25:00.399668+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take GEC sentences with three or four edits, compute exact Shapley values for each edit and for the edit formed by merging two same-sign edits, and check whether the merged value equals the sum of the two individual values; a systematic gap across many sentences would falsify the claim that consistency across granularities supports reliability.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Shapley value, the cooperative-game-theory mechanism used to distribute the sentence-score difference across edits."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the sampling approximation that reduces attribution cost from exponential to roughly one second per sentence."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the JFLEG validation set, the fluency-oriented dataset used for the other evaluation."}],"review_version":1}