{"id":"9ab458c9-2a6f-4727-a145-d2f75825716e","arxiv_id":"2412.12832","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"DSGram is a reference-free GEC evaluation metric that dynamically weights Semantic Coherence, Edit Level, and Fluency using LLM-generated AHP weights, and reports improved correlation with human judgments on the SEEDA benchmark.","lead":"The paper introduces DSGram, a metric for evaluating grammar correction systems that combines three sub-scores with weights that vary per sentence, generated by an LLM using the Analytic Hierarchy Process. A generalist might read it because it targets the problem that LLM-based grammar checkers often over-correct, which older metrics fail to detect.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's central claim that DSGram surpasses all baselines is contradicted by its own Table 3: SOME has higher sentence-level Kendall τ on both SEEDA-S (0.555 vs 0.551) and SEEDA-E (0.512 vs 0.499).","rationale":"The reader's strongest claim correctly identifies the Table 3 tension, and our concern is essentially that same contradiction made central. The reader's weakest_assumption (that LLM-generated AHP weights track human priorities) is a plausible mechanism-level risk, but it is not the most load-bearing issue: even if the weights are meaningful, the paper's own numbers still fail to establish the advertised 'surpasses all' claim because SOME has higher sentence-level Kendall τ on both SEEDA splits. I therefore do not need to invoke speculation about GPT-4's judgment matrices. The absence of significance tests compounds the problem: the few cells where DSGram is numerically ahead (e.g., system-level Spearman on SEEDA-S, sentence-level accuracy) could be within noise. The paper has genuine assets: it releases code and datasets, constructs a human-scored evaluation set, and tests several LLMs as scorers; those contributions can stand. But the headline superiority claim is not supported as written, and the conditional verdict should require either a significance-tested comparison or a rewritten claim that explicitly acknowledges SOME's superior or competitive sentence-level Kendall. This is why agreement_with_reader is partial rather than full: the reader's weakest assumption is a different (and, in my view, less decisive) concern than the direct internal contradiction in Table 3.","tokens_in":11244,"tokens_out":8163,"duration_ms":74274,"concrete_test":"Reproduce Table 3 from the released code and datasets (github.com/jxtse/GEC-Metrics-DSGram and the DSGram HF dataset), computing sentence-level Kendall τ for DSGram(G.w) and SOME on both SEEDA-S and SEEDA-E. Then run a paired bootstrap or permutation test over the sentence-level comparison pairs: resample the paired rankings with replacement, recompute τ_DSGram − τ_SOME, and report 95% confidence intervals. If the intervals for both datasets include 0, the claim must be weakened to 'comparable to SOME'; if the intervals favor SOME, the superiority claim is false; if they favor DSGram, the reader's conditional concern would be resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2 states that DSGram's correlation with human feedback 'surpasses that of all conventional reference-based metrics, as well as reference-free metrics like GLEU, Scribendi Score and DSGram weighted by 0.33.' Table 3 is the paper's own meta-evaluation, and it does not support this superiority claim. At the sentence level, DSGram (G.w) achieves Kendall τ = 0.551 on SEEDA-S and 0.499 on SEEDA-E, while SOME achieves 0.555 and 0.512 respectively. DSGram is therefore not the best reference-free metric on either sentence-level Kendall correlation, which is exactly the fine-grained human-alignment setting emphasized in the paper. On system-level Spearman, DSGram beats SOME on SEEDA-S (0.909 vs 0.867) but trails on SEEDA-E (0.944 vs 0.951). No significance tests or confidence intervals are reported, so even the comparisons where DSGram appears ahead cannot be distinguished from noise. The advertised 'surpasses all' conclusion is thus internally inconsistent with the reported numbers; the supported claim would be that DSGram is competitive with SOME and better only on some granularities or correlation types.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DSGram, a reference-free grammatical error correction (GEC) evaluation metric that combines three sub-metrics—Semantic Coherence, Edit Level, and Fluency—with per-sentence weights generated by an LLM through an Analytic Hierarchy Process (AHP) judgment matrix. The authors introduce two datasets, DSGram-Eval (human-scored) and DSGram-LLMs (GPT-4-labeled for fine-tuning), and validate the metric against human judgments on SEEDA, comparing with reference-based and reference-free baselines. The main claims are that DSGram's correlation with human feedback surpasses all conventional reference-based metrics and reference-free metrics such as GLEU, Scribendi Score, and uniformly weighted DSGram, and that the dynamic weighting mechanism outperforms equal weighting.","tokens_in":11505,"tokens_out":3878,"duration_ms":38916,"significance":"If the central claims were fully supported, DSGram would be a useful contribution: it addresses a real gap—evaluating LLM-based GEC systems that over-correct relative to gold references—and it introduces a plausible mechanism (AHP-driven dynamic weights) for adapting evaluation criteria to sentence context. The paper also contributes reusable resources: the DSGram-Eval human-annotated dataset, the DSGram-LLMs training set, and code/dataset releases, which enable reproducibility. The idea of using GPT-4 to generate per-sentence weights rather than fitting fixed parameters is a fresh direction, and the fine-tuning experiments for cost-effective LLaMA scorers are a practical addition. However, the empirical support is presently weaker than the claims, mainly because the headline 'surpasses all' statement is contradicted by the paper's own Table 3 and because the dynamic-weight advantage is reported without statistical inference.","major_comments":[{"comment":"The claim that DSGram's correlation with human feedback 'surpasses that of all conventional reference-based metrics, as well as reference-free metrics' is not supported by the numbers in Table 3. On sentence-level SEEDA-S, DSGram (G.w) has Kendall tau = 0.551 while SOME has 0.555; on SEEDA-E, DSGram has 0.499 while SOME has 0.512. On system-level SEEDA-E Spearman, DSGram (G.w) = 0.944 vs SOME = 0.951. The advertised conclusion should be narrowed to 'competitive with, and sometimes better than, existing metrics', or the authors should provide significance tests or confidence intervals that justify the 'surpasses all' phrasing for the particular configurations where DSGram is numerically ahead.","section":"Section 4.2, Table 3"},{"comment":"The dynamic-weighting advantage is the paper's central novelty, but the supporting evidence is reported as a single pair of Pearson correlations: 0.8764 for AHP-weighted human sub-scores versus 0.8544 for equal weighting, with the statement that the difference is 'significant at a relaxed significance level.' No test statistic, p-value, confidence interval, resampling procedure, or significance level is given. Without this information, the reader cannot verify that the 0.022 difference is real rather than sampling noise. Please report the exact test, the p-value, and ideally a bootstrap confidence interval for the difference.","section":"Section 4.4"},{"comment":"The consistency-check step is underspecified: line 8 says 'Adjust ratings in A until CR < θ' but does not state how the adjustment is performed—whether the LLM is re-prompted, whether a deterministic correction is applied, or whether the threshold is relaxed. This matters because the adjustment procedure could materially change the resulting weights and therefore the DSGram scores. In addition, because GPT-4 generates both the sub-scores and the judgment-matrix weights, the improved correlation with human overall scores could partly reflect self-consistency within a single model rather than alignment with human priorities. An ablation with scores from one LLM and weights from another would strengthen the causal interpretation.","section":"Section 3.2, Algorithm 1"},{"comment":"The human annotation protocol for DSGram-Eval is deferred entirely to an appendix, but the main-text validation in Section 4.3 and the dynamic-weight comparison in Section 4.4 depend on these human scores. The paper should report in the main text how many annotators were used, what instructions they received, how scores were aggregated, and what inter-annotator agreement was. Without this information, the human-alignment claims lack a reproducibility baseline.","section":"Section 4.3 and DSGram-Eval appendix"}],"minor_comments":[{"comment":"The text contains several typos, including 'arguement' for 'argument' and the malformed '486755' which appears to be a truncation of '0.486755'.","section":"Figure 5"},{"comment":"The reference list item beginning '1987' has no author name, and in Section 1 the citation is rendered as '(Ana 1987)'; this should be corrected to the full author and year.","section":"References"},{"comment":"The caption says the comparison indicates a favorable correlation, but the table includes a 'GEC Model Human Score' column whose provenance and normalization are unclear; please clarify how the human scores are averaged and ranked, and whether the fourteen models listed are exactly the set used for correlation computation in Table 3.","section":"Table 2"},{"comment":"The use of Cronbach's Alpha on weights generated for three text corpora is unusual; the paper should clarify the unit of analysis (individual sentences? datasets?) and whether alpha is computed on the sub-scores, the weights, or the weighted scores.","section":"Section 4.4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript has a strong central idea and a publicly available implementation, but the 'surpasses all' claim is refuted by the paper's own Table 3, and the dynamic-weight advantage lacks inferential statistics. These are fixable within the manuscript's scope by revising the claims and adding significance tests, so I recommend major revision rather than rejection. The authors should also be encouraged to make the DSGram-Eval annotation protocol available in the main text."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know: DSGram is a serious attempt at a reference-free GEC metric, with one genuinely new piece—an Edit Level sub-metric explicitly targeting over-correction—and the AHP+LLM dynamic weighting scheme. It ships code and datasets, and the fine-tuned LLaMA models are a sensible cost-reduction step. That's real value.\n\nWhat's new: it reorganizes SOME's three sub-metrics (Grammaticality and Fluency are highly correlated, so they merge them) and adds Edit Level, which is a real gap in existing metrics. The dynamic weighting via AHP pairwise comparisons from GPT-4 per sentence is a clever idea. The validation on SEEDA and a human-annotated DSGram-Eval shows competitive correlation.\n\nSoft spots: the headline claim in Section 4.2 says DSGram's correlation 'surpasses that of all conventional reference-based metrics, as well as reference-free metrics like GLEU, Scribendi Score and DSGram weighted by 0.33.' That's carefully worded to avoid claiming victory over SOME, but Table 3 includes SOME, and on sentence-level Kendall τ, SOME is higher on both SEEDA-S (0.555 vs 0.551) and SEEDA-E (0.512 vs 0.499). So the paper is competitive with, not superior to, SOME. The margins are small and no confidence intervals or significance tests are reported; the dynamic weight improvement over equal weights (0.876 vs 0.854 Pearson) is only significant at a 'relaxed significance level,' which isn't reassuring. The circularity worry is real but not fatal: GPT-4 generates both sub-scores and weights, but the validation target is human scores, so it's not fitting to the target. Still, the appendix with prompts and annotation details is not in the reviewed text, which makes the method harder to fully assess.\n\nBottom line: this is a worthwhile paper for anyone building or evaluating GEC metrics, especially for LLM-based systems that over-correct. It needs revision—tone down the superiority claim, add significance tests, and clarify the exact claim relative to SOME—but it deserves a serious referee.","headline":"DSGram is a serious, well-engineered GEC metric with a genuinely useful over-correction sub-metric and released code/data, but its claims against SOME are overstated and it lacks significance testing.","tokens_in":12012,"tokens_out":2813,"would_cite":true,"duration_ms":24759,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that GEC evaluation should use per-sentence dynamic weights over semantic coherence, edit level, and fluency, derived by an LLM through AHP, and that this outperforms fixed-weight and reference-based metrics on human…","keywords":["grammatical error correction","evaluation metric","dynamic weighting","Analytic Hierarchy Process","large language models","over-correction","human alignment","GEC evaluation"],"falsifier":"Compute DSGram on a held-out set of LLM over-corrections with per-sentence GPT-4 weights and with uniform 1/3 weights, comparing both against human overall scores at both system and sentence level; if the dynamic-weight version does not beat the uniform version in human correlation, the central claim fails.","tokens_in":11067,"feed_emoji":"⚖️","tokens_out":6396,"duration_ms":56855,"temperature":0.7,"pith_summary":"Evaluating grammatical error correction has become unreliable because LLM-based systems often produce fluent corrections that diverge from gold references. DSGram addresses this by scoring each corrected sentence on three qualities—semantic coherence, edit level, and fluency—and, crucially, weighing those qualities differently for every sentence. The weights come from an LLM that builds a pairwise comparison matrix, converted into normalized weights by the Analytic Hierarchy Process. The authors claim that on the SEEDA human-annotated benchmark, DSGram agrees with human rankings better than conventional reference-based metrics and better than equal-weight versions of itself, while the new Edit Level sub-metric specifically penalizes the over-correction that LLM-based GEC systems tend to produce. If true, automatic GEC evaluation can remain meaningful even when model outputs diverge from any single gold reference.","feed_headline":"Per-sentence weights beat fixed GEC evaluation metrics","feed_subtitle":"Weights for fluency, meaning, and edit necessity derived per sentence align better with human ratings.","key_machinery":"The central object is the per-sentence judgment matrix $A = (a_{ij})$, where $a_{ij}$ is the relative importance the LLM assigns to criterion $i$ over criterion $j$ on a 1–9 scale. The Analytic Hierarchy Process (AHP) converts this matrix into a normalized weight vector by computing the principal eigenvector, after discarding matrices whose consistency ratio exceeds 0.1. These dynamic weights are then applied to LLM-generated sub-metric scores, so the overall score is context-sensitive: a formal legal sentence receives heavier Edit Level and Semantic Coherence weights, while casual dialogue emphasizes Fluency. The mechanism thus turns a set of pairwise linguistic judgments into valid weights while making the evaluation adapt to each sentence's context.","core_discovery":"The central claim is that GEC quality is best modeled as a weighted sum of three sub-metric scores, $S = w_s s_s + w_e s_e + w_f s_f$, where the weights are not fixed constants but are recomputed per sentence from an LLM-built judgment matrix via the Analytic Hierarchy Process. An LLM rates the relative importance of Semantic Coherence, Edit Level, and Fluency for each sentence pair; after a consistency check requiring a consistency ratio below 0.1, the principal eigenvector of the matrix yields normalized weights. The paper reports a Pearson correlation of 0.876 between this dynamically weighted score and human overall scores, versus 0.854 under equal weighting, and system-level correlations on SEEDA that it states surpass all conventional reference-based and reference-free baselines. The accompanying novelty is Edit Level, a sub-metric designed to detect unnecessary or excessive edits, targeting the over-correction behavior observed in ChatGPT and other LLM-based GEC systems.","pith_inferences":["A testable consequence the paper leaves implicit is that the main benefit of DSGram may come from the Edit Level sub-metric rather than from dynamic weighting; ablating the weights to uniform values on a broader corpus would separate these contributions.","If LLM-generated weights truly mirror human priorities, then human GEC judgments are context-dependent, and fixed-weight evaluation benchmarks may systematically misrank systems on specialized text types such as legal or clinical language.","The same AHP-plus-LLM weighting procedure could transfer to other text-generation evaluation tasks, such as summarization or style transfer, where over-editing is also a concern."],"forward_implications":["DSGram should rank LLM-based GEC systems in closer agreement with human judges than metrics like BLEU, M2, ERRANT, GLEU, and equal-weight DSGram, particularly when systems over-correct.","The Edit Level sub-metric exposes over-correction that fluency-only or reference-based metrics miss, because a fluent but excessively rewritten sentence is explicitly penalized.","Fine-tuning small open-weight LLMs on the DSGram-LLMs dataset yields scores closer to human ratings than few-shot prompting, making the metric more affordable to deploy at scale.","The dynamic weighting approach could be incorporated into GEC training losses to steer models away from over-editing, as the paper suggests in its discussion of applications."],"supporting_citations":[{"why":"Supplies the SOME baseline and its three-sub-metric design, which DSGram reorganizes after finding Grammaticality and Fluency correlated at 0.89.","marker":"Yoshimura et al. 2020"},{"why":"Provides SEEDA, the human-annotated sentence- and edit-level benchmark used for all meta-evaluation comparisons.","marker":"Kobayashi et al. 2024"},{"why":"Documents ChatGPT's over-correction behavior, motivating the new Edit Level sub-metric.","marker":"Fang et al. 2023b"},{"why":"Establishes GPT-4's viability as a human-like GEC reviewer, the basis for using GPT-4 to score and annotate.","marker":"Sottana et al. 2023"},{"why":"Supplies the Analytic Hierarchy Process used to convert LLM pairwise comparisons into per-sentence weights.","marker":"Ana 1987"},{"why":"Introduces the reference-free three-sub-metric paradigm this paper extends and corrects.","marker":"Asano et al. 2017"}],"fun_headline_variants":["Dynamic weights for GEC evaluation beat fixed metrics","Per-sentence weighting improves GEC evaluation accuracy","Dynamic weighting for GEC metrics aligns with human ratings","AHP-based weights boost GEC evaluation correlation","LLM-driven dynamic weights for better GEC evaluation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument stands on the assumption that the pairwise importance judgments an LLM makes for a sentence reflect the relative weights human annotators actually use when scoring that sentence.","fun_headline_variants_meta":{"raw":{"variants":["Dynamic weights for GEC evaluation beat fixed metrics","Per-sentence weighting improves GEC evaluation accuracy","Dynamic weighting for GEC metrics aligns with human ratings","AHP-based weights boost GEC evaluation correlation","LLM-driven dynamic weights for better GEC evaluation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000454,"raw_usage":{"total_tokens":2249,"prompt_tokens":880,"completion_tokens":1369,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":496,"completion_tokens_details":{"reasoning_tokens":1296}},"tokens_in":496,"tokens_out":1369,"duration_ms":8735,"temperature":1.0,"reasoning_tokens":1296,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:40:19.884218+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute DSGram on a held-out set of LLM over-corrections with per-sentence GPT-4 weights and with uniform 1/3 weights, comparing both against human overall scores at both system and sentence level; if the dynamic-weight version does not beat the uniform version in human correlation, the central claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the SOME baseline and its three-sub-metric design, which DSGram reorganizes after finding Grammaticality and Fluency correlated at 0.89."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the reference-free three-sub-metric paradigm this paper extends and corrects."}],"review_version":1}