{"id":"5275d884-ba50-4a62-862a-ba223701a32f","arxiv_id":"2412.19906","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"SumAutoEval is an LLM-based entity-level summarization evaluator with four dimensions; its claimed human-correlation advantage is not consistently supported by the experiments.","lead":"This paper introduces SumAutoEval, a method that uses LLMs to score summaries on four fine-grained dimensions by breaking them into entities and checking each one against a reference. It claims better correlation with human judgments than ROUGE or G-Eval, but the results are mixed and based on small, filtered datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SummEval support for 'better human correlation' is compromised by a non-standard subset and post-hoc outlier removal; the headline empirical claim is not established.","rationale":"The reader's REJECT verdict is well supported. My stress-test pass identifies a slightly different weakest point than the one named in the reader's weakest_assumption. The reader focused on unvalidated LLM entity extraction as the load-bearing premise; I agree that extraction quality is important and unmeasured. However, the paper's own empirical evidence for the headline claim is more directly undermined by the evaluation protocol. The SummEval† subset is constructed by selecting, for each article, the machine summary with the most consistent expert scores; this is a non-standard dataset, and unless G-Eval is re-run on exactly this subset (which the paper does not report), the baseline numbers in Table 3 are not comparable. The subsequent outlier removal is also circular: the authors use SumAutoEval to identify human scores they believe are wrong, then remove those examples, and then report higher correlation. This procedure, without a pre-registered criterion or independent adjudication, can manufacture agreement. The paper's own appendix examples (A.1–A.4) show cases where the authors overrule expert ratings of 4 or 5, which would indeed increase correlation after removal. Because the consistency and coherence improvements over G-Eval are obtained only after these steps, the central claim rests on an insecure empirical foundation. The mixed results elsewhere (relevance 0.54 vs G-Eval 0.67; fluency 0.51 vs 0.59; readability 0.05 vs 0.27 in Table 2) further weaken the broad 'better human correlation' statement. My concern is therefore a concrete methodological flaw rather than a disagreement with consensus; it directly affects the validity of the headline result. Since the reader's verdict already reflects this weakness, I recommend no change to the verdict.","tokens_in":1826,"tokens_out":1443,"duration_ms":55790,"concrete_test":"Recompute Table 3 on the original, unmodified SummEval dataset, using all machine-generated summaries and the original expert scores, with no outlier removal. Run both SumAutoEval and G-Eval on the same inputs (or use G-Eval's published values for the full dataset) and report Spearman correlations per dimension. If SumAutoEval no longer beats G-Eval on consistency and coherence, the current head-to-head numbers are an artifact of the non-standard subset and the circular outlier filtering. If the authors want to retain outlier removal, they should pre-specify the exclusion rule and have an independent human panel adjudicate the disputed examples before any correlations are computed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claim—'better human correlation'—depends almost entirely on Table 3, but the comparison is built on a customized SummEval† subset and on post-hoc outlier removal. The authors 'chose a machine summary (from the pool of summaries) where expert evaluations were more consistent' for each source, which changes the test distribution and invalidates any comparison with published G-Eval correlations unless G-Eval is re-run on exactly the same subset; the paper does not state that it was. More seriously, §4.3.1 discards at least 10 human expert scores per dimension because they disagree with SumAutoEval, with the outliers chosen by the authors' own analysis, and the paper reports that removing them raises correlations by 20–23%. This is circular: the metric is used to veto the human labels that conflict with it, and only the remaining points are used to compute the headline correlations. The wins on consistency (0.70 vs 0.62) and coherence (0.75 vs 0.60) come from these filtered numbers. The reader's extraction concern is real but secondary: even with perfect extraction, the claimed advantage over G-Eval is not demonstrated once the evaluation protocol is corrected.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SumAutoEval, an LLM-based summarization evaluation method that decomposes summaries into atomic entities, then computes four objective scores (completeness, correctness, alignment, readability) from entity-level judgments obtained via multiple prompts. The method is evaluated on a 30-note medical summarization dataset and on a modified SummEval subset (SummEval†), with correlations against human expert scores compared to ROUGE, BARTScore, and G-Eval. The authors claim that SumAutoEval gives more interpretable and objective scores and empirically achieves better human correlation.","tokens_in":9507,"tokens_out":7194,"duration_ms":68034,"significance":"If validated, SumAutoEval would be a useful interpretable alternative to holistic LLM evaluators, particularly for medical note summarization where missing details and hallucinations matter. The design choices of entity-level scoring, multiple verification prompts, and explicit partial/contradict categories are reasonable. However, the current evidence does not support the headline claim. The SummEval comparison is compromised by a non-standard subset and post-hoc outlier exclusion, and the entity-extraction layer, which is the foundation of all scores, is never validated against human annotations. The paper's own medical results show lower readability correlation than G-Eval, so the claim of 'better human correlation' is overstated. The method may have promise, but it requires a properly controlled evaluation and independent validation of the entity extraction.","major_comments":[{"comment":"The SummEval† subset is constructed by selecting, for each source article, the machine summary whose expert scores were most consistent. This changes the test distribution, so published G-Eval scores computed on the full SummEval dataset are not directly comparable. The paper does not state that G-Eval was re-run on SummEval†, and therefore the reported wins in consistency (0.70 vs. 0.62) and coherence (0.75 vs. 0.60) do not establish an advantage over G-Eval.","section":"§4.3, Table 3"},{"comment":"The paper removes 'at least 10 instances' per dimension because the expert scores were deemed inaccurate after analysis that appears to be motivated by autoEval scores. The Appendix examples (A.1–A.4) show expert averages of 4 or 5 with autoEval scores of 1–2.6, and these are used as evidence of expert error. Removing human labels that disagree with the metric under evaluation is circular and inflates reported correlations; the paper states these exclusions improve correlations by 20–23%. Correlations without outlier removal must be reported, and outlier criteria must be pre-specified or independently adjudicated.","section":"§4.3.1"},{"comment":"All four scores are ratios over entity sets extracted by GPT-4-family models, but the extraction step is not validated against human entity annotations. No analysis measures how entity merging, splitting, or omission affects completeness and correctness. Because the same model family performs both extraction and entity-level classification, the high agreement in Table 1 may reflect model self-consistency rather than fidelity to the true information content. Human entity annotations or an independent extraction model are needed to support the claim that the scores are objective.","section":"§3.1, Eqs. (1)–(2)"},{"comment":"On the 30-note medical dataset, SumAutoEval's readability correlation (ρ=0.05) is far below G-Eval (0.27), and its organization correlation is only marginally higher (0.98 vs. 0.92). The abstract claims 'better human correlation' without qualification. The paper should either narrow the claim to the dimensions where the method wins or provide an explanation for the readability failure.","section":"§4.2, Table 2"}],"minor_comments":[{"comment":"In the Introduction, the text contains a garbled fragment 'hard tYesah o identify'; please fix this typo.","section":"Introduction"},{"comment":"Section 3.1 defines 'Entity' informally; please include a worked extraction example and a description of how the reference sourcing step handles unsupported entities.","section":"§3.1"},{"comment":"Table 2 labels a column 'G-Eval*' but the asterisk is never explained in the caption or text.","section":"Table 2"},{"comment":"The aggregation rules in Section 3.2.1 ('partial + partial + yes' -> yes; no agreement -> no) are ad hoc; please justify them or provide a sensitivity analysis.","section":"§3.2.1"},{"comment":"Some references contain encoding artifacts (e.g., 'Kry´sci´nski') and inconsistent capitalization (e.g., 'ROUGE' vs. 'Rouge'); please standardize.","section":"References"}],"recommendation":"reject","confidential_remarks":"The paper does not release code or data. The medical dataset is proprietary and small (30 notes), and the SummEval† subset is defined in a way that is not reproducible from the paper alone. Given the reliance on custom subsets and post-hoc exclusions, the lack of released artifacts is a serious reproducibility concern. The paper would need a substantially revised evaluation to be publishable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dan, quick take on arXiv:2412.19906. The core idea—decompose summarization quality into entity-level completeness, correctness, alignment, and readability, then score ratios over extracted entities—is sensible and potentially useful, especially for medical note summarization where interpretability matters. The three-prompt consensus design and the aggregation rules are reasonable engineering. And the authors actually ship a small 30-note medical dataset, which is something.\n\nBut the central claim, 'better human correlation,' is not supported by the evidence. The SummEval comparison in Table 3 is built on a non-standard subset (SummEval†) where the authors pick one machine summary per article that experts agreed on, and then filter both entities and articles. That alone breaks comparability with published G-Eval numbers. Worse, §4.3.1 discards roughly 10 human expert scores per dimension because they conflict with SumAutoEval, and the paper admits this raises correlations by 20–23%. That is circular: the metric is used to veto the very human labels it's supposed to track. The wins on consistency (0.70 vs 0.62) and coherence (0.75 vs 0.60) come entirely from those filtered numbers. The G-Eval correlations are not re-run on the same subset, so the comparison is apples-to-oranges.\n\nThe entity extraction itself is also unvalidated—there's no check against human entity annotations, and both the ground truth and target entities come from the same GPT-4 family. If extraction merges or drops concepts, the completeness and correctness ratios lose meaning. That's a secondary problem, but it compounds the main one. The medical dataset is tiny (30 notes), and readability correlation there is near zero (0.05 vs G-Eval's 0.27 in Table 2), which the paper honestly reports.\n\nSo what's genuinely useful here? The framework and the prompts are a start for entity-level evaluation, and the medical domain is a good testbed. But the validation is not there yet. A serious revision would need to re-run the comparison on the standard SummEval split, keep all human labels (or justify exclusion in a pre-registered way), and validate extraction against human entity annotations. As it stands, I'd desk reject. If you want a cautionary example for your reading group on how post-hoc filtering can manufacture a result, it's worth a look.\n\nRecommendation: don't send to review yet. If the authors fix the protocol, it could become worth a second look.","headline":"Entity-level evaluation framework is plausible, but the SummEval evidence is built on a filtered subset and post-hoc outlier removal, making the headline claim of better human correlation circular.","tokens_in":10026,"tokens_out":3520,"would_cite":false,"duration_ms":33017,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that summarization quality is best measured by scoring individual facts, and that the resulting entity-level scores agree with human experts more closely than n-gram, embedding, or holistic LLM baselines.","keywords":["summarization evaluation","LLM-as-judge","entity-level metrics","completeness","correctness","alignment","readability","medical note summarization"],"falsifier":"A decisive check is to recompute Table 3 on the full SummEval data with no outlier deletion: the paper states that roughly ten flagged examples shift correlations by 20–23%, so the consistency advantage (0.70 vs 0.62) surviving that recomputation would settle the claim.","tokens_in":9063,"feed_emoji":"📊","tokens_out":9751,"duration_ms":89238,"temperature":0.7,"pith_summary":"SumAutoEval aims to replace subjective 1–5 summary ratings with objective, interpretable scores computed at the level of individual facts. The paper's proposal is to break a ground-truth summary into atomic entities, check each entity against the candidate summary with several LLM prompts, and aggregate the checks into four ratios: completeness, correctness, alignment, and readability. The evidence is that on a 30-note medical dataset the completeness score agrees with expert labels at Cohen's kappa 0.81, and on a curated SummEval subset the method beats G-Eval on consistency and coherence after outlier removal. If the result holds, a summary score stops being an opaque number and becomes an audit trail of exactly which facts are missing, wrong, misplaced, or poorly written.","feed_headline":"Entity-level LLM scoring beats G-Eval on consistency and coherence","feed_subtitle":"SumAutoEval decomposes summaries into atomic facts, scores four quality dimensions, and agrees more closely with human experts.","key_machinery":"The load-bearing object is the entity, defined as a short phrase with exactly one key concept. The pipeline extracts entities from both the ground-truth and target notes in three steps — extraction, self-verification, and reference sourcing — and then runs three differently-phrased prompts per entity, with majority voting and partial credit (0.5 for 'partial', 1 for 'yes'). Per-entity answers are aggregated into the ratio scores, so every dimension reduces to counts over entities. This is what makes the evaluation auditable: a low completeness score can be traced to specific missing ground-truth entities.","core_discovery":"The paper claims that summarization quality is measurable as ratios over atomic entities. With $\\text{Completeness} = \\frac{\\text{total entity score in target}}{\\#\\text{ground-truth entities}}$ and $\\text{Correctness} = \\frac{\\text{total entity score in target}}{\\#\\text{target entities}}$, each dimension becomes a count that can be audited. Alignment is the share of target entities placed under the correct section, and readability is the share of sentences without writing issues. Using GPT-4o as the backend, the framework reaches Cohen's kappa 0.81 against expert entity labels for completeness on 30 medical notes, and on the curated SummEval subset it reaches Spearman 0.70 for consistency and 0.75 for coherence after removing erroneous expert scores, surpassing G-Eval's 0.62 and 0.60. The author's intended conclusion is that fine-grained entity-level evaluation gives a more understandable and more human-aligned assessment than holistic scores.","pith_inferences":["An unstated but natural product extension is an evidence-highlighting view: the reference-sourcing step already returns the supporting phrase from the source, so a clinician-facing tool could display why each entity was scored as present, missing, or contradicted.","Because the four scores are unit-free ratios over entity sets, the framework should transfer to other structured summarization domains, such as radiology or discharge summaries, wherever a reliable entity extraction step can be built.","The paper's own readability results—strong agreement on the SummEval subset with outlier removal, weak on whole-note medical flow—point to a testable extension: score readability at the passage or transition level rather than per sentence, and compare correlation with human judgments again."],"forward_implications":["Each score can be decomposed: a low completeness score names the specific ground-truth entities that the summary omitted.","Correctness and alignment scores identify the exact entities that are unsupported or placed in the wrong section, turning hallucination detection into a checklist.","Because the prompts ask for step-by-step reasoning and use majority voting, the scores are less sensitive to prompt tone than holistic LLM ratings, which the paper argues are easily manipulated.","In the medical setting, the method gives section-aware quality control, penalizing information placed under the wrong heading."],"supporting_citations":[{"why":"Defines ROUGE, the n-gram overlap baseline that SumAutoEval must outperform in Tables 1–3.","marker":"Lin, 2004"},{"why":"Supplies the SummEval dataset, its four dimensions, and the expert scores that become the SummEval† subset.","marker":"Fabbri et al., 2021"},{"why":"G-Eval is the central LLM-judge baseline; its holistic 1–5 ratings are the direct comparison in Tables 2 and 3.","marker":"Liu et al., 2023"},{"why":"BARTScore is a baseline that measures text-generation probability and is compared on completeness and correctness.","marker":"Yuan et al., 2021"},{"why":"Self-consistency prompting is the stated inspiration for the multi-prompt aggregation used in the entity classification tasks.","marker":"Wang et al., 2022"},{"why":"Establishes the four subjective summarization dimensions (coherence, consistency, fluency, relevance) that the paper recasts into objective metrics.","marker":"Kry´sci´nski et al., 2019"}],"fun_headline_variants":["Entity-level facts give summary scores that beat G-Eval","SumAutoEval scores four summary traits with entity counts","LLM atomic-fact evaluation outperforms G-Eval on human alignment","Measure summary completeness via entity ratios, not vague scores"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire method depends on the automatic fact-pulling step being accurate and complete: if the LLM merges, splits, or misses a key fact, every score changes even when the summaries themselves are the same.","fun_headline_variants_meta":{"raw":{"variants":["Entity-level facts give summary scores that beat G-Eval","SumAutoEval scores four summary traits with entity counts","LLM atomic-fact evaluation outperforms G-Eval on human alignment","Measure summary completeness via entity ratios, not vague scores"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000254,"raw_usage":{"total_tokens":1567,"prompt_tokens":946,"completion_tokens":621,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":562,"completion_tokens_details":{"reasoning_tokens":552}},"tokens_in":562,"tokens_out":621,"duration_ms":6288,"temperature":1.0,"reasoning_tokens":552,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:47:29.288674+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive check is to recompute Table 3 on the full SummEval data with no outlier deletion: the paper states that roughly ten flagged examples shift correlations by 20–23%, so the consistency advantage (0.70 vs 0.62) surviving that recomputation would settle the claim.","supporting_citations":[],"review_version":1}