{"id":"87de90d6-a4ba-41bb-8931-53c3725c6a26","arxiv_id":"1908.10383","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A facet-aware evaluation metric (FAR) measures information coverage in extractive summaries by matching extracted sentences to human-annotated support groups for each reference sentence, showing modestly better human correlation than ROUGE.","lead":"This paper introduces a new way to evaluate extractive summarization by checking whether the extracted sentences cover the meaning of each reference sentence, instead of counting word overlaps. It also releases a small human-annotated dataset to support this evaluation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline claim that FAR correlates better with human judgment than ROUGE rests on a 0.017 Spearman gap from a three-system human ranking with no confidence interval or significance test; annotation quality is not the main threat.","rationale":"Good-faith reading: the paper builds a reusable annotated resource and a sensible metric; the low-abstraction restriction and the explicit AutoFAR-L caveat show appropriate caution in those parts. My concern is with the central comparative claim, not with the construction. The reader's formal weakest assumption was annotation reliability (pairwise Jaccard 0.714). I see that as secondary because the annotations were further verified to consensus, and because even perfect annotations would not make a 0.017 correlation difference from a three-system ranking decisive. The numbers in Table 4 are the only direct support for 'better correlation with human judgment.' Without uncertainty quantification or a larger system sample, the direction of the difference could easily be an artifact of which documents, annotators, or ROUGE variant were chosen. This is a correctness risk for the headline claim, though not for the dataset or the metric as a proposal. I therefore keep the reader's CONDITIONAL verdict: the contribution is worth publishing, but the central claim should be presented as suggestive unless the proposed significance check lands.","tokens_in":19773,"tokens_out":8455,"duration_ms":86997,"concrete_test":"Re-run the Table 4 analysis with a paired bootstrap over the human-evaluated documents: resample documents 10,000 times, recompute Spearman's rho for FAR and for the same ROUGE variant on each resample, and report the 95% bootstrap CI for the difference (FAR minus ROUGE). If the interval includes zero, the abstract's claim should be downgraded to an exploratory observation. A stronger settling check is to collect human rankings for all six extractive systems listed in Table 3 rather than three and repeat both the Spearman comparison and the bootstrap CI.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is the abstract's assertion that facet-aware evaluation 'manifests better correlation with human judgment than ROUGE.' The only direct evidence is in Sec. 3.2 / Table 4: Spearman's rho is 0.457 for FAR versus 0.44 for ROUGE, based on human rankings of only three extractive systems. The observed difference is 0.017, no confidence interval, bootstrap, or significance test is reported, and the text does not say which ROUGE variant enters the comparison. With three systems the rank space is extremely coarse; if the rho is computed over document-level pairs it can be swayed by a small number of documents or annotators, and if it is computed at the system level it summarizes three points. Consequently, even granting perfect FAMs, the reported numbers do not demonstrate superiority over ROUGE. The later caution about AutoFAR-L (Sec. 4.2) concerns a separate extrapolation and does not repair this missing support.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a facet-aware evaluation setup for extractive summarization. Each reference summary sentence is treated as a facet, human annotators mark document sentences that support each facet, and the Facet-Aware Recall (FAR) metric measures how many facets an extracted summary covers by comparing extracted sentence indices with these support groups. The authors construct a 150-sample extractive CNN/Daily Mail benchmark with Facet-Aware Mappings (FAMs), evaluate six extractive systems under FAR, ROUGE, and Support-Aware Recall, report a human ranking study claiming that FAR correlates better with human judgment than ROUGE, and investigate automatic FAM creation via sentence regression and a linear-regression extension (AutoFAR-L). The paper also provides fine-grained analysis by category (noise, low abstraction, high abstraction) and comparative analysis of extractive systems.","tokens_in":19976,"tokens_out":3569,"duration_ms":38537,"significance":"If the central claims hold, the paper makes a useful contribution: it releases a new annotation resource for extractive summarization, proposes a simple index-matching metric that is interpretable and cheap to compute once FAMs exist, and provides category-level analysis that could inform future dataset curation. The concrete strengths are the public release of the annotated data, the clear formalization of FAR/SAR, and the candid caveats about AutoFAR-L. However, the headline claim of better human correlation than ROUGE rests on a very small evaluation, and the annotation agreement is moderate; these issues affect the main scientific contribution, not just the presentation.","major_comments":[{"comment":"The abstract's claim that facet-aware evaluation 'manifests better correlation with human judgment than ROUGE' rests on a single Spearman's rho comparison of 0.457 versus 0.44. The observed difference is 0.017, no confidence interval, bootstrap, or significance test is reported, and the text does not state which ROUGE variant is used. Since only three extractive systems were ranked by human annotators, the rank space is extremely coarse and a difference of 0.017 cannot be distinguished from noise. This evidence is insufficient to support the headline claim in its current strong form. Please add uncertainty quantification and a clearly stated ROUGE variant, or rephrase the claim as suggestive/preliminary.","section":"§3.2, Table 4"},{"comment":"The validity of FAR depends on reliable and complete annotation of support sentences and support groups. The reported pairwise Jaccard index of 0.714 is moderate, and the main FAR comparison in §3.2 is computed only over the low-abstraction category, excluding noisy and high-abstraction facets. The paper does not analyze how annotation disagreement or the exclusion of these categories affects system-level FAR rankings. Please report agreement on support-group boundaries (not only overall sentence-level overlap), and include a sensitivity analysis of the system ranking under FAR on alternative annotation subsets or with relaxed support-group matching.","section":"§2.2, Table 2; §3.2"},{"comment":"The AutoFAR-L extension fits a linear regression to ground-truth FAR on the small annotated set and then applies it to predict FAR on the entire CNN/Daily Mail test set. The text appropriately cautions that the reliability of this extrapolation is not guaranteed; however, this caution does not repair the insufficient evidence for the primary human-correlation claim in §3.2. The large-scale system ranking under AutoFAR-L should be presented only as an unvalidated pilot, and the abstract's wording should not imply fully automatic validation of FAR without that caveat.","section":"§4.2, Table 8"}],"minor_comments":[{"comment":"Please specify which ROUGE measure (ROUGE-1, ROUGE-2, ROUGE-L, and which F1/recall/precision variant) was used for the correlation with human rankings.","section":"§3.2, Table 4"},{"comment":"The pairwise Jaccard index of 0.714 is described as 'high agreement'; please report the number of annotators per sample, the range of agreement values, and whether the final consensus labels were produced jointly or by majority vote.","section":"§2.2"},{"comment":"The fine-grained analysis in Table 5 uses ROUGE-1 F1 rather than FAR; the text should clarify that the category breakdown comes from annotated FAMs, while the metric used in the table is still ROUGE.","section":"§3.3, Table 5"},{"comment":"The parenthetical values in columns 'Pearson's r', 'Spearman's ρ', and 'Kendall's τ' (e.g., '97.6 (42.9)') are not clearly labeled; please separate the fit on the human-annotated subset from the extrapolated AutoFAR-L results.","section":"§4.2, Table 8"},{"comment":"The statement that FAM annotation was 'easier than previously believed' should be reconciled with the low recall of sentence regression approaches in Table 6; human effort and automatic support-sentence discovery are distinct and the contrast should be stated explicitly.","section":"§2.2, Table 6"}],"recommendation":"major_revision","confidential_remarks":"The paper has a valuable dataset and a coherent metric, but the central superiority claim over ROUGE is not statistically supported in the current version. The authors can likely fix this either by adding proper uncertainty quantification and a larger human evaluation, or by honestly downgrading the claim to a preliminary observation. The publication decision should hinge on whether that central claim is appropriately qualified and whether the annotation agreement analysis is strengthened."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this is a practical, reusable evaluation resource, but the headline claim about human correlation is not backed by the evidence. That doesn't mean reject the work; it means the paper's central assertion needs either a bigger human study or a more cautious claim.\n\nWhat's actually new: the facet-aware mapping (FAM) idea—treating each reference sentence as a facet and grouping document sentences into support groups—is a clean extension of Pyramid and HighRES. It's sentence-level, so it enables automatic evaluation of extractive systems by index matching. The annotated extractive CNN/Daily Mail subset (150 document-summary pairs, with category labels for noisy/low/high abstraction) is a genuine resource. Releasing the data is the most valuable part of this paper.\n\nWhat the paper does well: the fine-grained breakdown (noisy vs. low vs. high abstraction) gives real insight into the dataset, like the high proportion of noisy reference summaries. The evaluation of sentence regression approaches against human FAMs is a good idea and yields results worth having. The authors are also honest about limitations—they explicitly say the AutoFAR-L extrapolation isn't guaranteed and needs future work. That's more candid than most.\n\nThe soft spots are smaller than they look. The main issue is not annotation quality (pairwise Jaccard 0.714 is moderate, but not fatal). The problem is the paper's central empirical claim: FAR correlates better than ROUGE with human judgment. The evidence is a Spearman's rho of 0.457 vs 0.44 from human rankings of three systems, with no confidence interval, no significance test, and not even a specification of which ROUGE variant is used. With three systems, the ranking space is tiny and the difference is 0.017. That is not enough to 'demonstrate' better correlation. The paper's other analyses are fine; this one figure is the load-bearing wall, and it's made of straw.\n\nThe AutoFAR-L regression is mildly circular (fit to ground-truth FAR, using the same annotated data to validate), but the authors flag it as a pilot and leave rigorous study for future work, so it's a minor point. The assumption that each reference sentence is a single facet is acknowledged as a balance between cost and agreement; fine.\n\nWho should read this: summarization researchers who want an alternative to ROUGE, and anyone working with CNN/Daily Mail. The metric and dataset deserve engagement. But the paper should be sent to peer review with expectations of major revision—either strengthen the human evidence or soften the claim. My own verdict is skeptical on the superiority claim but positive on the resource.","headline":"Useful metric and dataset, but the human-correlation claim is a 0.017 Spearman gap without significance testing—worth reviewing, not worth accepting as-is.","tokens_in":20463,"tokens_out":2516,"would_cite":true,"duration_ms":23520,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Extractive summaries should be judged by semantic facet coverage, not token overlap; a new recall metric matches human ranking better than ROUGE.","keywords":["extractive summarization","evaluation metrics","facet-aware recall","ROUGE","support sentences","information coverage","CNN/Daily Mail","sentence regression"],"falsifier":"Collect human preference rankings on a new set of document-summary pairs, especially pairs where the reference summary shares few words with the document, and compute the Spearman correlation of both FAR and ROUGE with those rankings; if FAR does not beat ROUGE, or if re-annotating the support sentences with different annotators changes which system FAR ranks first, the paper's central claim would be refuted.","tokens_in":1920,"feed_emoji":"🧩","tokens_out":3398,"duration_ms":87665,"temperature":0.7,"pith_summary":"This paper tries to establish that information coverage in extractive summarization should be measured by whether the extracted sentences cover the semantic facets of the reference summary, not by token-level n-gram overlap. It defines each reference sentence as a facet, annotates which document sentences express each facet as support sentences grouped into support groups, and proposes Facet-Aware Recall (FAR), the fraction of facets for which at least one complete support group is covered by the extracted summary. To test this, the authors hand-annotate facet-aware mappings on a sample of CNN/Daily Mail and report that FAR correlates better with human rankings than ROUGE, supports fine-grained analysis by facet category, and can be approximated automatically via sentence regression. If these claims hold, summarization evaluation would move from rewarding superficial lexical similarity to rewarding genuine information coverage, and current benchmark results would need to be re-read in that light.","feed_headline":"Facet-aware metric beats ROUGE at mirroring human ranking","feed_subtitle":"A new recall measure checks whether extracted sentences cover the reference summary's facets, and ranks systems closer to human preference.","key_machinery":"The central object is the Facet-Aware Mapping (FAM): a relation from each reference sentence $r_i$, treated as one facet, to one or more support groups $S_i^j$, where each support group is a set of document sentences that together express that facet's semantics. The evaluation metric that runs on FAMs is Facet-Aware Recall (FAR), defined as the fraction of facets for which at least one complete support group is a subset of the extracted sentence set $E$. FAR counts a facet as covered exactly once, so it neither penalizes long extracted sentences that carry extra details nor rewards covering the same facet multiple times. A companion metric, Support-Aware Recall (SAR), merges all support sentences and measures the proportion of salient document sentences extracted, letting the framework separate salience from redundancy in comparative analysis.","core_discovery":"The central claim is that an extractive summary should be scored by whether it conveys the reference summary's information, and that this can be operationalized at the sentence-facet level. The paper treats each reference sentence $r_i$ as a facet, annotates one or more support groups $S_i^j$ for it, where each support group is a set of document sentences that together express the facet's semantics, and defines Facet-Aware Recall as $FAR = \\frac{1}{R}\\sum_{i=1}^R \\mathrm{Any}(I(S_i^1,E),\\ldots,I(S_i^N,E))$, with $E$ the set of extracted sentences and $I(X,Y)=1$ if $X \\subseteq Y$. The authors report that on their annotated extractive CNN/Daily Mail set, FAR ranks the three systems compared with human judges with Spearman $\\rho = 0.457$, slightly above ROUGE's $0.44$, and the system FAR ranks first also receives the most first-place human ranks. They also show that facet-aware evaluation can be broken down by facet category, exposing that both extractive and abstractive systems perform markedly worse on noisy and high-abstraction facets, and that machine-created support mappings built by sentence regression correlate strongly with human annotations for system ranking, making automatic large-scale facet-aware evaluation feasible.","pith_inferences":["A likely unstated consequence is that leaderboards built on ROUGE may have rewarded systems for high lexical overlap without verifying semantic coverage; switching to facet-aware metrics could change model selection and even training objectives, for example by training extractive models to maximize expected FAR rather than ROUGE.","On datasets where summaries are more abstractive, or where documents and summaries share fewer words, the correlation advantage of FAR over ROUGE should be larger than the observed $0.457$ vs $0.44$, because token overlap is even less informative there; annotating FAMs on such a dataset would test this.","The sentence-level facet definition likely undercounts sub-sentence semantic units; splitting facets below the sentence level, as Pyramid does with content units, might match human judgment even better at the cost of more annotation effort.","The reported pairwise Jaccard agreement of $0.714$ on support-sentence annotations suggests the ground truth is not perfectly stable; a sensitivity analysis that perturbs support sets and measures how stable FAR's system rankings are would quantify how much annotation noise the metric tolerates."],"forward_implications":["On CNN/Daily Mail, choosing FAR instead of ROUGE changes which extractive system is ranked best: UnifiedSum(E) leads under FAR, while ROUGE-1 F1 gives almost no discrimination among four compared methods.","Because FAR treats every facet equally and only checks whether a facet is covered, extractive systems are no longer penalized for outputting long sentences that add detail beyond the facet's core meaning.","Facet-category analysis provides a diagnostic: all tested extractive and abstractive systems score far worse on high-abstraction and noisy facets, suggesting that current strong performance on CNN/Daily Mail is largely driven by low-abstraction, copyable content.","Machine-created FAMs via sentence regression, especially when three support groups are allowed, correlate strongly with human-annotated FAMs in system ranking, so facet-aware evaluation can be scaled to the full test set without additional human annotation.","Because FAMs allow multiple alternative support groups for the same facet, the evaluation setup is ready to extend to multi-document summarization, where the same facet may be covered by redundant sentences across documents."],"supporting_citations":[{"why":"provides ROUGE, the token-overlap baseline that the paper argues against and compares FAR against.","marker":"(Lin, 2004)"},{"why":"supplies the CNN/Daily Mail documents and reference summaries on which the facet-aware mappings are annotated.","marker":"(Nallapati et al., 2016)"},{"why":"defines Pyramid content units, the closest prior content-based evaluation approach that FAR distinguishes itself from.","marker":"(Nenkova and Passonneau, 2004)"},{"why":"contributes the Refresh system and the human-evaluation sample used for ranking correlation and comparative analysis.","marker":"(Narayan et al., 2018c)"},{"why":"contributes FastRL(E), an extractive system re-evaluated under FAR and used in the correlation study.","marker":"(Chen and Bansal, 2018)"},{"why":"contributes UnifiedSum(E), the extractive system that FAR ranks first among the compared methods.","marker":"(Hsu et al., 2018)"},{"why":"contributes NeuSum, a compared extractive system whose ranking changes under facet-aware evaluation.","marker":"(Zhou et al., 2018)"},{"why":"contributes BanditSum, a compared extractive system used in the FAR and SAR analyses.","marker":"(Dong et al., 2018)"},{"why":"provides the sentence-regression scoring approaches the paper evaluates for automatic FAM creation.","marker":"(Zopf et al., 2018)"}],"fun_headline_variants":["Facet-aware recall beats ROUGE in human correlation","Summary evaluation: facets beat tokens for human alignment","New metric ranks summaries closer to human judgment","FAR: a better yardstick for extractive summary quality"],"cache_read_input_tokens":22784,"weakest_assumption_plain":"The argument rests on the assumption that each reference sentence is one self-contained facet and that the annotated support sentences fully capture that facet's semantics; if the human annotations are incomplete or inconsistent, the FAR scores and the claimed correlation with human judgment would not hold.","fun_headline_variants_meta":{"raw":{"variants":["Facet-aware recall beats ROUGE in human correlation","Summary evaluation: facets beat tokens for human alignment","New metric ranks summaries closer to human judgment","FAR: a better yardstick for extractive summary quality"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000645,"raw_usage":{"total_tokens":2985,"prompt_tokens":984,"completion_tokens":2001,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":600,"completion_tokens_details":{"reasoning_tokens":1938}},"tokens_in":600,"tokens_out":2001,"duration_ms":14944,"temperature":1.0,"reasoning_tokens":1938,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T10:44:19.293571+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect human preference rankings on a new set of document-summary pairs, especially pairs where the reference summary shares few words with the document, and compute the Spearman correlation of both FAR and ROUGE with those rankings; if FAR does not beat ROUGE, or if re-annotating the support sentences with different annotators changes which system FAR ranks first, the paper's central claim would be refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"defines Pyramid content units, the closest prior content-based evaluation approach that FAR distinguishes itself from."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"contributes UnifiedSum(E), the extractive system that FAR ranks first among the compared methods."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"contributes NeuSum, a compared extractive system whose ranking changes under facet-aware evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"provides the sentence-regression scoring approaches the paper evaluates for automatic FAM creation."}],"review_version":1}