{"id":"aae10dd4-b033-43e5-9893-94a0ff8be92a","arxiv_id":"2608.08180","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A linguistically refined and normalized index for quantifying relation-level hallucination in abstractive summaries, tested on four models across three datasets.","lead":"This paper proposes a refined metric, the Relation Hallucination Index (RHI), to measure made-up or distorted relationships in automatically generated summaries. It combines linguistic preprocessing with a normalized score intended for stable comparison across summarization models and datasets.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (13)'s RHI_min/RHI_max are never defined and Table IV and Table V report conflicting RHInorm values for identical entries, so the paper's central claim of stable, discriminative normalized scoring is unsupported.","rationale":"The paper's own Section V-D claims that 'normalization provides additional stabilization' but Section IV-D Eq. (13) leaves RHI_min and RHI_max undefined, and the two result tables contradict each other on the exact quantities used to support that claim. This is not a matter of disagreeing with an external benchmark or a question of community standards; it is an internal consistency failure that blocks the central claim of stable, discriminative measurements. Even if external human annotation were provided, the metric would still be unusable as reported because its normalized output is not reproducible from the equations and tables in the manuscript. The reader's weakest assumption identified the same load-bearing issue: the decomposition factors and Eq. (12) weights, together with the unspecified RHI_min and RHI_max, must correctly quantify relation-level hallucination. My check sharpens this by showing the two tables cannot both be correct under any consistent normalization. I also note the paper provides no code, data, or human evaluation, which further weakens the empirical support, but the arithmetic contradiction alone is sufficient to sustain the reader's reject verdict. The verdict should therefore remain unchanged.","tokens_in":8416,"tokens_out":3505,"duration_ms":30581,"concrete_test":"Reproduce Table V: take the six factors per model, compute RHI via Eq. (12), then apply Eq. (13) using the natural theoretical bounds from the factor definitions (minimum and maximum possible RHI when each factor lies in [0,1]) as RHI_min and RHI_max. Check whether the resulting RHInorm values match Table IV and/or Table V for all models. Additionally, check whether any single pair (RHI_min, RHI_max) can make Table IV's column and Table V's column simultaneously true; if not, the reported values cannot both be correct, independently of any external validation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that 'the grounded extraction process yields more stable and discriminative hallucination measurements' and that normalized RHI gives 'clearer separation and stable ranking across datasets' (abstract, Section V-C). This requires the quantity RHInorm in Eq. (13) to be well-defined and reproducible. Two properties fail. First, RHI_min and RHI_max are never defined, no derivation is given, and their values do not appear anywhere in the paper or appendices. Second, the reported RHInorm values are internally inconsistent: for the XSUM+XLSUM set, Table IV reports BART=0.7211, PEGASUS=0.6961, T5=0.7012, GPT-3.5=0.6830, while Table V reports BART=0.6912, PEGASUS=0.6553, T5=0.7055, GPT-3.5=0.6155 for the same models on the same data. T5's ordering also flips from third-highest in Table IV to highest in Table V. Reconstructing the unnormalized RHI from Table V's six factors with Eq. (12) gives, for BART, 1+(0.0249+0.0635)/2-(0.2616+0.0873+0.2418+0.6907)/4 = 0.724, which matches neither normalized entry. The tables are mutually incompatible under any single choice of RHI_min and RHI_max unless the extraction or the aggregation changed between tables without explanation. Since the claimed 'stable separation' is literally the difference between these numbers, the quantitative backbone of the paper is missing.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a refined framework for evaluating relation-level hallucination in abstractive summarization. It introduces a dependency-aware relation extraction pipeline with lemmatization, named-entity grounded subject resolution, passive agent recovery, negation handling, reporting-verb filtering, nominal fallback, clausal propagation, and deduplication. The extracted subject-verb-object triples feed a six-factor decomposition (EF, PH, OF, NH, LF, LH) that is aggregated into a Relation Hallucination Index (RHI) and then normalized via Eq. (13). The authors evaluate RHI on XSUM+XLSUM and a CNN/DailyMail SumEval subset using BART, PEGASUS, T5, GPT-3.5, and DistilBERT (also called DistilBART), and claim the grounded extraction yields more stable and discriminative hallucination measurements with consistent model rankings across datasets.","tokens_in":8744,"tokens_out":8421,"duration_ms":79273,"significance":"If the central claim were established, RHI would be a practically useful automatic metric: the linguistic refinements in Algorithms 1-4 are concrete, the six-factor decomposition is interpretable, and the goal of scale-invariant cross-dataset comparison is worthwhile. The paper also provides algorithmic pseudocode, which is a strength for reproducibility of the proposed extraction procedure. However, the manuscript currently lacks any external validation: the relation extraction pipeline is never evaluated against gold triples, RHI is never correlated with human judgments of hallucination, and the normalization constants in Eq. (13) are unspecified. More seriously, the reported RHInorm values in Tables IV and V are mutually inconsistent for the same models and dataset. As presented, the paper does not substantiate its headline claim of stable, discriminative hallucination measurement.","major_comments":[{"comment":"RHI_min and RHI_max are never defined; no values, derivations, or fitting procedures are given anywhere in the paper or appendices. Since every RHInorm value in Tables II, IV, and V depends on these constants, the normalized scores cannot be computed or reproduced, and the claim of 'scale-invariant comparison' is unsupported.","section":"Section IV-D4, Eq. (13)"},{"comment":"For the XSUM+XLSUM evaluation, the two tables report incompatible RHInorm values for identical model-dataset entries: BART is 0.7211 in Table IV but 0.6912 in Table V; PEGASUS is 0.6961 vs 0.6553; T5 is 0.7012 vs 0.7055; and GPT-3.5 is 0.6830 vs 0.6155. T5's rank changes from third in Table IV to first in Table V. Moreover, applying Eq. (12) to the Table V factor values for BART gives 1 + (0.0249 + 0.0635)/2 - (0.2616 + 0.0873 + 0.2418 + 0.6907)/4 ≈ 0.724, which matches neither reported normalized value under any single fixed pair of RHI_min and RHI_max. The quantitative backbone of the paper is therefore internally inconsistent.","section":"Section V, Tables IV and V"},{"comment":"The relation extraction pipeline is never evaluated for accuracy on the test data, and RHI is never validated against human annotations of relation-level hallucination. The central claim that RHI provides 'stable and discriminative' hallucination measurements rests solely on score separation among a handful of models; this does not establish that the scores track actual relational faithfulness. A human correlation study or comparison against established hallucination benchmarks is necessary.","section":"Section IV-C and Section V"},{"comment":"The statistical validation is reported only as 'paired t-test results show statistically significant performance differences (p < 0.01).' No test statistics, degrees of freedom, effect sizes, multiple-testing corrections, or unit-of-analysis details are provided. This is insufficient to support the assertions of 'clearer separation and stable ranking across datasets.'","section":"Section V-C"},{"comment":"The six decomposition factors and the equal weighting in Eq. (12) are introduced without justification or sensitivity analysis, and the framework assumes they are independent and exhaustive. Because these choices determine every RHI value, the behavior of the metric under alternate plausible weightings or factor definitions must be reported before the aggregation can be accepted as a faithful measure of hallucination.","section":"Section IV-D3-D4, Eqs. (6)-(12)"}],"minor_comments":[{"comment":"Table I lacks a RHInorm row, while Table II includes one; the reason for this asymmetry should be stated.","section":"Tables I and II"},{"comment":"The model is called DistilBERT in Table II but DistilBART in the text and Section V-E; the naming should be made consistent.","section":"Table II and Section V-E"},{"comment":"Eq. (2) defines EF1 while Eq. (6) defines EF as a decomposition factor; the similar notation is confusing and should be disambiguated.","section":"Eqs. (2) and (6)"},{"comment":"The condition 'token.pos̸=VERB' appears to be a rendering error and should be written as 'token.pos != VERB'.","section":"Algorithm 2, line 4"},{"comment":"The construction of the cumulative distributions is not described, and no statistical support is given for the claimed 'consistent ordering' of model curves.","section":"Section V-F, Figures 3 and 4"},{"comment":"The paper gives dataset subset sizes but no sampling protocol or code/data release, which limits reproducibility of the reported numbers.","section":"Section IV-A"},{"comment":"Coherence is defined as adjacent sentence embedding similarity; the connection of this fluency-oriented measure to relation-level hallucination is not motivated.","section":"Eq. (4)"}],"recommendation":"reject","confidential_remarks":"The manuscript is an incremental extension of the authors' own prior RHI work in [1], and the evaluative evidence is entirely internal: the improved metric is compared with its own predecessor and with generic lexical/semantic metrics, with no external human ground truth. The undefined normalization constants and the contradictory RHInorm values in Tables IV and V make the central quantitative result unreproducible. If the authors can supply a fully specified normalization, reconcile the tables, and add a human-judgment correlation study, a resubmission could be reconsidered; in its current form, the paper does not contain a verifiable central result."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The new part of this paper is the dependency-aware extraction pipeline: lemmatization, named-entity grounded subject resolution, passive recovery, negation handling, reporting-verb filtering, nominal fallback, and deduplication. That is a sensible set of linguistic refinements to relation triple extraction, and the decomposition into six factors (EF, PH, OF, NH, LF, LH) is interpretable. Compared to the authors' own earlier RHI, this is a real, though incremental, step.\n\nThe soft spots are where the evidence should be. Eq. (13) defines a normalized RHI using RHI_min and RHI_max, but those constants are never defined or derived anywhere. Without them, the normalized scores are unreproducible. Worse, the numbers do not match across tables. Table IV lists RHInorm for BART on XSUM+XLSUM as 0.7211, while Table V lists the same quantity as 0.6912. T5's ordering flips from third to first. These are not rounding differences; they are incompatible under any single choice of normalization constants unless the extraction or aggregation changed silently between tables.\n\nThere is also no human validation of the metric, no comparison to existing relation-level or factuality benchmarks, and no evaluation of the extraction pipeline's accuracy on the test data. The aggregation weights in Eq. (12) are asserted without justification. No code or data are released. The statistical section reports p < 0.01 without specifying the test details or effect sizes.\n\nSo the central claim—that the framework gives stable, discriminative hallucination measurements—is simply unsupported. The algorithmic description is coherent, but the quantitative backbone is missing. If the authors define the normalization, correct the inconsistencies, and validate against human judgments or existing benchmarks, the framework could become useful. As it stands, I would not accept it, but I would send it to review because the flaws are fixable and the topic matters. A good referee would catch the normalization problem immediately and could push the authors toward a much stronger paper.\n\nFor a reading group, it is a decent case study in how not to validate a metric. I would not cite it in its current form.","headline":"The extraction refinements are decent, but the metric's normalization is undefined and the reported numbers contradict each other, so the central claim of stable, discriminative scoring is not supported.","tokens_in":9255,"tokens_out":1897,"would_cite":false,"duration_ms":20296,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes a grounded, decomposed metric, the Relation Hallucination Index (RHI), for evaluating relation-level hallucination in abstractive summarization, and claims it yields stable, discriminative measurements across models and…","keywords":["relation-level hallucination","abstractive summarization","faithfulness evaluation","relation extraction","hallucination decomposition","normalized metric","dependency parsing"],"falsifier":"Have human annotators rate a sample of generated summaries for relation-level faithfulness, compute RHI on the same summaries, and check whether the human ranking of models matches the RHI ranking; if the correlation is weak or the ranking flips, the claim that RHI provides stable and discriminative measurements would be falsified.","tokens_in":8202,"feed_emoji":"📊","tokens_out":2981,"duration_ms":31469,"temperature":0.7,"pith_summary":"The paper is trying to establish that relation-level hallucination in abstractive summarization can be measured automatically and reliably by extracting structured subject-verb-object triples from documents, references, and generated summaries, then decomposing their mismatches into interpretable hallucination factors. The central claim is that linguistically grounded extraction, combined with a normalized aggregation formula, produces a metric that separates relation-preserving from hallucination-prone models more clearly than ROUGE, entailment-based scores, or entity-level metrics. If true, RHI would give developers a concrete, interpretable tool for diagnosing factual distortion in generated summaries without requiring human annotation.","feed_headline":"New metric catches distorted relations in summaries","feed_subtitle":"The normalized Relation Hallucination Index ranks summarizers by how faithfully they preserve entity-event relations.","key_machinery":"The key machinery is a dependency-aware relation extraction algorithm that produces normalized subject-verb-object triples through lemmatization, named-entity grounded subject resolution, passive-agent recovery, negation-aware verb modeling, reporting-verb filtering, nominal relation fallback, clausal propagation, and systematic deduplication. These mechanisms are designed to suppress spurious triples and stabilize the structural comparison that RHI relies on.","core_discovery":"The paper's discovery is an empirical evaluation framework built around the Relation Hallucination Index (RHI). RHI is computed from six decomposition factors that capture extractive faithfulness, positive hallucination, over-focus, negative hallucination, leakage from the reference, and leakage from the input, then aggregated by the formula RHI = 1 + (EF + PH)/2 - (OF + NH + LH + LF)/4, with a min-max normalized variant RHInorm for scale-invariant cross-dataset comparison. The authors report that this grounded and normalized formulation yields consistent model rankings across XSUM, XLSUM, and CNN/DailyMail SumEval, and that refinement of the relation extraction pipeline contributes the main improvement over their earlier RHI.","pith_inferences":["Implicit but not pursued in the paper: RHI's rankings on the SumEval subset could be checked against the human factual-consistency judgments already available in that benchmark; a positive correlation would strengthen the claim of discriminative validity.","Because the extraction pipeline itself is not separately evaluated for accuracy, the metric's behavior on unseen syntactic constructions is an open question; an error analysis of extracted triples would reveal whether score differences reflect text properties or parser failures.","The aggregation weights in Equation (12) are chosen rather than learned; a natural extension is to fit them to human judgments or to estimate them per domain, which could improve interpretability and transfer.","The same grounded-extraction and decomposition machinery could transfer to other structured generation tasks, such as data-to-text, where subject-verb-object triples are easier to verify against structured sources."],"forward_implications":["RHI can rank summarization models by relational faithfulness automatically, without human annotation, and the ranking is claimed to stay stable across abstractive and long-document datasets.","Because RHI separates relational correctness from lexical overlap, it can flag cases where ROUGE is high but the summary distorts entity-event relations.","The six-factor decomposition gives a diagnostic breakdown of how hallucination occurs, not just a single quality score, enabling targeted analysis of omission, distortion, or unsupported relation generation.","The normalized formulation allows comparing models across datasets that differ in summary length and abstraction level, which existing lexical and entailment metrics do not directly provide.","The framework identifies generation collapse, such as near-zero relational output, as a distinct failure mode that fluency-oriented measures would miss."],"supporting_citations":[{"why":"Supplies the ROUGE lexical-overlap baseline that the new metric is contrasted against.","marker":"[4]"},{"why":"Defines the Entity Hallucination Index, the entity-level metric that RHI extends and contrasts with.","marker":"[5]"},{"why":"Provides the sentence-embedding coherence measure used alongside RHI in the evaluation.","marker":"[20]"},{"why":"Provides the SummaC entailment-based consistency baseline used as a comparative metric.","marker":"[21]"},{"why":"Defines the Extractive-Faithfulness F1 (EF1) relation alignment measure that RHI builds upon and compares against.","marker":"[22]"}],"fun_headline_variants":["New metric decomposes relation hallucination in summaries","Grounded framework pinpoints relation-level summary errors","Normalized metric ranks summaries by relation faithfulness","RHI: a decomposed metric for summary relation fidelity"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the six decomposition factors and the aggregation weights in Equation (12), together with the normalization constants in Equation (13), correctly quantify relation-level hallucination, with no human annotation study validating the scores.","fun_headline_variants_meta":{"raw":{"variants":["New metric decomposes relation hallucination in summaries","Grounded framework pinpoints relation-level summary errors","Normalized metric ranks summaries by relation faithfulness","RHI: a decomposed metric for summary relation fidelity"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000158,"raw_usage":{"total_tokens":1212,"prompt_tokens":920,"completion_tokens":292,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":536,"completion_tokens_details":{"reasoning_tokens":233}},"tokens_in":536,"tokens_out":292,"duration_ms":3409,"temperature":1.0,"reasoning_tokens":233,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:17:51.718032+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have human annotators rate a sample of generated summaries for relation-level faithfulness, compute RHI on the same summaries, and check whether the human ranking of models matches the RHI ranking; if the correlation is weak or the ranking flips, the claim that RHI provides stable and discriminative measurements would be falsified.","supporting_citations":[{"cited_title":"ROUGE: A Package for Automatic Evaluation of Sum- maries,","cited_arxiv_id":null,"evidence_quote":"Supplies the ROUGE lexical-overlap baseline that the new metric is contrasted against."},{"cited_title":"Entity Hallu- cination Index in Abstractive Summarization—A Metric,","cited_arxiv_id":null,"evidence_quote":"Defines the Entity Hallucination Index, the entity-level metric that RHI extends and contrasts with."},{"cited_title":"Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks,","cited_arxiv_id":null,"evidence_quote":"Provides the sentence-embedding coherence measure used alongside RHI in the evaluation."},{"cited_title":"About Evaluation of F1 Score for RECENT Relation Extraction System","cited_arxiv_id":"2305.09410","evidence_quote":"Defines the Extractive-Faithfulness F1 (EF1) relation alignment measure that RHI builds upon and compares against."}],"review_version":1}