REVIEW 5 major objections 5 minor 1 cited by
Context-Aware Hierarchical Merging for Long Document Summarization
T0 review · 5 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash
Pith's one-line read Injecting source passages into each merge step improves faithfulness of over-100K-token summaries.
desk verdict A useful, honest empirical study of using source context to reduce hallucination in hierarchical long-document summarization, but the headline claim runs ahead of single-run metrics and a one-book human check. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the Incorporate Context (IC) module, a component inserted at each level of the hierarchy that returns a short set of source passages matched to each intermediate summary. At the first level, chunks are processed to obtain both an abstractive summary and relevant input contexts; at later levels, the module consumes concatenated summaries and passages from the previous level and repeats the selection, so every merge step sees evidence drawn from the original document. Three selectors choose the passages - extractive summarization, retrieval with the intermediate summary as query, and citation-frequency ranking over labeled passages - and two usage modes determine how they enter the next prompt: Support uses the context as proofreading evidence beside the summary, while Replace substitutes the context for the summary. Passages are length-matched to summaries so the two can serve as substitutes, and the design keeps all selected evidence traceable to the source.
What would settle it
Take the best system, Extract-Support, and swap its supporting passages for randomly selected source passages of equal length; if SUMMAC, AlignScore, and PRISMA scores do not fall, the benefit is prompt lengthening rather than evidence relevance. Alternatively, compute paired significance tests over repeated runs on both 100K-token corpora; if the Extract-Support versus plain hierarchical merging gap falls within noise, the central claim fails.
Extended reading notes
Core claim
The paper's central discovery is that hallucinations amplified by recursive merging can be reduced by feeding source passages into every merge step. It tests six context-augmented variants against zero-shot and two no-context hierarchical baselines on legal and book-length inputs over 100K tokens, using two model sizes. Across the board, a context-aware variant takes the top score on nearly every metric; Extract-Support, which uses extractively selected sentences as supporting evidence while keeping the abstractive summary as the backbone, is the overall best. A claim-level manual analysis of one book summary finds 72.7% of its atomic claims correct versus 59.1% for plain hierarchical merging. The authors also show that replacing summary text with source passages helps input-based faithfulness metrics but hurts reference-based coverage, so the mechanism that carries the gains is evidence-supported refinement, not evidence substitution.
Load-bearing premise
The load-bearing premise is that the automatic faithfulness metrics and the single-book manual check accurately reflect hallucination levels, so the observed gaps are real rather than metric noise.
Editorial extensions
If this is right
- Faithfulness of very-long-document summarization can be improved without fine-tuning or enlarging the context window, purely by changing what each merge step sees.
- Keeping the abstractive summary as the backbone and using source passages as proofreading evidence (Support) is the design that best balances coverage and factuality.
- Replacing intermediate summaries with source passages (Replace) boosts input-based faithfulness scores but hurts reference-based coverage, so it is best reserved for tasks that prioritize source fidelity over content selection.
- Citation-based context selection is the least reliable of the three selectors, since models struggle to obey citation instructions while summarizing.
- The gains hold across legal and narrative domains, though book-length narrative summaries remain harder than legal ones.
Reading between the lines
- The reported gaps are not accompanied by significance tests or repeated runs, so differences between close variants such as Extract-Support and Retrieve-Support should be treated as provisional.
- A control condition feeding random or irrelevant source passages of equal length would separate the effect of evidence relevance from the effect of simply giving the model more tokens.
- The same evidence-injection pattern could apply to other hierarchical aggregation tasks such as long-document question answering or multi-document synthesis, where merge steps are also hallucination bottlenecks.
- Making context selection adaptive, skipping augmentation when the intermediate summary is already faithful, would likely cut the runtime cost of Support, which is currently the slowest variant.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes augmenting hierarchical merging for long-document summarization (>100K tokens) with source-derived context. It introduces three context-selection methods (extractive summarization via fine-tuned MemSum, BM25 retrieval using intermediate summaries as queries, and citation-based selection) and two ways of using the selected context: replacing intermediate summaries (Replace) or using it as supporting evidence during merging (Support). Experiments on Multi-LexSum and a new SuperSummary book corpus with Llama 3.1 8B and 70B report that Extract-Support is the best variant on reference-based metrics (e.g., PRISMA) and that Replace variants show large gains on input-based faithfulness metrics (SummaC, AlignScore). A manual analysis on one book suggests Extract-Support has the highest proportion of supported claims. The paper releases code and data.
Significance. If the central claim is robust, the paper offers a practical, training-free enhancement to hierarchical merging that could reduce hallucination in very long document summarization, and it contributes a new long-narrative benchmark. The strength of the paper is its systematic comparison of six context-augmentation configurations with transparent prompts and appendices. However, the current evidence is not strong enough to support the sweeping 'consistently outperforms' claim: the flagship quantitative gaps are small, there are no error bars or significance tests, and the human validation is limited to one book with one annotator. The paper is a useful contribution if these gaps are addressed.
major comments (5)
- [§5, Tables 2 and 3] The central claim that Extract-Support outperforms HMerge and other baselines rests on single-run evaluations without variance or significance testing. For instance, the PRISMA advantage of Extract-Support over HMerge is 2.0 points on Multi-LexSum and 1.4 points on SuperSummary for 8B, and 1.5 and 3.4 points for 70B. These differences are within the noise of LLM-based evaluation and could easily flip with different seeds or metric instantiations. I request repeated runs or at least paired significance tests (e.g., bootstrap or Wilcoxon signed-rank over documents) for the key comparisons. This is load-bearing for the abstract's 'consistently outperforms' and for the practical claim that Extract-Support reduces hallucination.
- [Abstract and §5 (first paragraph)] The abstract states that contextual augmentation 'consistently outperforms zero-shot and hierarchical merging baselines,' but Tables 2 and 3 show several context-augmentation variants below HMerge on reference-based metrics. For example, on Multi-LexSum with 8B, Cite-Replace PRISMA is 36.3 vs. HMerge 45.5 and Cite-Support is 43.0; on SuperSummary with 8B, Extract-Replace is 22.3 vs. HMerge 37.8. The consistent-improvement claim holds only for a subset of variants and metrics. Please constrain the claim to what the data show (e.g., 'Extract-Support and Retrieve-Support improve reference-based faithfulness on these datasets') and revise the abstract accordingly.
- [§4 (Evaluation Metrics) and §5 (Replace models)] The large gains of Replace systems on SummaC and AlignScore are partly by construction: these metrics measure consistency with the source text, and Replace systems are prompted with source passages during generation. The manuscript acknowledges this in §5, but the abstract and parts of the conclusion do not carry the qualification. Because the load-bearing evidence for hallucination reduction must come from reference-based or human evaluation, the small PRISMA margins and limited manual check are not sufficient to support the strong formulation of the central claim. I recommend either adding a non-source-grounded control or explicitly limiting the Replace-effectiveness claim to input-based consistency.
- [§7 (Limitations) and Table 5] The manual analysis that anchors the faithfulness claim is based on one book, one annotator, and roughly 22–33 atomic claims per system, with no inter-annotator agreement reported. The headline gap, 72.7% vs. 59.1% correct claims for Extract-Support vs. HMerge, corresponds to a difference of about three claims. This is a weak anchor, as the paper's own limitations section acknowledges. I recommend expanding the human evaluation to at least a few books with two annotators and reporting counts and agreement, or clearly labeling the manual analysis as diagnostic rather than conclusive.
- [§5 (Adding more contexts yields better summaries for Replace) and Table 4] The context-length ablation in Table 4 is conducted only for Retrieve-Replace, and the gains (e.g., +4.6 PRISMA at 32K on SuperSummary) are again single-run measurements. Since this analysis is used to argue that 'adding more contexts yields better summaries for Replace,' and since it only covers one variant, the conclusion should be tempered. Showing similar trends for at least Extract-Replace or reporting variance would strengthen this point.
minor comments (5)
- [§4 (Model Comparisons)] The text 'yields six distinct model invita- tions' appears to be a typo for 'variants' or 'instantiations.'
- [Tables 2 and 3] The table headers contain typos: 'ROUGEL-L' should be 'ROUGE-L', and the spacing in 'S UMMA C' and 'AScore' is inconsistent.
- [Appendix B (Algorithm 1)] In the pseudocode, the variable `labelToPick` is used before assignment; the text appears to switch between `labelToSelect` and `labelToPick`. Also, 'converageSections' is a typo for 'coverageSections'. Please fix the pseudocode so the selection logic is unambiguous.
- [§3.3 (Cite)] For the Cite method at subsequent levels in the Support case, the prompt in Table 8 mentions 'supporting contexts' but does not explicitly show how the citation labels from the previous level are re-assigned; clarifying this would help reproducibility.
- [Table 10 (header)] The header 'FACT-P REC' contains an inconsistent space; it should be 'FACT-PREC'.
Circularity Check
Only partial by-construction component is the acknowledged Replace gain on input-based metrics; central Extract-Support claim is independently grounded.
-
other
[Section 5, paragraph 'Replace models show big improvements according to input-based metrics']
"Examining input-based metrics more closely, both SummaC and AlignScore show significant improvements when input passages are incorporated via Replace. ... These results align with our expectations, given that both metrics evaluate performance against input texts, and Replace explicitly leverages input context in summary generation."
Replace systems are given source passages as their generation input at merge time, while SummaC and AlignScore score the final summary by its consistency with the same source text. Feeding the scoring reference into the generator therefore makes high input-based scores partly entailed by the experimental design, rather than an independent measure of improved faithfulness; HMerge and Zero-shot must reconstruct source information from compressed summaries. The paper explicitly acknowledges this mechanism, so the step is disclosed rather than hidden.
full rationale
The core derivation chain for the headline recommendation (contextual augmentation, especially Extract-Support, improves faithfulness) is largely self-contained. It rests on reference-based metrics against gold summaries (PRISMA, ROUGE, BERTScore) and on an independent manual claim-level annotation over one book; these evaluations do not use the proposed system's own inputs as the scoring reference. The only near-circular element is the Replace-variant advantage on SummaC and AlignScore: those metrics measure consistency with the source document, and Replace explicitly feeds source passages into generation, so part of the gain is by construction. The paper states this expectation itself, and the final conclusion is not built on that gain; Extract-Support is recommended despite Replace's large input-based scores, because reference-based metrics and human annotation favor Support. Self-citations to Mahon and Lapata (2024) for PRISMA and to earlier Lapata co-authored work on extractive summarization are standard tool/method citations, not uniqueness theorems or load-bearing justifications. The remaining concerns, such as lack of significance tests and a single-book human evaluation, are robustness and correctness risks rather than circularity.
Assumptions & free parameters
free parameters (5)
- Chunk size for hierarchical merging =
8K tokens
- Top-k passages for Retrieve and Cite =
k chosen so concatenated passages match average summary length
- Maximum merging context length =
8K, 16K, 32K tokens
- Maximum number of extracted sentences for MemSum =
20
- Passage length for Retrieve and Cite =
100 tokens
assumptions (5)
- domain assumption Automatic metrics (SUMMAC, AlignScore, PRISMA) validly measure faithfulness and quality of >100K-token summaries.
- domain assumption Llama-3.1-8B and Llama-3.1-70B (with GPTQ-INT4 quantization) are representative of LLM behavior for hierarchical summarization.
- ad hoc to paper MemSum fine-tuned on 1,000 Llama-generated chunk summaries transfers to legal and narrative domains at all merge levels.
- domain assumption Reference summaries (Multi-LexSum long summaries and SuperSummary plot summaries) are accurate gold standards.
- ad hoc to paper One human-annotated book is sufficient to corroborate automatic faithfulness metrics.
Cite this review
Pith. "Pith review of Context-Aware Hierarchical Merging for Long Document Summarization." pith.science (2026). https://pith.science/paper/IEXJMYDT
@misc{pith2026250200977,
author = {Pith},
title = {Pith review of: Context-Aware Hierarchical Merging for Long Document Summarization},
year = {2026},
howpublished = {\url{https://pith.science/paper/IEXJMYDT}},
note = {Machine review of arXiv:2502.00977}
}
abstract
Hierarchical Merging is a technique commonly used to summarize very long texts ($>$100K tokens) by breaking down the input into smaller sections, summarizing those sections individually, and then merging or combining those summaries into a final coherent summary. Although it helps address the limitations of large language models (LLMs) with fixed input length constraints, the recursive merging process can amplify LLM hallucinations, increasing the risk of factual inaccuracies. In this paper, we seek to mitigate hallucinations by enriching hierarchical merging with context from the source document. Specifically, we propose different approaches to contextual augmentation ranging from \emph{replacing} intermediate summaries with relevant input context, to \emph{refining} them while using the context as supporting evidence, and \emph{aligning} them implicitly (via citations) to the input. Experimental results on datasets representing legal and narrative domains show that contextual augmentation consistently outperforms zero-shot and hierarchical merging baselines for the Llama 3.1 model family. Our analysis further reveals that refinement methods tend to perform best when paired with extractive summarization for identifying relevant input.
Figures
Forward citations
Cited by 1 Pith paper
-
Contextualized Early Detection of Online Firestorms: A Sequential LLM-Based Approach
An LLM judging sliding windows of Reddit threads can flag escalating firestorm threads early (recall 0.98 at a mean 8.56 comments) and classify complete threads retrospectively with 0.915 accuracy.
Reference graph
Works this paper leans on
-
[4]
In The Twelfth International Conference on Learning Representations
Booookscore: A systematic exploration of book-length summarization in the era of LLMs. In The Twelfth International Conference on Learning Representations. 9 Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. 2023. Extending context window of large language models via positional interpolation. Preprint, arXiv:2306.15595. Yukang Chen, Shengju ...
arXiv 2023
-
[6]
LongEval: Guidelines for human evaluation of faithfulness in long-form summarization. In Proceed- ings of the 17th Conference of the European Chap- ter of the Association for Computational Linguistics, pages 1650–1669, Dubrovnik, Croatia. Association for Computational Linguistics. Wojciech Kryscinski, Nazneen Rajani, Divyansh Agar- wal, Caiming Xiong, and...
arXiv 2022
-
[8]
Hybrid Long Document Summarization using C2F-FAR and ChatGPT: A Practical Study
Hybrid long document summarization us- ing c2f-far and chatgpt: A practical study. ArXiv, abs/2306.01169. Louis Mahon and Mirella Lapata. 2024. A modular ap- proach for multimodal summarization of TV shows. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8272–8291, Bangkok, Thailand...
work page Pith review arXiv 2024
-
[9]
Teaching language models to support answers with verified quotes. Preprint, arXiv:2203.11147. Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettle- moyer, and Hannaneh Hajishirzi. 2023. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Confe...
arXiv 2023
-
[11]
Hierarchical context merging: Better long context understanding for pre-trained llms. Preprint, arXiv:2404.10308. Google Gemini Team. 2024a. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. ArXiv, abs/2403.05530. Meta Llama 3 Team. 2024b. The llama 3 herd of models. Preprint, arXiv:2407.21783. Weizhi Wang, Li Dong, Hao ...
arXiv 2023
-
[12]
" means the statement is supported by the source book text
AlignScore: Evaluating factual consistency with a unified alignment function. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11328–11348, Toronto, Canada. Association for Computational Linguistics. 11 Shiyue Zhang, David Wan, and Mohit Bansal. 2023. Extractive is not faithful: An i...
work page 2023
-
[2018]
Ranking sentences for extractive summariza- tion with reinforcement learning. In Proceedings of the 2018 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pa- pers), pages 1747–1759, New Orleans, Louisiana. Association for Computational Linguistics. Mir Tafseer Nayeem a...
work page Pith review arXiv 2018
-
[2020]
Longformer: The long-document transformer. Preprint, arXiv:2004.05150. Amanda Bertsch, Uri Alon, Graham Neubig, and Matthew R Gormley. 2023. Unlimiformer: Long- range transformers with unlimited length input. arXiv preprint arXiv:2305.01625. Junyi Bian, Xiaodi Huang, Hong Zhou, and Shan- feng Zhu. 2023. Gosum: Extractive summariza- tion of long documents ...
arXiv 2004
Show all 12 references
-
[2022]
Yapei Chang, Kyle Lo, Tanya Goyal, and Mohit Iyyer
Attributed question answering: Evaluation and modeling for attributed large language models. Yapei Chang, Kyle Lo, Tanya Goyal, and Mohit Iyyer
-
[2023]
In Proceedings of the 2023 Con- ference on Empirical Methods in Natural Language Processing, pages 6465–6488, Singapore
Enabling large language models to generate text with citations. In Proceedings of the 2023 Con- ference on Empirical Methods in Natural Language Processing, pages 6465–6488, Singapore. Associa- tion for Computational Linguistics. Nianlong Gu, Elliott Ash, and Richard Hahnloser...
2023 arXiv
-
[2024]
Preprint, arXiv:2411.03012
Leveraging large language models in code question answering: Baselines and issues. Preprint, arXiv:2411.03012. Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2023. Self-rag: Learning to retrieve, generate, and critique through self-reflection. Preprin...
2023 arXiv
-
[9474]
Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, and Wenhu Chen
Curran Associates, Inc. Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, and Wenhu Chen. 2024. Long-context llms struggle with long in-context learning. Preprint, arXiv:2404.02060. 10 Chin-Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- t...
2024 arXiv
Reviewed August 9, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.