REVIEW 5 major objections 5 minor 2 cited by
HERA: Improving Long Document Summarization using Large Language Models with Context Packaging and Reordering
T0 review · 5 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash
Pith's one-line read HERA improves long document summarization by grouping event-related paragraphs and reordering them into narrative order.
desk verdict HERA is a training-free packaging-and-reordering pipeline for long-document summarization that shows consistent gains across four LLMs; the core idea is useful, but the evaluation is thin and the reordering claim needs one more control. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the packaging-and-reordering pipeline. HERA first segments a document by paragraph and uses BRIO to produce a one-sentence summary of each paragraph as a cheap retrieval key. An LLM then ranks these keys against each event extracted from the paragraph summaries, keeps the top-k paragraphs in a segment bag, and the NAON sentence-ordering model reorders the bag by its summary sentences with the paragraphs following that order. Finally the LLM summarizes each reordered bag and aggregates the event summaries into the overall document summary. This lets the LLM read event-coherent, narratively ordered context instead of the full document.
What would settle it
Annotate gold event boundaries for a sample of arXiv and PubMed documents and run HERA twice, once with extracted events and once with gold events; if the summaries differ substantially, event extraction is the bottleneck. A simpler check is to annotate the retrieved top-k paragraphs and test whether every clause in the reference summary appears in some segment bag.
Extended reading notes
Core claim
The central claim, stated on the paper's own terms, is that context packaging and reordering together let an LLM produce more faithful and fluent long-document summaries than the LLM produces from the raw document. The reported numbers show, for example, that Gemini 1.5 with HERA gains 8.8% relative ROUGE-1 and 17.9% relative FactCC on arXiv, and that HERA combined with any of the four LLMs outperforms those LLMs alone on every metric. The ablation on LLaMA 3 shows that context packaging alone improves scores and that adding reordering improves them further, so both components carry part of the gain. The authors read this as evidence that LLMs' long-context failures are partly an input-organization problem, not only a context-length problem.
Load-bearing premise
HERA's gains depend on the event-extraction step: an LLM is asked to pull the most important events out of short paragraph summaries, and if that event list is incomplete or muddles distinct events, retrieval will fetch the wrong paragraphs and the final summary will drop or distort information.
Editorial extensions
If this is right
- Any current or future LLM can inherit the gain without retraining, since HERA only changes the input context.
- Context packaging alone accounts for part of the improvement by removing distracting content; reordering adds a further gain on every metric in the ablation.
- The bag-size results imply a practical tuning rule: keep enough paragraphs to cover the event but not so many that the LLM is again surrounded by noise, with top-5 best on both datasets.
- Faithfulness scores (FactCC and SummaC) rise alongside ROUGE and BERTScore, so the reordered input improves factual consistency rather than only lexical overlap.
- The added inference time is about 1.5 times the base LLM, which the authors treat as a small price for the quality gain.
Reading between the lines
- If event extraction is the true bottleneck, supplying gold event labels should improve HERA further; if it does not, the gain comes almost entirely from reordering rather than retrieval.
- The same packaging idea should transfer to other long-context tasks, such as multi-part question answering or multi-document synthesis, where evidence for one answer is scattered and order matters.
- Because HERA extracts events from BRIO's paragraph summaries, errors in those summaries propagate into retrieval; measuring event-extraction precision and recall directly would separate this failure mode from the reordering effect.
- The gains on scientific papers may not transfer to less structured narratives such as novels or news articles, where events are more interleaved and paragraph summaries are noisier.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes HERA, a training-free framework for long-document summarization that packages context by retrieving paragraphs relevant to extracted events and reordering those paragraphs into narrative order before the LLM generates a summary. The pipeline segments the document into paragraphs, obtains sentence-length local summaries (via BRIO) for each paragraph, uses an LLM to extract events and retrieve top-k paragraphs per event, orders the selected paragraphs with the NAON sentence-ordering model applied to the local summaries, and then generates and aggregates per-event summaries. Experiments on 500-document subsets of arXiv and PubMed with four LLMs (LLaMA 2, LLaMA 3, Gemini 1.5, GPT-4) report consistent gains over the base LLMs on ROUGE-1/2/L, BERTScore, FactCC, and SummaC. An ablation using LLaMA 3 separates packaging from packaging-plus-reordering, and a hyperparameter study varies the bag size k.
Significance. If validated, HERA would offer a practical, no-fine-tuning method to improve LLM faithfulness and informativeness on long documents, and its explicit packaging-and-reordering mechanism directly addresses the widely studied lost-in-the-middle and reading-order sensitivity of LLMs. The paper's strengths are its clarity, the breadth of LLMs tested (two open-weight and two API models), the consistent positive direction of gains across all models and both datasets, the transparent reporting of prompts and software versions, the explicit measurement of inference-time overhead, and the honest Limitations section. The main gap is experimental: the paper's central novelty, reordering, is not cleanly isolated from the extra inputs and inference structure of the reordering stage, and the bag-size hyperparameter appears to be selected on the test set. These issues are fixable with additional controlled experiments and should be addressed before the quantitative claims are accepted.
major comments (5)
- [§3.3, Table 3] The ablation does not isolate the reordering mechanism. The 'w packaging' condition removes the entire reordering step, including the paragraph-summary-sentence inputs that NAON sorts and the extra inference pass, relative to 'w both'. No condition feeds the same selected paragraphs in original, random, or reversed order while keeping all other steps identical. Therefore the gain between 'w packaging' and 'w both' could be driven by the additional input content (the BRIO-generated summary sentences) or by the changed pipeline structure, rather than by restoration of narrative order. This matters because the Introduction and Table 1 motivate the method specifically by 'messy narrative order' impairing LLM understanding. To support the reordering claim, the authors should add a control condition, e.g., the same selected paragraphs with randomly shuffled order, or with the summary-sentence inputs present but without reordering, and report those numbers.
- [§3.4 (Table 4) and §3.1 Implementation] The bag size k is selected on the test set, and the reporting is internally inconsistent. Section 3.1 states 'HERA use BRIO ... and selects the Top 6 paragraphs to form a segment bag,' while Table 4 shows k=5 as the best for arXiv on R-L/BS/FC and PubMed k=6 giving slightly higher R-L/BS/SC than k=5. The main results in Tables 2 and 3 do not state which k was used. If k was chosen after inspecting test-set scores, the reported gains are optimistic and the procedure does not reflect a deployment scenario. Please specify the k used in the main tables, justify the choice using the held-out validation splits, and report sensitivity of the main results across reasonable k values.
- [Introduction and §3.2] The claim of 'achieves state-of-the-art performance on both fluency and faithfulness metrics' is not supported by the baseline set. Table 2 compares only with FactorSum and Lodoss, both from 2022, and does not include recent long-context prompting strategies, hierarchical summarization pipelines, or other LLM-based summarization baselines. The phrase 'state-of-the-art' should either be replaced with a precise statement about gains over the selected baselines, or the baseline set should be expanded to include contemporary strong baselines on arXiv and PubMed.
- [§3.1 and Tables 2–4] No variance or significance information is reported. The test set is 500 randomly sampled documents per dataset, and no error bars, bootstrap intervals, or multiple runs are given. Table 4 shows differences of 0.1–0.2 points between adjacent values of k (e.g., PubMed R-L at k=5 vs k=6), and the text uses the word 'significantly' without a statistical test. The authors should report standard deviations or confidence intervals for the main results and key ablations so readers can assess the reliability of the reported margins.
- [Appendix A (Table 6) and Limitations] The event-extraction step is a load-bearing component of the pipeline, but its quality is never analyzed. The prompt in Table 6 asks the LLM to 'Extract the most important events from the following summary sentences,' and the retrieved paragraphs depend directly on the extracted event set. If events are missed, duplicated, or split across bags, the final summary can drop or distort information. The Limitations section acknowledges that the retrieval method is not powerful, but no diagnostic evidence is provided (e.g., oracle event extraction, retrieval recall against the gold summary, or human evaluation of event segmentation). At minimum, please report retrieval quality or an ablation with gold/oracle events to show how much of the gain depends on the event extraction prompt.
minor comments (5)
- [§3.2 and Table 2] ROUGE and BERTScore are described as 'fluency' metrics, but they measure lexical overlap and embedding similarity, not fluency. Please rephrase the metric grouping, e.g., 'informativeness (ROUGE, BERTScore)' and 'faithfulness (FactCC, SummaC)'.
- [§3.4] The sentence 'the trends of the results of the two datasets are not monotonous and similar' is confusing, and the subsequent claim that scores decrease for bag size greater than 5 is not true for PubMed, where k=6 gives the best R-L and BS. Please clarify the per-dataset trends.
- [§2 and §3.1] BRIO is described as a 'small summarization model,' but BRIO is built on BART-large; 'small' should be qualified (e.g., 'a compact summarization model relative to the long documents') to avoid implying a lightweight tool.
- [§3.5] The abstract and contributions say HERA does not require 'additional resources,' but Table 5 shows approximately 1.5x inference time and the pipeline also depends on BRIO and NAON. Please rephrase 'without additional resources' to 'without additional training or new model checkpoints' or state the computational overhead explicitly in the abstract.
- [§2, Context Reordering] The paper says HERA uses 'the current state-of-the-art sentence reordering model NAON,' but no comparison or citation evidence is given that NAON is state-of-the-art for paragraph ordering via summary-sentence representatives. Please either soften the claim or provide a benchmark citation for NAON's ordering quality.
Circularity Check
No circularity: HERA's headline gains are measured against external gold references and NLI-based metrics, and no pipeline component is defined in terms of the evaluation target.
full rationale
HERA is an empirical system paper without a formal derivation; its central claim that LLMs combined with HERA outperform the base LLMs is evaluated with ROUGE and BERTScore against external gold summaries and with FactCC and SummaC against the source documents, so the reported result is not defined in terms of the method's own outputs. The components (paragraph summarization by BRIO, reordering by NAON, and event extraction, retrieval, and aggregation by prompted LLMs) are all independent of the evaluation metrics; no step fits a parameter to the target metric and then renames that fit as a prediction. The only caveats are experimental rather than circular: Section 3.4 sweeps the bag size k on the same test subsets used for the main tables, so the reported configuration is test-set selected, and the Table 3 ablation removes the whole reordering stage rather than isolating the effect of paragraph order. These are validity concerns, not definitional reductions, and they do not make the headline comparison equivalent to its inputs. There are no load-bearing self-citations and no uniqueness or existence claim is imported from the authors' prior work.
Assumptions & free parameters
free parameters (1)
- bag size k (top-k paragraphs per event) =
5 or 6 (paper inconsistent)
assumptions (4)
- domain assumption BRIO one-sentence paragraph summaries preserve enough detail for retrieval and reordering
- domain assumption LLM event extraction from summary sentences yields the important events
- domain assumption Reordering paragraph summaries by NAON into narrative order improves LLM comprehension of the actual paragraphs
- domain assumption Automatic metrics (ROUGE, BERTScore, FactCC, SummaC) are valid proxies for fluency and faithfulness
Cite this review
Pith. "Pith review of HERA: Improving Long Document Summarization using Large Language Models with Context Packaging and Reordering." pith.science (2026). https://pith.science/paper/H3NTZPKR
@misc{pith2026250200448,
author = {Pith},
title = {Pith review of: HERA: Improving Long Document Summarization using Large Language Models with Context Packaging and Reordering},
year = {2026},
howpublished = {\url{https://pith.science/paper/H3NTZPKR}},
note = {Machine review of arXiv:2502.00448}
}
read the original abstract
Despite the rapid growth of context length of large language models (LLMs) , LLMs still perform poorly in long document summarization. An important reason for this is that relevant information about an event is scattered throughout long documents, and the messy narrative order impairs the accurate understanding and utilization of LLMs for long documents. To address these issues, we propose a novel summary generation framework, called HERA. Specifically, we first segment a long document by its semantic structure and retrieve text segments about the same event, and finally reorder them to form the input context. We evaluate our approach on two long document summarization datasets. The experimental results show that HERA outperforms foundation models in ROUGE, BERTScore and faithfulness metrics, while HERA does not require additional fine-tuning and resources.
Figures
Forward citations
Cited by 2 Pith papers
-
ResearchStudio-Reel: Automate the Last Mile of Research from Paper to Poster, Video, and Blog
A five-skill agent pipeline with one shared paper extractor and hard render gates produces editable posters, videos, and bilingual blogs, leading the Paper2Poster benchmark on aesthetics.
-
A Stepwise Questioning Expert-Editor Multi-Agent Framework for Long-Document Summarization
An expert-editor stepwise-questioning multi-agent pipeline improves ROUGE/BERTScore/FactCC for long scientific summarization on two datasets relative to direct generation and HERA.
Reference graph
Works this paper leans on
-
[1]
Yi Bin, Wenhao Shi, Bin Ji, Jipeng Zhang, Yujuan Ding, and Yang Yang. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.277 Non-autoregressive sentence ordering . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 4198--4214, Singapore. Association for Computational Linguistics
-
[2]
Sangwoo Cho, Kaiqiang Song, Xiaoyang Wang, Fei Liu, and Dong Yu. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.8 Toward unifying text segmentation and long document summarization . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 106--118, Abu Dhabi, United Arab Emirates. Association for Computational Li...
-
[3]
Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. https://doi.org/10.18653/v1/N18-2097 A discourse-aware attention model for abstractive summarization of long documents . In Proceedings of the 2018 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human...
-
[4]
Zican Dong, Tianyi Tang, Junyi Li, Wayne Xin Zhao, and Ji-Rong Wen. 2024. https://aclanthology.org/2024.lrec-main.188 BAMBOO : A comprehensive benchmark for evaluating long text modeling capacities of large language models . In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLI...
work page 2024
-
[5]
Context versus Prior Knowledge in Language Models
Kevin Du, Vésteinn Snæbjarnarson, Niklas Stoehr, Jennifer C. White, Aaron Schein, and Ryan Cotterell. 2024. https://arxiv.org/abs/2404.04633 Context versus prior knowledge in language models . Preprint, arXiv:2404.04633
work page Pith review arXiv 2024
-
[6]
Marcio Fonseca, Yftah Ziser, and Shay B. Cohen. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.426 Factorizing content and budget decisions in abstractive summarization of long documents . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6341--6364, Abu Dhabi, United Arab Emirates. Association for Computa...
-
[7]
Deepanway Ghosal, Navonil Majumder, Rada Mihalcea, and Soujanya Poria. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.683 ST a CK : Sentence ordering with temporal commonsense knowledge . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8676--8686, Online and Punta Cana, Dominican Republic. Association fo...
-
[8]
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. 2024. https://arxiv.org/abs/2404.06654 Ruler: What's the real context size of your long-context language models? Preprint, arXiv:2404.06654
arXiv 2024
Show all 28 references
-
[9]
Sainan Jia, Wei Song, Jiefu Gong, Shijin Wang, and Ting Liu. 2023. https://doi.org/10.18653/v1/2023.findings-acl.592 Sentence ordering with a coherence verifier . In Findings of the Association for Computational Linguistics: ACL 2023, pages 9301--9314, Toronto, Canada. Associa...
2023 doi
-
[10]
Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.750 Evaluating the factual consistency of abstractive text summarization . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Pro...
2020 doi
-
[11]
Sawan Kumar and Partha Talukdar. 2021. https://doi.org/10.18653/v1/2021.findings-acl.395 Reordering examples helps during priming-based few-shot learning . In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4507--4518, Online. Association for ...
2021 doi
-
[12]
Bennett, and Marti A
Philippe Laban, Tobias Schnabel, Paul N. Bennett, and Marti A. Hearst. 2022. https://doi.org/10.1162/tacl_a_00453 S umma C : Re-visiting NLI -based models for inconsistency detection in summarization . Transactions of the Association for Computational Linguistics, 10:163--177
2022 doi
-
[13]
Shaopeng Lai, Ante Wang, Fandong Meng, Jie Zhou, Yubin Ge, Jiali Zeng, Junfeng Yao, Degen Huang, and Jinsong Su. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.186 Improving graph-based sentence ordering with iteratively predicted pairwise orderings . In Proceedings of the ...
2021 doi
-
[14]
Junyi Li, Xiaoxue Cheng, Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.397 H alu E val: A large-scale hallucination evaluation benchmark for large language models . In Proceedings of the 2023 Conference on Empirical Methods in Natur...
2023 doi
-
[15]
Chin-Yew Lin. 2004. https://aclanthology.org/W04-1013 ROUGE : A package for automatic evaluation of summaries . In Text Summarization Branches Out, pages 74--81, Barcelona, Spain. Association for Computational Linguistics
2004
-
[16]
Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang
Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. https://doi.org/10.1162/tacl_a_00638 Lost in the Middle: How Language Models Use Long Contexts . Transactions of the Association for Computational Linguistics, 12...
2024 doi
-
[17]
Yixin Liu, Pengfei Liu, Dragomir Radev, and Graham Neubig. 2022. https://doi.org/10.18653/v1/2022.acl-long.207 BRIO : Bringing order to abstractive summarization . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers...
2022 doi
-
[18]
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. https://doi.org/10.18653/v1/2022.acl-long.556 Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity . In Proceedings of the 60th Annual Meeting of th...
2022 doi
-
[19]
Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.923 Is C hat GPT good at search? investigating large language models as re-ranking agents . In Proceedings of the 2023 ...
2023 doi
-
[20]
Derek Tam, Anisha Mascarenhas, Shiyue Zhang, Sarah Kwan, Mohit Bansal, and Colin Raffel. 2023. https://doi.org/10.18653/v1/2023.findings-acl.322 Evaluating the factual consistency of large language models through news summarization . In Findings of the Association for Computat...
2023 doi
-
[21]
Siye Wu, Jian Xie, Jiangjie Chen, Tinghui Zhu, Kai Zhang, and Yanghua Xiao. 2024. https://arxiv.org/abs/2404.03302 How easily do irrelevant inputs skew the responses of large language models? Preprint, arXiv:2404.03302
2024 arXiv
-
[22]
Zhiyong Wu, Yaoxiang Wang, Jiacheng Ye, and Lingpeng Kong. 2023. https://doi.org/10.18653/v1/2023.acl-long.79 Self-adaptive in-context learning: An information compression perspective for in-context example selection and ordering . In Proceedings of the 61st Annual Meeting of ...
2023 doi
-
[23]
Chong Zhang, Ya Guo, Yi Tu, Huan Chen, Jinyang Tang, Huijia Zhu, Qi Zhang, and Tao Gui. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.846 Reading order matters: Information extraction from visually-rich documents by token path prediction . In Proceedings of the 2023 Confer...
2023 doi
-
[24]
Weinberger, and Yoav Artzi
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. https://openreview.net/forum?id=SkeHuCVFDr Bertscore: Evaluating text generation with bert . In International Conference on Learning Representations
2020
-
[25]
Jun Zhao, Can Zu, Hao Xu, Yi Lu, Wei He, Yiwen Ding, Tao Gui, Qi Zhang, and Xuanjing Huang. 2024. https://arxiv.org/abs/2402.11550 Longagent: Scaling language models to 128k context through multi-agent collaboration . Preprint, arXiv:2402.11550
2024 arXiv
-
[26]
Yutao Zhu, Kun Zhou, Jian-Yun Nie, Shengchao Liu, and Zhicheng Dou. 2021. https://doi.org/10.1609/aaai.v35i16.17722 Neural sentence ordering based on constraint graphs . Proceedings of the AAAI Conference on Artificial Intelligence, 35(16):14656--14664
2021 doi
-
[27]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
-
[28]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 9, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.