REVIEW 3 major objections 5 minor 1 cited by
Policies and Evaluation for Online Meeting Summarization
T0 review · 3 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash
Pith's one-line read Live meeting summarization is feasible, and the update policy—not the summarizer—controls the quality-latency trade-off.
desk verdict First systematic study of online meeting summarization; useful metrics but the headline comparisons are weakened by test-set selection and an overly idealized latency metric. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the policy: a control loop that at each step chooses READ (consume the next chunk of transcript), WRITE (emit a summary sentence), or ERASE (amend earlier output). Five policies instantiate this: length-based, model-based, sliding window, full rewriting, and fully incremental. Two purpose-built metrics carry the evaluation: Expected Latency (EL), computed as $\mathrm{EL} = \frac{1}{T}\sum_{t\in S\cup\{0\}}\frac{(N(t)-t)^2}{2}$, the expected wait from a uniform random time until the next write, and R1-AUC, the area under the Rouge-1 F1 curve as the meeting progresses. EL rests on the assumption that each emitted summary captures all content since the previous write; R1-AUC treats the user-visible summary at every moment as the system's output.
What would settle it
Take a meeting with known content-level timestamps. Build two online systems with identical write schedules—one that always writes all content since the last write, and one that deliberately defers half of it until the next write. If Expected Latency gives the two systems the same score while human annotators (or a content-alignment measure) judge the second system as noticeably later, the metric's core assumption is falsified.
Extended reading notes
Core claim
The paper's central claim is that online meeting summarization—maintaining a useful summary as the meeting happens—is feasible and that the choice of update policy dominates the quality-latency trade-off. On the AutoMin 2023 test set, the best online systems score within about one ROUGE-1 point of the offline GPT-4 baseline (42.7 vs 43.7), and in human evaluation the model-based and sliding-window policies match or exceed the offline Bart-based Zoom-long system. The authors further claim that their two new metrics—Expected Latency, which reduces latency measurement to the gap between a random time point and the next write event, and Rouge-curve AUC, which integrates intermediate summary quality over the meeting—are expressive enough to distinguish systems that final Rouge alone cannot. The strongest systems are the adaptive ones: they dynamically select how much transcript to read per summary unit instead of using fixed chunk lengths.
Load-bearing premise
The Expected Latency metric assumes each summary the system writes contains everything important that happened since the last write; if summaries omit or hold back content, two systems with identical write schedules get the same latency score even though one is much later in practice.
Editorial extensions
If this is right
- Meeting platforms can offer live catch-up summaries to late joiners, with a freshness-versus-quality knob set by the policy.
- Because policies wrap offline summarizers unchanged, any future improvement in the underlying model transfers directly to the online setting.
- The proposed metrics give the field a common yardstick—Expected Latency for delay, R1-AUC for intermediate quality—so systems can be compared on one trade-off curve.
- The adaptive policies' advantage over fixed chunking suggests that how the transcript is segmented is a primary quality lever, and the same segmentation could feed offline summarizers as well.
Reading between the lines
- Expected Latency could be sharpened by content-level alignment—say, embedding similarity between each summary sentence and the source turn that first supplies it—breaking ties between systems that write on the same schedule but delay different content.
- The READ/WRITE/ERASE policy loop is generic enough to extend beyond meetings to other live document-generation tasks, such as real-time news digests or action-item trackers, with EL and R1-AUC as portable metrics.
- Because AutoMin lacks audio timestamps, the authors model each dialog turn as one time unit; validating EL on a corpus with real wall-clock timing (or ASR word timestamps) would test whether turn-count latency matches human-perceived delay.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces the task of online meeting summarization, proposes several policies (length-based, model-based, sliding window, full rewriting, fully incremental) built on top of offline summarizers (Bart and GPT-4), and defines novel evaluation metrics: Expected Latency (EL), R1-AUC for intermediate summary quality, redundancy factor, and normalized erasure. The experiments on the AutoMin 2023 test set (12 meetings) with human evaluation lead to three claims: (1) online models can produce strong summaries, (2) the proposed metrics enable a detailed quality-latency analysis including intermediate outputs, and (3) adaptive policies outperform fixed scheduled ones.
Significance. If the claims hold, this is a useful first systematic study of online meeting summarization, establishing a task formulation, baseline policies, and evaluation metrics that the community can build on. The paper is commendable for publishing pseudocode, prompts, full per-system results, and a human evaluation that partially validates the automatic metrics. However, the central empirical claims are weakened by two load-bearing issues: the Expected Latency metric relies on an unvalidated coverage assumption that makes it a measure of write cadence rather than content-level delay, and the headline comparisons select the best chunk size per policy on the test set. The evidence for the third claim is also mixed, as only one of the two adaptive policies clearly outperforms the fixed-length baseline in the human evaluation.
major comments (3)
- [Section 6.2, Eq. (1)] The Expected Latency metric is derived under the explicit assumption that "the summarization output contains all relevant content from the source that appeared since the last output." Under that assumption, Eq. (1) depends only on the set of write timestamps S, not on what the summaries actually contain. Consequently, two systems with identical write schedules but vastly different content coverage receive the same EL, even though the paper defines latency as "the time between a piece of content appearing in the output and its appearance in the source." The paper then uses EL to compare systems in the quality-latency trade-off (e.g., sliding window Bart-256 EL 11.7 vs. length-based Bart-768 EL 25.9 in Table 4), so the latency axis can reflect "how often the system writes" rather than "how long until content appears." The assumption is stated but never validated or bounded, which undermines abstract claim (2). I recommend either validating the assumption on a sample via content alignment, or explicitly redefining EL as a cadence metric and providing a separate content-aware latency measure.
- [Table 1 and Section 7.1] The headline results in Table 1 select, for each policy and backend model, the chunk size with the best final ROUGE score on the test set. This is a form of test-set selection bias: each system is allowed to pick its most favorable hyperparameter after seeing the evaluation data, which inflates the reported quality and makes cross-policy comparisons (e.g., sliding window Bart-256 vs. length-based Bart-768 vs. model-based Bart-512) unfair. The full results in Table 4 partially mitigate this, but the main narrative and the human evaluation use only the best-per-policy rows. The paper should either report results for a fixed chunk size across all policies, or use a validation split for model selection and report the selected configurations.
- [Section 7.2, Table 3] Abstract claim (3), that "adaptive policies perform better than fixed scheduled ones," is not fully supported by the reported evidence. In the human evaluation of intermediate summaries (Table 3), the model-based policy receives an average score of 3.5, essentially tied with the length-based policy at 3.4, while only the sliding window policy (3.7) is clearly better. In the final-summary human evaluation (Table 2), the model-based system (4.4) is rated higher than the length-based system (3.6), but this rests on two annotators and no inter-annotator agreement or significance testing. Given the 12-meeting test set, the claim should be qualified as "some adaptive policies can outperform fixed ones based on automatic metrics and a limited human evaluation."
minor comments (5)
- [Section 4] The assumption of unit duration per dialog turn for latency calculation is stated in one sentence, but it has a direct effect on the absolute EL values and their interpretability; a brief discussion of how varying turn lengths (e.g., a long monologue vs. a one-word reply) would affect EL would be helpful.
- [Table 4] The LAAL column is reported for all systems but never used in the analysis or discussion; consider removing it or adding a note explaining why it is included.
- [Section 7.2] The description "The convex shape of the sliding window model's curve" is mathematically imprecise; a curve that rises quickly and then plateaus is typically concave or saturating, and the relationship between convexity and high AUC is not obvious. Please rephrase.
- [Author affiliation] There is a typo in the affiliation: "Karlsruhe Intstitute of Technology" should be "Karlsruhe Institute of Technology."
- [Section 7.1] There is a typo: "GPT-4 is still rated highest, but thesliding window Bart" should be "the sliding window Bart."
Circularity Check
No significant circularity: the Expected Latency metric rests on an acknowledged simplifying assumption rather than a circular reduction, and the central claims are grounded in external data and human evaluation.
full rationale
The paper's load-bearing claims are (1) online models can produce strong summaries, (2) the proposed metrics enable a quality-latency analysis that includes intermediate outputs, and (3) adaptive policies outperform fixed scheduled ones. None of these claims is equivalent to its inputs by construction. The Expected Latency metric in Section 6.2 is explicitly introduced as a proxy: 'That the summarization output contains all relevant content from the source that appeared since the last output. Given this simplification, calculating latency reduces to answering the question: “Sampling a random point in the meeting, how long do I have to wait for the next summary?”' This is an acknowledged modeling assumption that makes EL depend only on write timestamps, not a hidden derivation of content-level latency from the metric itself. The paper does not claim EL measures content-level delay under that assumption; it explicitly states the simplification. An untested or overly strong assumption is a correctness or validity risk, not circularity under the review rules. The R1-AUC metric for intermediate summaries is defined from ROUGE curves and then checked against human ratings, not fitted to the human scores; the human evaluation in Sections 7.1 and 7.2 provides an external sanity check on the ranking of policies. The policies themselves are parameter-free and applied post-training to an offline summarizer, so no fitted parameter is renamed as a prediction. The only self-citations are peripheral: the Zoom-long offline baseline and the deidentification-reversal preprocessing trick, both from Schneider and Turchi (2023). These are not load-bearing for the central adaptive-versus-fixed comparison, which is evaluated against the paper's own length-based systems and confirmed by human annotators. No uniqueness theorem is imported from the authors, and no known result is merely renamed. The derivation chain is therefore self-contained, with the caveat that EL's validity as a content-level latency measure depends on the stated coverage assumption, a concern about correctness rather than circularity.
Assumptions & free parameters
free parameters (1)
- Chunk size per policy (best selected for headline results) =
Length-based Bart 768; Model-based Bart 512; Sliding Window Bart 256; Full Rewriting GPT-4 512; Full Incremental GPT-4…
assumptions (5)
- domain assumption The AutoMin 2023 test set (12 meetings) is representative of general meetings and has high-quality references.
- domain assumption Expected Latency can be computed under the assumption that a summary contains all relevant content since the last write.
- domain assumption Each dialog turn is assigned unit duration for latency calculation.
- domain assumption ROUGE-1 F1 is the preferred automatic quality metric for this dataset.
- domain assumption Partial summaries at any time can be scored by ROUGE-1 F1 against the full final human reference.
Cite this review
Pith. "Pith review of Policies and Evaluation for Online Meeting Summarization." pith.science (2026). https://pith.science/paper/2QACEE4T
@misc{pith2026250203111,
author = {Pith},
title = {Pith review of: Policies and Evaluation for Online Meeting Summarization},
year = {2026},
howpublished = {\url{https://pith.science/paper/2QACEE4T}},
note = {Machine review of arXiv:2502.03111}
}
read the original abstract
With more and more meetings moving to a digital domain, meeting summarization has recently gained interest in both academic and commercial research. However, prior academic research focuses on meeting summarization as an offline task, performed after the meeting concludes. In this paper, we perform the first systematic study of online meeting summarization. For this purpose, we propose several policies for conducting online summarization. We discuss the unique challenges of this task compared to the offline setting and define novel metrics to evaluate latency and partial summary quality. The experiments on the AutoMin dataset show that 1) online models can produce strong summaries, 2) our metrics allow a detailed analysis of different systems' quality-latency trade-off, also taking into account intermediate outputs and 3) adaptive policies perform better than fixed scheduled ones. These findings provide a starting point for the wider research community to explore this important task.
Figures
Forward citations
Cited by 1 Pith paper
-
AutoMeet: a proof-of-concept study of genAI to automate meetings in automotive engineering
Engineers in an automotive R&D department estimated that a fully integrated genAI meeting-documentation pipeline could cancel about one in five meetings, saving roughly 10.5% of working time, and named privacy control...
Reference graph
Works this paper leans on
-
[1]
Evlampios Apostolidis, Eleni Adamantidou, Alexandros I Metsai, Vasileios Mezaris, and Ioannis Patras. 2021. Video summarization using deep neural networks: A survey. Proceedings of the IEEE, 109(11):1838--1863
work page 2021
-
[2]
Naveen Arivazhagan, Colin Cherry, Isabelle Te, Wolfgang Macherey, Pallavi Baljekar, and George Foster. 2020. Re-translation strategies for long form, simultaneous, spoken language translation. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7919--7923. IEEE
work page 2020
-
[3]
Abedelkadir Asi, Song Wang, Roy Eisenstadt, Dean Geckt, Yarin Kuper, Yi Mao, and Royi Ronen. 2022. An end-to-end dialogue summarization system for sales calls. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Track, pages 45--53
work page 2022
-
[4]
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gr...
2020
-
[5]
Alexander R Fabbri, Wojciech Kry \'s ci \'n ski, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021 a . Summeval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391--409
work page 2021
-
[6]
Alexander Richard Fabbri, Faiaz Rahman, Imad Rizvi, Borui Wang, Haoran Li, Yashar Mehdad, and Dragomir Radev. 2021 b . Convosumm: Conversation summarization benchmark and improved abstractive summarization with argument mining. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confe...
work page 2021
-
[7]
Nikhil Garg, Benoit Favre, Korbinian Reidhammer, and Dilek Hakkani-T \"u r. 2009. Clusterrank: A graph based method for meeting summarization. In Interspeech, Brighton (UK)
work page 2009
-
[8]
Tao Ge, Lei Cui, Baobao Chang, Sujian Li, Ming Zhou, and Zhifang Sui. 2016. News stream summarization using burst information networks. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 784--794
work page 2016
Show all 41 references
-
[9]
Tirthankar Ghosal, Ond r ej Bojar, Marie Hled \' kov \'a , Tom Kocmi, and Anna Nedoluzhko. 2023. https://aclanthology.org/2023.inlg-genchal.19 Overview of the second shared task on automatic minuting ( A uto M in) at INLG 2023 . In Proceedings of the 16th International Natural...
2023
-
[10]
Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. 2019. Samsum corpus: A human-annotated dialogue dataset for abstractive summarization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 70--79
2019
-
[11]
J Gu, G Neubig, K Cho, and VOK Li. 2017. Learning to translate in real-time with neural machine translation. In European Chapter of the Association for Computational Linguistics (EACL) Conference, 2017. The Association for Computational Linguistics
2017
-
[12]
Adam Janin, Don Baron, Jane Edwards, Dan Ellis, David Gelbart, Nelson Morgan, Barbara Peskin, Thilo Pfau, Elizabeth Shriberg, Andreas Stolcke, et al. 2003. The icsi meeting corpus. In 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceed...
2003
-
[13]
Wessel Kraaij, Thomas Hain, Mike Lincoln, and Wilfried Post. 2005. The ami meeting corpus. In Proc. International Conference on Methods and Techniques in Behavioral Research
2005
-
[14]
Shamit Lal, Shivam Duggal, and Indu Sreedevi. 2019. Online video summarization: Predicting future to better summarize present. In 2019 IEEE Winter Conference on applications of computer vision (WACV), pages 471--480. IEEE
2019
-
[15]
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58t...
2020
-
[16]
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74--81
2004
-
[17]
Zhengyuan Liu and Nancy F Chen. 2022. Dynamic sliding window modeling for abstractive meeting summarization. In INTERSPEECH, pages 5150--5154
2022
-
[18]
Gabriel Murray, Giuseppe Carenini, and Raymond Ng. 2010. Generating and validating abstracts of meeting conversations: a user study. In Proceedings of the 6th international natural language generation conference
2010
-
[19]
Anna Nedoluzhko, Muskaan Singh, Marie Hled \' kov \'a , Tirthankar Ghosal, and Ond r ej Bojar. 2022. https://aclanthology.org/2022.lrec-1.340 ELITR minuting corpus: A novel dataset for automatic minuting from multi-party meetings in E nglish and C zech . In Proceedings of the ...
2022
-
[20]
Jan Niehues, Ngoc-Quan Pham, Thanh-Le Ha, Matthias Sperber, and Alex Waibel. 2018. Low-latency neural speech translation. arXiv preprint arXiv:1808.00491
2018 arXiv
-
[21]
Andrei Olariu. 2014. Efficient online summarization of microblogging streams. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, volume 2: Short Papers, pages 236--240
2014
-
[22]
OpenAI. 2023. Gpt-4 technical report. https://cdn.openai.com/papers/gpt-4.pdf
2023
-
[23]
Sara Papi, Marco Gaido, Matteo Negri, and Marco Turchi. 2022 a . https://doi.org/10.18653/v1/2022.findings-emnlp.11 Does simultaneous speech translation need simultaneous models? In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 141--153, Abu Dhab...
2022 doi
-
[24]
Sara Papi, Marco Gaido, Matteo Negri, and Marco Turchi. 2022 b . Over-generation cannot be rewarded: Length-adaptive average lagging for simultaneous speech translation. In Proceedings of the Third Workshop on Automatic Simultaneous Translation, pages 12--17
2022
-
[25]
Minh Quang Pham, Sathish Reddy Indurthi, Shamil Chollampatt, and Marco Turchi. 2023. Select, prompt, filter: Distilling large language models for summarizing conversations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12257--12265
2023
-
[26]
Felix Schneider and Marco Turchi. 2023. https://aclanthology.org/2023.inlg-genchal.14 Team zoom @ A uto M in 2023: Utilizing topic segmentation and LLM data augmentation for long-form meeting summarization . In Proceedings of the 16th International Natural Language Generation ...
2023
-
[27]
Royal Sequiera, Luchen Tan, and Jimmy Lin. 2018. Overview of the trec 2018 real-time summarization track. In TREC
2018
-
[28]
Antoine Tixier, Polykarpos Meladianos, and Michalis Vazirgiannis. 2017. Combining graph degeneracy and submodularity for unsupervised extractive summarization. In Proceedings of the workshop on new frontiers in summarization, pages 48--58
2017
-
[29]
Gokhan Tur, Andreas Stolcke, Lynn Voss, John Dowding, Beno \^ t Favre, Raquel Fern \'a ndez, Matthew Frampton, Michael Frandsen, Clint Frederickson, Martin Graciarena, et al. 2008. The calo meeting speech recognition and understanding system. In 2008 IEEE Spoken Language Techn...
2008
-
[30]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30
2017
-
[31]
Alex Waibel, Michael Bett, Michael Finke, and Rainer Stiefelhagen. 1998. Meeting browser: Tracking and summarizing meetings. In Proceedings of the DARPA broadcast news workshop, pages 281--286
1998
-
[32]
Klaus Zechner. 2002. Automatic summarization of open-domain multiparty dialogues in diverse genres. Computational Linguistics, 28(4):447--485
2002
-
[33]
Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B Hashimoto. 2023. Benchmarking large language models for news summarization. arXiv preprint arXiv:2301.13848
2023 arXiv
-
[34]
Yusen Zhang, Ansong Ni, Ziming Mao, Chen Henry Wu, Chenguang Zhu, Budhaditya Deb, Ahmed Awadallah, Dragomir Radev, and Rui Zhang. 2022. Summn: A multi-stage summarization framework for long input dialogues and documents: A multi-stage summarization framework for long input dia...
2022
-
[35]
Bin Zhao and Eric P Xing. 2014. Quasi real-time summarization for consumer videos. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2513--2520
2014
-
[36]
Ming Zhong, Pengfei Liu, Yiran Chen, Danqing Wang, Xipeng Qiu, and Xuan-Jing Huang. 2020. Extractive summarization as text matching. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6197--6208
2020
-
[37]
Ming Zhong, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2022. Dialoglm: Pre-trained model for long dialogue understanding and summarization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 11765--11773
2022
-
[38]
Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, et al. 2021. Qmsum: A new benchmark for query-based multi-domain meeting summarization. In Proceedings of the 2021 Conference of the North American Chapte...
2021
-
[39]
Chenguang Zhu, Ruochen Xu, Michael Zeng, and Xuedong Huang. 2020. A hierarchical network for abstractive meeting summarization with cross-domain pretraining. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 194--203
2020
-
[40]
URL: " 'urlintro :=
ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...
-
[41]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 9, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.