REVIEW 4 major objections 5 minor 18 references
LiveLongBench: Tackling Long-Context Understanding for Spoken Texts from Live Streams
T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read This paper introduces LiveLongBench, a bilingual benchmark of live-stream transcripts averaging 97K tokens, and uses it to claim that no current LLM or compression method reliably understands highly redundant spoken text.
desk verdict LiveLongBench fills a real gap with a useful spoken-text benchmark, but the undefined 'Score' metric makes the headline empirical results unauditable. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the benchmark itself plus its evaluation protocol. Transcripts come from Douyin e-commerce live streams, transcribed with Whisper and manually proofread, with only light filtering so filler words and repetitions survive. Tasks are categorized by information layout, including single-span retrieval, multi-span, semantic multi-span, and global spans, and grouped into retrieval-dependent, reasoning-dependent, and hybrid categories. Compression methods are grouped into token pruning, attention sparsification, and KV-cache quantization, and their combinations are ranked by Data Envelopment Analysis, which treats memory as input and score as output. The semantic multi-span task type is introduced as a novel extension, requiring models to integrate conceptually related but dispersed segments.
What would settle it
Take a sample of LiveLongBench transcripts, produce two versions, the original noisy transcript and a cleaned rewrite that removes filler words, repetitions, and topic drift while keeping all factual content, then run the same models on both under identical settings. If scores on the cleaned versions do not rise substantially, or if a single method starts to dominate both, the paper's central claims about redundancy and the absence of a best method would be undercut.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that long-context spoken text is a distinct failure regime: even a 1M-token model trails human annotators overall, and retrieval tasks are the hardest, not the long-reasoning ones. On the compression side, single methods show task-specific preferences; quantization preserves retrieval accuracy, while token pruning helps reasoning by removing noise, and no single method dominates. The paper's proposed hybrid baseline, MInference with LLMLingua 4x, reaches the best overall performance, and the three-way combination of KIVI 4-bit, MInference, and LLMLingua 2x gives the best performance-per-memory trade-off according to Data Envelopment Analysis. A further claim is that compression in high-redundancy contexts can improve accuracy, with LLMLingua 4x beating LLMLingua 2x.
Load-bearing premise
The load-bearing premise is that the poor results come from spoken redundancy specifically, yet the evaluation never compares against a cleaned or written version of the same transcripts, so length, task difficulty, or answer format could drive the gap instead.
Editorial extensions
If this is right
- If the benchmark is accepted as representative, current leaderboard results on written long-context benchmarks overstate readiness for deployed conversational AI, since retrieval from redundant speech is systematically worse.
- The finding that compression can improve accuracy implies that redundancy filtering is a legitimate inference-time technique, not just a cost-saving one, and that 4x pruning may be preferable to 2x in noisy input.
- Hybrid compression combinations should be treated as a design space, and the paper's DEA ranking provides a principled way to choose among them under memory constraints.
- Domain-specific fine-tuning helps hybrid tasks but hurts reasoning, so specialization is a trade-off rather than a free lunch.
- The benchmark can serve as a testbed for studying the 'lost in the middle' effect in spoken rather than written inputs.
Reading between the lines
- If redundancy is the causal driver, then a matched control, the same transcripts after filler-word removal or written-style rewriting, should reproduce the performance gap; the paper does not run that control, and such an experiment would separate redundancy from raw length or task difficulty.
- The benchmark's transcripts come from one platform and genre, e-commerce live streams, so the claim that it represents spoken texts generally is an extrapolation; applying the same pipeline to lectures, meetings, or news broadcasts would show how far the findings carry.
- The DEA efficiency ranking depends on the specific models, context windows, and memory measurements used; retraining the ranking on larger or newer models could shift which combination is optimal, while the qualitative finding that hybrids beat singles might survive.
- Because the dataset keeps ASR errors very low after proofreading, it could double as a controlled testbed for compression robustness; one could inject synthetic disfluencies at varying rates to map exactly how redundancy hurts retrieval.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces LiveLongBench, a bilingual (Chinese-English) long-context benchmark constructed from e-commerce live-stream transcripts with an average sequence length of roughly 97K tokens. It defines nine tasks grouped into three categories—retrieval-dependent, reasoning-dependent, and hybrid—and evaluates both closed and open LLMs as well as KV-cache compression methods (KIVI, MInference, LLMLingua) used individually and in combination. A DEA-based analysis is used to recommend a performance-memory trade-off. The central claims are that current LLMs, even with long context windows, perform substantially below humans on redundant spoken inputs; that no single compression method consistently outperforms the others; and that hybrid compression combinations such as MInference+LLMLingua-4x and KIVI+MInference+LLMLingua-2x provide the best balance of performance and memory use.
Significance. If the reported results are auditable, LiveLongBench fills a genuine and practical gap: long-context evaluation has focused on written documents, whereas live-stream transcripts with repetition, fillers, and topic drift are an important real-world domain. The paper's strengths include a detailed dataset-construction pipeline (Whisper transcription with manual proofreading, WER 0.53%, category and length statistics in Table 3), a documented six-step human-annotation quality-control protocol, an adapted Needle-in-a-Haystack stress test on spoken-style backgrounds, and a released code and benchmark repository. The hybrid-compression framing and the DEA analysis address a real deployment question. However, the significance is currently conditional: the primary evaluation metric is never defined, the attribution of model failures to redundancy lacks a matched written-text control, and the reliability of the human gold labels is not quantified.
major comments (4)
- [§4.1, Tables 2, 5, 6; Figures 3–4] The metric 'Score' is never defined. Section 4.1 states only that Score 'offers a softer and more fine-grained assessment by capturing partial correctness and enabling a continuous measure of model performance across tasks,' but no formula, rubric, per-task scoring procedure, or annotation protocol appears in the paper or appendix. Score is used for every Overall and Avg. column in Tables 2, 5, and 6, for the human-vs-model comparisons, and as the DEA output variable in Figure 4. The Exact Match columns cannot substitute: on reasoning tasks, human Exact Match is 4.8% and 8.3% while the corresponding Score values are 41.0 and 65.8, so the two metrics carry different content. Because Score is undefined, a reader cannot verify the core empirical claims that Gemini-1.5-pro outperforms other LLMs overall, that no single method consistently outperforms others, and that MInference+Lingua-4x or KIVI+MInference+Lingua-2x is optimal. Please provide a full definition, the scoring instructions, and the scoring code.
- [Abstract; §4.1] The conclusion that current methods 'perform poorly on highly redundant inputs' is not supported by the experimental design. LiveLongBench contains only spoken live-stream transcripts; there is no matched written-text corpus and no de-redundified or cleaned version of the same transcripts. The observed performance gaps could therefore be caused by task difficulty, answer format, input length, or domain-specific vocabulary rather than by redundancy specifically. A control condition—for example, running the same questions on a version of the same transcripts with repetition and fillers removed, or on written e-commerce text of matched length and topic—is needed to attribute the degradation to redundancy.
- [§1; §3.3; Table 3] The paper announces 'semantic multi-span' as a novel task type—an advanced form of multi-span reasoning over semantically distributed spans—but no task in Table 3 corresponds to it and Section 3.3 does not define how it is operationalized or scored. If semantic multi-span is a contribution, it needs an explicit task definition and dataset statistics; if it is intended to be covered by the 'Multiple Document QA' or 'Price Comparison' tasks, that mapping should be stated directly.
- [Appendix A.1] No inter-annotator agreement statistic is reported for the human gold labels, and no variance or repeated-run statistics are reported for model Scores. Because the human scores are the reference point for every model comparison in Tables 2, 5, and 6, a reliability measure (e.g., Cohen's kappa or per-task agreement) is needed to establish that the labels are stable enough to support the benchmark's conclusions. Without such a measure, it is difficult to know how much of the reported human advantage over models is due to annotation noise.
minor comments (5)
- [§3.2; §A.2; Table 3 caption] The benchmark name is inconsistently rendered as 'LiveLongBench,' 'LongLiveBench,' and 'LifelongBench'; please standardize it throughout.
- [Table 4] The table reports infinite audio durations ('∞') for Whisper and Paraformer-zh; please clarify what this means, presumably that these systems can process arbitrarily long segments without a fixed length limit.
- [Figure 5] The word cloud in Figure 5 is not rendered as readable text in the submitted manuscript; please replace it with a legible figure.
- [References] The reference list contains two entries for 'Leave no document behind' (Wang et al., 2024a and 2024b) with the same title; please disambiguate or merge them.
- [Appendix A.1] The annotation-cost calculation ('five full-time students over two days... total cost... around 400 RMB') is difficult to reconcile with a monthly salary of 800 RMB per student; please clarify the computation.
Circularity Check
No circularity: the benchmark is externally grounded by human gold labels and pre-existing methods; the undefined 'Score' metric is an auditability flaw, not a circular derivation.
full rationale
I find no circular step. The dataset is built from externally sourced Douyin live-stream transcripts, transcribed with Whisper and manually proofread, so the input texts are not constructed from any model output or from the metric being reported. Task definitions are motivated by prior independent work (Wang et al., 2024a; Kwan et al., 2023), not by the authors' own theorems or datasets with overlapping authorship. Gold labels and human scores come from a separate annotation process by students, providing external grounding that is independent of the evaluated LLMs and compression methods. Model and compression-method scores are measurements against those labels, not quantities re-injected into task construction. The DEA efficiency analysis is a post-hoc ranking whose input is measured memory usage and whose output is measured average Score; it does not define the benchmark or the gold answers. The proposed hybrid baseline combines previously published methods (KIVI, MInference, LLMLingua) and is selected from the evaluation table; it is not fitted to the data in a way that would force the reported conclusions. There are no load-bearing self-citations, no imported uniqueness theorem, and no ansatz smuggled in via citation. The one serious reporting defect is that 'Score' is described only as 'a complementary metric, Score, which offers a softer and more fine-grained assessment by capturing partial correctness' (Section 4.1) without a formula, rubric, or annotation protocol, and this undefined metric drives all overall rankings, Table 2, Table 5, and the DEA analysis. That is a reproducibility and auditability problem, but it is not a circularity problem: absent a definition, I cannot exhibit any reduction of a claimed result to its own inputs, which is the standard required for a circularity finding. Accordingly, the circularity score is 0.
Assumptions & free parameters
free parameters (1)
- repetition filter threshold =
sentences repeated more than 10 times removed
assumptions (3)
- domain assumption Whisper transcriptions, after manual proofreading, faithfully preserve the authentic spoken characteristics (filler words, repetition) of the live-stream audio.
- domain assumption Human-written gold answers are reliable references for partial-credit scoring.
- domain assumption LLaMA-3.1-8B-Instruct is representative of how compression methods interact with long spoken contexts.
invented entities (1)
-
semantic multi-span task type
Cite this review
Pith. "Pith review of LiveLongBench: Tackling Long-Context Understanding for Spoken Texts from Live Streams." pith.science (2026). https://pith.science/paper/HGFOCRWF
@misc{pith2026250417366,
author = {Pith},
title = {Pith review of: LiveLongBench: Tackling Long-Context Understanding for Spoken Texts from Live Streams},
year = {2026},
howpublished = {\url{https://pith.science/paper/HGFOCRWF}},
note = {Machine review of arXiv:2504.17366}
}
read the original abstract
Long-context understanding poses significant challenges in natural language processing, particularly for real-world dialogues characterized by speech-based elements, high redundancy, and uneven information density. Although large language models (LLMs) achieve impressive results on existing benchmarks, these datasets fail to reflect the complexities of such texts, limiting their applicability to practical scenarios. To bridge this gap, we construct the first spoken long-text dataset, derived from live streams, designed to reflect the redundancy-rich and conversational nature of real-world scenarios. We construct tasks in three categories: retrieval-dependent, reasoning-dependent, and hybrid. We then evaluate both popular LLMs and specialized methods to assess their ability to understand long-contexts in these tasks. Our results show that current methods exhibit strong task-specific preferences and perform poorly on highly redundant inputs, with no single method consistently outperforming others. We propose a new baseline that better handles redundancy in spoken text and achieves strong performance across tasks. Our findings highlight key limitations of current methods and suggest future directions for improving long-context understanding. Finally, our benchmark fills a gap in evaluating long-context spoken language understanding and provides a practical foundation for developing real-world e-commerce systems. The code and benchmark are available at https://github.com/Yarayx/livelongbench.
Figures
Figures from the paper (7 more)
Reference graph
Works this paper leans on
-
[2]
arXiv preprint arXiv:2308.14508
Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508. Jian Chen, Peilin Zhou, Yining Hua, Yingxin Loh, Kehui Chen, Ziyuan Li, Bing Zhu, and Junwei Liang
-
[7]
arXiv preprint arXiv:2407.02490
Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention. arXiv preprint arXiv:2407.02490. Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zirui Liu, Chia-Yuan Chang, Huiyuan Chen, and Xia Hu
-
[8]
arXiv preprint arXiv:2401.01325
Llm maybe longlm: Self-extend llm context window without tuning. arXiv preprint arXiv:2401.01325. Tomáš Koˇcisk`y, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Ed- ward Grefenstette
-
[9]
arXiv preprint arXiv:2310.19240
M4le: A multi-ability multi- range multi-task multi-domain long-context evalua- tion benchmark for large language models. arXiv preprint arXiv:2310.19240. Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu
-
[11]
arXiv preprint arXiv:2305.16300
Landmark attention: Random-access infinite con- text length for transformers. arXiv preprint arXiv:2305.16300. Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Vic- tor Rühle, Yuqing Yang, Chin-Yew Lin, et al
-
[12]
arXiv preprint arXiv:2403.12968
Llmlingua-2: Data distillation for efficient and faith- ful task-agnostic prompt compression. arXiv preprint arXiv:2403.12968. Bo Peng, Xinyi Ling, Ziru Chen, Huan Sun, and Xia Ning
-
[13]
arXiv preprint arXiv:2402.08831
ecellm: Generalizing large lan- guage models for e-commerce from large-scale, high-quality instruction data. arXiv preprint arXiv:2402.08831. Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Fi- rat, Julian Schrittwieser, et al
-
[14]
arXiv preprint arXiv:2403.05530
Gemini 1.5: Un- locking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530. Yale Song, Jordi Vallmitjana, Amanda Stent, and Alejan- dro Jaimes
Show all 18 references
-
[16]
CoRR, abs/2302.13971
Llama: Open and efficient foundation language models. CoRR, abs/2302.13971. Minzheng Wang, Longze Chen, Fu Cheng, Shengyi Liao, Xinghua Zhang, Bingli Wu, Haiyang Yu, Nan Xu, Lei Zhang, Run Luo, Yunshui Li, Min Yang, Fei Huang, and Yongbin Li. 2024a. Leave no document behind: B...
2024 arXiv
-
[17]
arXiv preprint arXiv:1801.07243
Personalizing dialogue agents: I have a dog, do you have pets too. arXiv preprint arXiv:1801.07243. Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Hao, Xu Han, Zhen Thai, Shuo Wang, Zhiyuan Liu, et al. 2024a.∞ bench: Ex- tending long context evaluation b...
-
[18]
The pass key is <PASS KEY>. Remember it. <PASS KEY> is the pass key
Results. Our results highlight the unique advan- tage of low-bit quantization in preserving retrieval performance, aligning with previous findings that retaining more information is critical for accurate retrieval. KIVI effectively reduces memory usage while maintaining retrie...
2023
-
[2013]
What to do about bad language on the internet. In Human Language Technologies: Conference of the North American Chapter of the Association of Computational Linguistics, Proceed- ings, June 9-14, 2013, Westin Peachtree Plaza Hotel, Atlanta, Georgia, USA, pages 359–369. The Asso...
2013
-
[2017]
arXiv preprint arXiv:1710.03957
Dailydialog: A manually labelled multi-turn dialogue dataset. arXiv preprint arXiv:1710.03957. Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paran- jape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024a. Lost in the middle: How language models use long contexts. Trans...
-
[2018]
arXiv preprint arXiv:1805.10190
Snips voice plat- form: an embedded spoken language understanding system for private-by-design voice interfaces. arXiv preprint arXiv:1805.10190. Jacob Eisenstein
-
[2019]
arXiv preprint arXiv:1906.01749
Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model. arXiv preprint arXiv:1906.01749. John J Godfrey, Edward C Holliman, and Jane Mc- Daniel
1906 arXiv
-
[2021]
In Findings of the Asso- ciation for Computational Linguistics: EMNLP 2021, pages 4543–4548
Do we know what we don’t know? studying unanswerable questions beyond squad 2.0. In Findings of the Asso- ciation for Computational Linguistics: EMNLP 2021, pages 4543–4548. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, B...
2021
-
[2023]
arXiv preprint arXiv:2307.11088
L-eval: Instituting standardized evaluation for long context language models. arXiv preprint arXiv:2307.11088. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al
-
[2024]
arXiv preprint arXiv:2405.09980
Fintextqa: A dataset for long- form financial question answering. arXiv preprint arXiv:2405.09980. Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Calta- girone, Thibaut Lavril, et al
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.