Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

QwenLong-CPRS: Towards $\infty$-LLMs with Dynamic Context Optimization

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A 7B token-critic compressor lets short-context LLMs beat proprietary long-context models on Ruler-128K and InfiniteBench.

desk verdict A genuinely novel context-compression system with likely in-distribution NIAH results and headline numbers that don't add up; useful for retrieval-heavy tasks, but the SOTA claim needs a held-out test. read the letter →

arxiv 2505.18092 v2 pith:45UQKP2N submitted 2025-05-23 cs.CL

Weizhou Shen , Chenliang Li , Fanqi Wan , Shengyi Liao , Shaopeng Lai , Bo Zhang , Yingcheng Shi , Yuning Wu
show 7 more authors
Gang Fu Zhansheng Li Bin Yang Ji Zhang Fei Huang Jingren Zhou Ming Yan
This is my paper · ORCID
classification cs.CL
keywords contextcompressionlong-contextlanguagemodelsdynamicoptimizationtokencriticwindow-parallelinferencelostinthemiddleretrieval-augmentedgenerationsparseattention
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

QwenLong-CPRS is a context-compression model that takes a long document, a user query, and a natural-language instruction about granularity, and returns a short list of query-relevant tokens, sentences, or paragraphs. The paper's central claim is that feeding this compressed context to any generative LLM is both more accurate and much cheaper than feeding the raw long context, because relevance can be scored in independent 8,192-token windows. If the claim holds, small open-source LLMs with modest context windows can match or beat far larger proprietary models on 128K-token and million-token benchmarks, turning context length from a hardware limit into a compression problem. The paper reports that cascading with Qwen2.5-32B-Instruct beats proprietary baselines by 4.85 points on Ruler-128K and 10.88 points on InfiniteBench, with 21.59x compression and a 19.15-point average improvement across five flagship LLMs.

What carries the argument

The load-bearing mechanism is the token critic: the base language model's own vocabulary head is repurposed to score each token with a semantic label, while a secondary head assigns positional boundary tags, so the joint decision space is vocabulary labels cross position tags. Upper Transformer layers use bidirectional attention to see forward and backward context, while lower layers keep causal masking to preserve pretrained language knowledge. Window-parallel inference (Eq. 4) splits the long context into independent 8,192-token windows, reducing the prefill complexity from $O(|X_l|^2)$ to $O(w/\rho \, |X_l|) + O(|X_s|^2)$, where $w$ is the window size, $\rho$ the parallelism factor, and $X_s$ the optimized context. Training uses a 126K-sample corpus built from multi-granularity extraction and query-aware synthesis.

What would settle it

Build a 200K-token document where answering a question requires combining a fact from window 1 with a fact from window 25, and confirm that a full-context model with a sufficiently large window answers correctly. If QwenLong-CPRS's compressed input fails on the same question while the full-context model succeeds, the window-parallel locality assumption is overturned.

Watch

Extended reading notes

Core claim

The core discovery is that token-level importance labeling, trained through a language-modeling-head token critic, transfers across architectures: one 7B compressor, initialized from Qwen2-7B, can optimize contexts for GPT-4o, Gemini 2.0 Pro, Claude 3.7 Sonnet, DeepSeek-V3, and Qwen2.5-max. The compressor retains only query-relevant spans, at word, sentence, or paragraph granularity, and because it reads the document in parallel 8,192-token windows, prefill cost grows linearly rather than quadratically. Empirically the paper finds that smaller short-context LLMs augmented by QwenLong-CPRS outperform larger long-context counterparts, and that the framework beats RAG and sparse-attention baselines in both accuracy and tokens consumed.

Load-bearing premise

A text span's usefulness for answering a query must be recognisable from that span alone, within one 8,192-token window, with no information from other windows.

Editorial extensions

If this is right

  • Cascaded with Qwen2.5-32B-Instruct, QwenLong-CPRS reports 92.67 on Ruler-128K and 73.81 on InfiniteBench, beating the best proprietary baselines by 4.85 and 10.88 points respectively.
  • The compressor is architecture-agnostic: all five tested flagship LLMs gain an average of 19.15 points while their input tokens fall by 21.59x.
  • Context length stops being a hard hardware limit: window-parallel inference makes the compressor's prefill cost linear in the document length, so documents beyond 1M tokens can be optimized.
  • Smaller short-context LLMs augmented by QwenLong-CPRS outperform larger long-context ones, e.g., Qwen2.5-7b-instruct with QwenLong-CPRS beats Qwen2.5-72b-instruct direct prompting on Ruler-128K and InfiniteBench.
  • On LongBench V2 the gains are smaller, about +3.0 points on average and concentrated in the longest contexts, indicating the method helps most where models already struggle.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The window-parallel design implies a locality bet: a token's relevance to the query must be decidable from its own 8,192-token window. The modest LongBench V2 gains are consistent with this bet limiting performance on multi-hop reasoning that stitches evidence across distant windows, but the paper does not isolate this failure mode.
  • A natural extension would be a two-stage pipeline that first does coarse retrieval to select candidate windows, then applies QwenLong-CPRS's token critic inside those windows; this could push prefill cost toward constant time while keeping the accuracy advantages over RAG.
  • The token-critic mechanism (vocabulary labels plus boundary tags) could be reused for other compression targets, such as compressing chains of intermediate agent steps or tool outputs, where a query-dependent notion of relevance also applies.
  • The 'infinite context' framing applies to the optimizer's per-window memory; the downstream generative LLM still consumes the compressed context, so total system cost remains linear in the number of windows rather than constant.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The manuscript presents QwenLong-CPRS, a context-compression framework built on a Qwen2-7B base model that is fine-tuned to extract query-relevant tokens, sentences, or paragraphs from long contexts before passing the compressed context to a downstream LLM. The method combines a hybrid causal/bidirectional attention architecture, a token-critic mechanism that reuses the LM head for sequence labeling, and window-parallel inference to reduce prefill cost. The authors evaluate on Ruler-128K, InfiniteBench, LongBench V1/V2, and a custom NIAH test, reporting consistent gains over direct prompting, RAG, and sparse attention baselines, including the claim that Qwen2.5-32B-Instruct augmented with CPRS outperforms proprietary long-context LLMs by 4.85 points on Ruler-128K and 10.88 points on InfiniteBench.

Significance. If the reported results are robust, the framework offers a practical plug-and-play alternative to RAG and sparse attention for long-context processing, with the notable finding that small short-context models with compression can outperform much larger long-context counterparts. The paper provides a public model release, detailed training-data construction, and latency measurements, which are strengths. However, the central quantitative claims currently rest on Ruler-128K NIAH results that are vulnerable to training/evaluation overlap, and the paper's headline compression and gain figures are not consistent with the numbers in its own figures and tables. These issues need to be resolved before the significance of the empirical contributions can be fully assessed.

major comments (4)
  1. [Section 2.3 and Table 4] Section 2.3 states that the sentence-granularity training data include 'a set of training data inspired by the Needle-in-a-Haystack task, enabling the model to extract needle sentences from long documents.' Table 4 then shows that the Ruler-128K-NIAH system prompt is the verbatim Ruler generation template: 'Extract the `needles` in the format of `One of the special magic {type_needle_v} for {key} is: {value}.` from the document.' Since Ruler-128K NIAH uses this exact template, any training instance with the same template (even with different keys or values) may teach the model to emit the needle format without learning general context optimization. The near-ceiling scores in Table 1 for NIAH and NIAH-sub (99.65–99.95 for Qwen2.5-32b-instruct+CPRS) are consistent with template memorization. The paper provides no training/evaluation overlap analysis, no unseen-template or unseen-key ablation, and no variation of distractor distribution. Because the 4.85-point Ruler-128K margin over Qwen-Long and the 'small short-context LLMs outperform larger long-context counterparts' claim are driven by these NIAH numbers, the central claim is not yet supported on this benchmark.
  2. [Abstract vs. Figure 1a and Section 4.2] The abstract claims '21.59× context compression alongside 19.15-point average performance gains,' but Figure 1a reports compression rates of 72.6×, 145.2×, 290.5×, 290.5×, and 290.5× and gains of +66.2, +59.7, +17.7, +18.4, and +21.3 for the five proprietary models. No averaging of these values yields 21.59× or 19.15 points. Similarly, Section 4.2 states average gains of 54.9, 49.0, 21.7, and 15.7 for models with 32K, 64K, 128K, and 1M context windows, but Figure 5 shows per-model gains of 66.2 (32K), 59.7 (64K), 35.6/17.7/18.4 (128K), and 12.8/21.4 (1M), which average to different values. These inconsistencies affect the paper's headline claims and must be corrected or reconciled with the underlying tables.
  3. [Section 2.2, Eq. (4), and Figure 3b] Window-parallel inference partitions the long context into independent 8192-token windows, so the compressor scores each window without seeing the rest of the document. The paper's general claim of consistent superiority over RAG and sparse attention therefore assumes that a span's relevance to the query can be decided locally. This assumption is not validated for tasks requiring evidence from multiple distant windows; the modest LongBench V2 gains (+0.3 to +5.7 in Table 3) are consistent with this limitation. The authors should either demonstrate the method on tasks with cross-window evidence requirements or qualify the scope of the claimed improvements.
  4. [Table 5 and Section 4.1] Table 5 compares QwenLong-CPRS on LLaMA3.1-8b-instruct with NSA on DeepSeekMOE, attributing the larger average gain (+5.14 vs +2.62) to 'superior scalability.' Because the base models differ in architecture, scale, and training, the gains are not comparable, and the conclusion in Section 4.1 that CPRS offers superior scalability over NSA is not supported by this comparison.
minor comments (6)
  1. [Section 2.1, Eq. (2)] The mutual-information objective in Eq. (2) is presented as the optimization target, but the training loss is the SFT token-critic objective described in Section 2.2; the connection between Eq. (2) and the trained model should be stated explicitly.
  2. [Footnote 2] Footnote 2 asserts without evidence that Qwen-2-Base initialization outperforms Qwen-2.5-Base; please provide the comparison or remove the claim.
  3. [Throughout] The model name appears inconsistently as 'QWEN LONG -CPRS', 'QwenLong-CPRS', and 'QWENLONG-CPRS' across the title, abstract, body, and tables; please standardize.
  4. [Section 4.5] Section 4.5 states the two prompt configurations are 'statistically comparable,' but no statistical test or variance estimate is reported.
  5. [Table 4] The construction and scoring of the NIAH-sub subset derived from Ruler multi-key and multi-value tasks are not fully specified; please clarify the composition and evaluation protocol.
  6. [Throughout] There are several typographical and grammatical errors, including 'substential' (Section 1), 'use's' (Section 2.2), and a missing 'and' in 'the vocabulary V the positional tag set' (Section 2.2).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: reported gains are external-benchmark measurements; the only overlapping concern (NIAH-style training data) is a generalization risk, not a by-construction reduction.

full rationale

The paper's central claims are empirical: QwenLong-CPRS is trained with SFT on a token-critic task and then cascaded with various generative LLMs, with results reported on external benchmarks (Ruler-128K, InfiniteBench, LongBench V1/V2, and the NIAH pressure test). These gains are measurements, not consequences of the paper's own equations. Eq. (2) is a conceptual mutual-information objective that the paper does not actually optimize; the training procedure is supervised fine-tuning on token labels, so the objective is not used to derive the headline numbers. Eq. (4) is a straightforward arithmetic consequence of windowed partitioning and does not fold the evaluation into its own assumptions. The self-citations (Qwen technical reports, MOBA, etc.) are used for architecture initialization and baselines rather than to justify the novel claims, and no 'uniqueness theorem' is invoked. The only element that could superficially resemble circularity is Section 2.3's statement that training data were 'inspired by the Needle-in-a-Haystack task' coupled with Table 4's Ruler-128K NIAH test prompt using the exact Ruler template. However, the paper never states that the exact Ruler instances or template appeared in the 126K-sample training corpus; 'inspired by' indicates a task-family overlap rather than a demonstrated identity. The absence of a held-out NIAH variant and the lack of a training/test overlap analysis are legitimate correctness and generalization concerns, but they do not constitute a by-construction circular derivation. Under the rule that circularity requires a quotable reduction, and with the system otherwise evaluated against independent external benchmarks, the appropriate finding is no significant circularity (score 0).

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The paper's central claims rest on a small set of design choices: hand-crafted per-task prompts, window-parallel inference that assumes local relevance decisions are sufficient, and the repurposing of the language modeling head as a token critic. No new conceptual entities are introduced, and no parameters are fitted to benchmark scores in the sense of a derived prediction; the per-task prompts are, however, selected to match each benchmark's expected output granularity.

free parameters (5)
  • per-task system prompts = 5 distinct prompts (Table 4) for Ruler-128K and InfiniteBench subsets
    The benchmark scores depend on hand-designed granularity and instruction prompts; the paper does not show sensitivity to prompt wording.
  • window size w = 8192 tokens
    Used in window-parallel inference (Sections 2.2 and 4.4); affects both latency and the locality of extraction decisions.
  • parallelism factor rho = 5
    Used in the latency experiment (Section 4.4); does not affect accuracy but is part of the efficiency claim.
  • random gradient masking rate = 50% of non-critical token positions
    Training detail (Section 3.1) that may affect the learned critic but is not ablated.
  • SFT hyperparameters (LR, epochs, batch size) = 1e-5, 3 epochs, batch 256
    Training configuration (Section 3.1) that is not swept or justified.
assumptions (4)
  • domain assumption The language modeling head can be repurposed as a token critic without a dedicated classification head
    Section 2.2 describes 'Language Modeling as Token Critic' but does not provide an ablation or analysis showing this transfer preserves pretrained knowledge effectively.
  • domain assumption Bidirectional attention in upper layers improves boundary detection without degrading language modeling
    Sections 2.2 and 3.1 reconfigure layers 22-28 to bidirectional attention; the claim is asserted, not demonstrated via ablations.
  • domain assumption Token relevance can be decided from an 8192-token window independently of the rest of the context
    Window-parallel inference in Section 2.2 and Eq. 4 partitions the document; the compressor never sees cross-window dependencies, which is untested for multi-hop or distributed-evidence tasks.
  • ad hoc to paper Qwen2-Base initialization is superior to Qwen2.5-Base
    Footnote 2 asserts this without presenting supporting experiments in the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of QwenLong-CPRS: Towards $\infty$-LLMs with Dynamic Context Optimization." pith.science (2026). https://pith.science/paper/45UQKP2N

@misc{pith2026250518092,
  author       = {Pith},
  title        = {Pith review of: QwenLong-CPRS: Towards $\infty$-LLMs with Dynamic Context Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/45UQKP2N}},
  note         = {Machine review of arXiv:2505.18092}
}
abstract

This technical report presents QwenLong-CPRS, a context compression framework designed for explicit long-context optimization, addressing prohibitive computation overhead during the prefill stage and the "lost in the middle" performance degradation of large language models (LLMs) during long sequence processing. Implemented through a novel dynamic context optimization mechanism, QwenLong-CPRS enables multi-granularity context compression guided by natural language instructions, achieving both efficiency gains and improved performance. Evolved from the Qwen architecture series, QwenLong-CPRS introduces four key innovations: (1) Natural language-guided dynamic optimization, (2) Bidirectional reasoning layers for enhanced boundary awareness, (3) Token critic mechanisms with language modeling heads, and (4) Window-parallel inference. Comprehensive evaluations across five benchmarks (4K-2M word contexts) demonstrate QwenLong-CPRS's threefold effectiveness: (1) Consistent superiority over other context management methods like RAG and sparse attention in both accuracy and efficiency. (2) Architecture-agnostic integration with all flagship LLMs, including GPT-4o, Gemini2.0-pro, Claude3.7-sonnet, DeepSeek-v3, and Qwen2.5-max, achieves 21.59$\times$ context compression alongside 19.15-point average performance gains; (3) Deployed with Qwen2.5-32B-Instruct, QwenLong-CPRS surpasses leading proprietary LLMs by 4.85 and 10.88 points on Ruler-128K and InfiniteBench, establishing new SOTA performance.

Figures

Figures reproduced from arXiv: 2505.18092 by the authors.

Figure 1
Figure 1. Illustration of the performance of QWENLONG-CPRS. Figure 1a compares the input token consumption and model performance of various LLMs on Ruler-128K before (marked with ✸) and after (marked with ∆) cascading QWENLONG-CPRS. Figure 1b highlights the performance improvements of QWENLONG-CPRS over other context management methods, such as RAG [22] and sparse attention [16]. been proposed: Retrieval-augmented generation … view at source ↗
Figure 2
Figure 2. The concept of dynamic context optimization, which aims to enhance context processing efficiency by maximizing information density. Given a long-context input, this paradigm dynamically compresses it into query-specific content at varying granularities, facilitating concise and accurate information extraction for different user queries. For instance, keywords for search queries, sentences for question answering, and… view at source ↗
Figure 3
Figure 3. The model architecture and workflow of Q [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Performance of QWENLONG-CPRS in NIAH test with input length is upto 1M. Qwen2.5-max (32K) Deepseek-v3 (64K) Qwen3-32b (128K) GPT-4o (128K) Claude-3.7-sonnet (128K) Qwen-turbo-1M (1M) Gemini-2.0-pro (1M) (a) Ruler-128K 20 40 60 80 100 Score 27.1 31.6 57.0 75.0 72.3 76.8…
Figure 5
Figure 5. Figure 5: Comparative performance analysis of LLMs with and without [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Performance comparison between RAG and QWENLONG-CPRS across varying retrieved token quantities 16k 24k 32k 40k 48k 56k 64k 72k 80k 88k 96k 104k 112k 120k 128k Input Length 0 5 10 15 20 25 System Latency (s) Qwen2.5-7b-instruct Qwen2.5-7b-instruct + RAG (Recall 16K toke…
Figure 7
Figure 7. Figure 7: System latency of different context management methods with various input length. [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Example case #1: multi-value Needle-in-a-Haystack test. [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Example case #2: English multi-hop QA. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: Example case #2: Contract element extraction. [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Do Generative AI Assistants Respect robots.txt? Tracing Web Access Beyond Visible Answers

    cs.CY 2026-07 conditional novelty 6.0 of 10

    Under controlled tests, several major AI assistants (Gemini, Grok, DeepSeek, Qwen) accessed robots.txt-disallowed web pages without ever requesting the robots.txt file, while Claude and Mistral generally complied.

  2. ECHO: Prune To Act, Trace To Learn With Selective Turn Memory In Agentic RL

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    Source-indexed selective turn memory plus provenance-guided credit assignment raises BrowseComp-Plus accuracy to 43.4% over GRPO and SUPO while cutting turn proliferation.

Reference graph

Works this paper leans on

51 extracted references · 17 canonical work pages · cited by 2 Pith papers

  1. [1]

    The llama 3 herd of models, 2024

    Meta AI. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783

    arXiv 2024

  2. [2]

    Claude 3.7 sonnet

    Anthropic. Claude 3.7 sonnet. https://www.anthropic.com/claude/sonnet, 2025

    work page 2025

  3. [3]

    Longalign: A recipe for long context alignment of large language models, 2024

    Yushi Bai, Xin Lv, Jiajie Zhang, Yuze He, Ji Qi, Lei Hou, Jie Tang, Yuxiao Dong, and Juanzi Li. Longalign: A recipe for long context alignment of large language models, 2024. URL https://arxiv.org/abs/2401.18058

    arXiv 2024

  4. [4]

    Longbench: A bilingual, multitask benchmark for long context understanding

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers), pages 3119–3137, 2024

    work page 2024

  5. [5]

    Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks, 2025

    Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Ji- azheng Xu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks, 2025. URL https: //arxiv.org/abs/2412.15204

    arXiv 2025

  6. [6]

    Introduction to the CoNLL-2004 shared task: Semantic role labeling

    Xavier Carreras and Lluís Màrquez. Introduction to the CoNLL-2004 shared task: Semantic role labeling. In Proceedings of the Eighth Conference on Computational Natural Language Learning (CoNLL-2004) at HLT-NAACL 2004, pages 89–97, Boston, Massachusetts, USA, May 6 - May 7 2004. Association for Computational Linguistics. URL https://aclanthology. org/W04-2412/

    work page 2004

  7. [7]

    Walking down the memory maze: Beyond context limit through interactive reading, 2023

    Howard Chen, Ramakanth Pasunuru, Jason Weston, and Asli Celikyilmaz. Walking down the memory maze: Beyond context limit through interactive reading, 2023. URL https: //arxiv.org/abs/2310.05029

    arXiv 2023

  8. [8]

    Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y

    Damai Dai, Chengqi Deng, Chenggang Zhao, R.x. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y . Wu, Zhenda Xie, Y .k. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. DeepSeekMoE: Towards ultimate expert special- ization in mixture-of-experts language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, edito...

    work page

Show all 51 references
  1. [9]

    Introducing gemini 2.0: Our new ai model for the agentic era

    Google Deepmind. Introducing gemini 2.0: Our new ai model for the agentic era. https://blog.google/technology/google-deepmind/ google-gemini-ai-update-december-2024/#ceo-message , 2024

    2024

  2. [10]

    Gemini 2.5: Our most intelligent ai model

    Google Deepmind. Gemini 2.5: Our most intelligent ai model. https://blog.google/ technology/google-deepmind/gemini-model-thinking-updates-march-2025/ #gemini-2-5-thinking , 2025

    2025

  3. [11]

    Deepseek-v3 technical report, 2025

    DeepSeek-AI. Deepseek-v3 technical report, 2025. URL https://arxiv.org/abs/2412. 19437

    2025

  4. [12]

    Looking right is sometimes right: Investigating the capabilities of decoder-only LLMs for sequence labeling

    David Duki´c and Jan Snajder. Looking right is sometimes right: Investigating the capabilities of decoder-only LLMs for sequence labeling. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics ACL 2024 , pages 1416...

    2024

  5. [13]

    DuReader: a Chinese machine reading comprehension dataset from real-world applications

    Wei He, Kai Liu, Jing Liu, Yajuan Lyu, Shiqi Zhao, Xinyan Xiao, Yuan Liu, Yizhong Wang, Hua Wu, Qiaoqiao She, Xuan Liu, Tian Wu, and Haifeng Wang. DuReader: a Chinese machine reading comprehension dataset from real-world applications. In Eunsol Choi, Minjoon Seo, Danqi Chen, R...

    2018 doi

  6. [14]

    Ruler: What’s the real context size of your long-context language models?, 2024

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What’s the real context size of your long-context language models?, 2024. URL https://arxiv.org/abs/2404.06654

    2024 arXiv

  7. [15]

    Llmlingua: Com- pressing prompts for accelerated inference of large language models, 2023

    Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. Llmlingua: Com- pressing prompts for accelerated inference of large language models, 2023. URL https: //arxiv.org/abs/2310.05736

    2023 arXiv

  8. [16]

    Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention

    Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir Abdi, Dongsheng Li, Chin-Yew Lin, et al. Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention. Advances in Neural Information Processing ...

    2024

  9. [17]

    Needle in a haystack - pressure testing llms

    Gregory Kamradt. Needle in a haystack - pressure testing llms. https://github.com/ gkamradt/LLMTestNeedleInAHaystack/tree/main, 2023

    2023

  10. [18]

    Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav...

    2019 doi

  11. [19]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention, 2023. URL https://arxiv.org/abs/2309. 06180

    2023

  12. [20]

    The third international Chinese language processing bakeoff: Word seg- mentation and named entity recognition

    Gina-Anne Levow. The third international Chinese language processing bakeoff: Word seg- mentation and named entity recognition. In Proceedings of the Fifth SIGHAN Workshop on Chinese Language Processing, pages 108–117, Sydney, Australia, July 2006. Association for Computationa...

    2006

  13. [21]

    Retrieval-augmented generation for knowledge-intensive nlp tasks, 2021

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks, 2021. URL https://arx...

    2021 arXiv

  14. [22]

    Towards general text embeddings with multi-stage contrastive learning, 2023

    Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning, 2023. URL https: //arxiv.org/abs/2308.03281

    2023 arXiv

  15. [23]

    Retrieval augmented generation or long-context llms? a comprehensive study and hybrid approach, 2024

    Zhuowan Li, Cheng Li, Mingyang Zhang, Qiaozhu Mei, and Michael Bendersky. Retrieval augmented generation or long-context llms? a comprehensive study and hybrid approach, 2024. URL https://arxiv.org/abs/2407.16833

    2024 arXiv

  16. [24]

    Label supervised llama finetuning, 2023

    Zongxi Li, Xianming Li, Yuzhang Liu, Haoran Xie, Jing Li, Fu lee Wang, Qing Li, and Xiaoqin Zhong. Label supervised llama finetuning, 2023. URL https://arxiv.org/abs/2310. 01208

    2023

  17. [25]

    Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

    Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts, 2023. URL https://arxiv.org/abs/2307.03172. 14

    2023 arXiv

  18. [26]

    Thus spake long-context large language model, 2025

    Xiaoran Liu, Ruixiao Li, Mianqiu Huang, Zhigeng Liu, Yuerong Song, Qipeng Guo, Siyang He, Qiqi Wang, Linlin Li, Qun Liu, Yaqian Zhou, Xuanjing Huang, and Xipeng Qiu. Thus spake long-context large language model, 2025. URL https://arxiv.org/abs/2502.17129

    2025

  19. [27]

    Zhang, Zhilin Yang, Xinyu Zhou, Mingxing Zhang, and Jiezhong Qiu

    Enzhe Lu, Zhejun Jiang, Jingyuan Liu, Yulun Du, Tao Jiang, Chao Hong, Shaowei Liu, Weiran He, Enming Yuan, Yuzhi Wang, Zhiqi Huang, Huan Yuan, Suting Xu, Xinran Xu, Guokun Lai, Yanru Chen, Huabin Zheng, Junjie Yan, Jianlin Su, Yuxin Wu, Neo Y . Zhang, Zhilin Yang, Xinyu Zhou, ...

    2025 arXiv

  20. [28]

    Minimax-01: Scaling foundation models with lightning attention, 2025

    MiniMax. Minimax-01: Scaling foundation models with lightning attention, 2025. URL https://arxiv.org/abs/2501.08313

    2025 arXiv

  21. [29]

    Summarunner: A recurrent neural network based sequence model for extractive summarization of documents, 2016

    Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. Summarunner: A recurrent neural network based sequence model for extractive summarization of documents, 2016. URL https:// arxiv.org/abs/1611.04230

    2016 arXiv

  22. [30]

    Gpt-4o system card, 2024

    OpenAI. Gpt-4o system card, 2024. URL https://arxiv.org/abs/2410.21276

    2024 arXiv

  23. [31]

    Gpt-4 technical report, 2024

    OpenAI. Gpt-4 technical report, 2024. URL https://arxiv.org/abs/2303.08774

    2024 arXiv

  24. [32]

    Vicky Zhao, Lili Qiu, and Dongmei Zhang

    Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Victor Rühle, Yuqing Yang, Chin-Yew Lin, H. Vicky Zhao, Lili Qiu, and Dongmei Zhang. Llmlingua-2: Data distillation for efficient and faithful task-agnostic prompt compression, 2024. URL ...

    2024 arXiv

  25. [33]

    Yarn: Efficient context window extension of large language models, 2023

    Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models, 2023. URL https://arxiv.org/abs/2309. 00071

    2023

  26. [34]

    Qwen2.5 technical report, 2025

    Qwen. Qwen2.5 technical report, 2025. URL https://arxiv.org/abs/2412.15115

    2025 arXiv

  27. [35]

    Zero: Memory optimiza- tions toward training trillion parameter models, 2020

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimiza- tions toward training trillion parameter models, 2020. URL https://arxiv.org/abs/1910. 02054

    2020

  28. [36]

    Know what you don‘t know: Unanswerable questions for SQuAD

    Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don‘t know: Unanswerable questions for SQuAD. In Iryna Gurevych and Yusuke Miyao, editors, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (V olume 2: Short Papers) , pages 784–...

    2018 doi

  29. [37]

    Liu, and Christopher D

    Abigail See, Peter J. Liu, and Christopher D. Manning. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers), pages 1073–1083, Vancouver, Canada, July...

    2017 doi

  30. [38]

    Roformer: Enhanced transformer with rotary position embedding, 2023

    Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2023. URL https://arxiv.org/ abs/2104.09864

    2023 arXiv

  31. [39]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024

    Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024. URL https://arxiv.org/abs/2403.05530

    2024 arXiv

  32. [40]

    Generalizing an llm from 8k to 1m context using qwen-agent

    Qwen Team. Generalizing an llm from 8k to 1m context using qwen-agent. https://qwenlm. github.io/blog/qwen-agent-2405/ , 2024

    2024

  33. [41]

    Qwen3: Think deeper, act faster

    Qwen Team. Qwen3: Think deeper, act faster. https://qwenlm.github.io/blog/qwen3/, 2025

    2025

  34. [42]

    Tjong Kim Sang and Fien De Meulder

    Erik F. Tjong Kim Sang and Fien De Meulder. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003 , pages 142–147, 2003. URL https: //www.aclweb.org/ant...

    2003

  35. [43]

    MuSiQue: Multihop questions via single-hop question composition

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. MuSiQue: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539–554, 2022. doi: 10.1162/tacl_a_00475. URL https: //aclanthology.or...

    2022 doi

  36. [44]

    Smith, Daniel Khashabi, and Hannaneh Hajishirzi

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instruc- tions, 2023. URL https://arxiv.org/abs/2212.10560

    2023 arXiv

  37. [45]

    Qwen2 technical report, 2024

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze...

    2024 arXiv

  38. [46]

    Qwen2.5-1m technical report, 2025

    An Yang, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoyan Huang, Jiandong Jiang, Jianhong Tu, Jianwei Zhang, Jingren Zhou, Junyang Lin, Kai Dang, Kexin Yang, Le Yu, Mei Li, Minmin Sun, Qin Zhu, Rui Men, Tao He, Weijia Xu, Wenbiao Yin, Wenyuan Yu, Xiafei Qiu, Xingzhang R...

    2025 arXiv

  39. [47]

    Cohen, Ruslan Salakhut- dinov, and Christopher D

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhut- dinov, and Christopher D. Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering, 2018. URL https://arxiv.org/abs/1809.09600

    2018 arXiv

  40. [48]

    Infinite retrieval: Attention enhanced llms in long-context processing, 2025

    Xiaoju Ye, Zhichun Wang, and Jingyuan Wang. Infinite retrieval: Attention enhanced llms in long-context processing, 2025. URL https://arxiv.org/abs/2502.12962

    2025 arXiv

  41. [49]

    Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Y . X. Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Wenfeng Liang, and Wangding Zeng. Native sparse attention: Hardware-aligned and natively trainable sparse attentio...

    2025 arXiv

  42. [50]

    One of the special magic numbers for deep-astrology is: 3365093

    Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, and Maosong Sun.∞bench: Extending long context evaluation beyond 100k tokens, 2024. URL https://arxiv.org/abs/2402.13718. 16 A Case Study User’s Quer...

    2024 arXiv

  43. [2024]

    doi: 10.18653/v1/2024.acl-long.70

    Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.70. URL https://aclanthology.org/2024.acl-long.70/

    2024 doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.