Pith. sign in

REVIEW 3 major objections 5 minor 4 cited by

CompLLM claims that segment-wise soft compression cuts long-context Q&A cost by up to 4x while keeping or improving answer quality, without retuning the LLM.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A segment-wise soft compression method that provides up to 4x TTFT speedup, 2x KV cache reduction, and comparable or better QA quality at 2x compression.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection A promising segment-wise soft compression design whose long-context claims currently hinge on an undocumented positional-encoding assumption. the 3 major comments →

arxiv 2509.19228 v2 pith:3HEMHZAG submitted 2025-09-23 cs.CL

CompLLM: Compression for Long Context Q&A

Gabriele Berton , Jayakrishnan Unnikrishnan , Son Tran , Mubarak Shah This is my paper
classification cs.CL
keywords soft context compressionlong-context question answeringconcept embeddingsKV cache reductiontime to first tokensegment-wise compressionhidden-state distillationLLM inference acceleration
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CompLLM tries to establish that soft context compression can be made practical by compressing a long context segment by segment instead of as a single block. The method splits the context into short chunks, maps each chunk from token embeddings to half as many learned Concept Embeddings, then feeds those to a frozen LLM alongside the uncompressed question. The paper argues this design makes compression linear in context length, lets a compressor trained on 2k-token contexts work on 100k+ contexts, and lets compressed document chunks be cached and reused across queries. With a compression rate of 2, it reports up to 4x faster Time-To-First-Token on long contexts, a halved KV cache, and answer quality that matches or exceeds the uncompressed pipeline at long lengths. The reader should care because this points to a way to run long-context Q&A on smaller models without fine-tuning the model itself.

Core claim

The central claim is that the information a long context contributes to a question's answer can be carried by a sequence of learned vectors—Concept Embeddings—that are half the length of the original token sequence, provided the context is compressed in independent segments. A compressor built from the same LLM, using a LoRA adapter and a linear head, reads a 20-token segment and emits 10 Concept Embeddings at the output positions of appended EOS tokens. Training matches, layer by layer, the hidden states of the answer tokens produced from the compressed context against those produced from the uncompressed context, using a Smooth-L1 loss normalized per layer. This teaches the compressor to p

What carries the argument

Concept Embeddings (CEs): vectors in the latent space of the LLM's token embeddings that are not part of the vocabulary but are accepted by the frozen model; each CE replaces multiple token embeddings, so fewer embeddings pass through self-attention. The load-bearing design is segment-wise compression: the context is split into sentences truncated to 20 tokens, and each segment is compressed independently into CEs, so attention within the compressor is local and total compression cost is O(N·S), linear in context length N. The compressor reuses the generation LLM's own weights through a LoRA adapter, with a single linear layer reading CEs from EOS positions. The training signal is per-layer

Load-bearing premise

The premise that the base LLM can accept a sequence of Concept Embeddings much longer than its pretrained context window—without any specified position-encoding adaptation—carries the long-context results; if it fails, those results don't transfer.

What would settle it

Feed the same trained compressor and frozen LLM a context of 128k effective Concept Embeddings while holding the LLM's rotary position-encoding base and max position fixed, without any extrapolation or interpolation hacks; if answer accuracy collapses compared with the paper's reported long-context numbers, the scalability claim is not model-agnostic.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • At compression rate 2, prefill latency (the time to first token) falls by up to 4x as context grows, since prefill cost drops from O(N²) to O(N²/C²), and compression time becomes negligible for long inputs.
  • The context portion of the KV cache is halved, roughly halving memory for the cache at the same compression rate.
  • A compressor trained only on short contexts (segments at most 20 tokens, training contexts no longer than 2k tokens) transfers to contexts of 100k tokens without retraining, because each segment is compressed independently.
  • Because segments don't interact during compression, a cached compressed document remains valid across different questions and can be combined with any other compressed documents; incremental updates need only recompress the changed segment.
  • At very long context lengths, answers with CompLLM are reported to be at least as accurate as without compression, and sometimes more accurate; the paper attributes this to less attention dilution when fewer embeddings compete for attention.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If segment-wise compression is the mechanism, then the compressor architecture is largely interchangeable: a smaller, cheaper encoder could replace the LoRA-adapted LLM and inherit the same linear-cost, cacheable properties—this is a testable design choice the paper leaves open.
  • The unspoken condition on positional encodings is the main caveat for extrapolation; a straightforward experiment is to plot accuracy versus context length on a model with a hard 32k position limit and see whether the 128k-embedding results require changing the position-encoding schedule.
  • Hidden-state distillation on answer tokens suggests the compressor learns a task-relevant summary rather than a general text summary; that means it may need retraining for tasks where the relevant output segment differs (e.g., summarization or code completion), even though the compressed chunk itself could be reused.
  • The success at 2x compression raises the question of how far the rate can go; the paper suggests dynamic rates could help, and a natural test would be to let segment compression rate vary with an estimate of segment informativeness.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes CompLLM, a soft prompt-compression method for long-context QA. The context is split into segments of at most S=20 tokens; each segment is independently compressed into S/C concept embeddings (CEs) using a LoRA and linear head attached to the same frozen LLM used for generation. At inference, the concatenated CEs plus the uncompressed question are fed to the LLM. The compressor is trained by per-layer Smooth-L1 distillation of answer-token hidden states between the teacher (uncompressed context) and the student (compressed context). Experiments on NarrativeQA, SQuAD, RACE, QuAIL, and the LOFT RAG datasets with Gemma3-4B and Qwen3-4B report accuracy comparable to or better than the uncompressed context, a 2x reduction in KV-cache size, and up to 4x reduction in time-to-first-token at compression rate C=2.

Significance. The central design is clear and the paper provides a coherent complexity analysis: per-segment compression costs O(N·S), prefill cost drops from O(N^2) to O(N^2/C^2), and generation cost drops from O(NT) to O(NT/C). The reported speedup and KV-cache ratios are consistent with this analysis. The empirical design is thoughtful: cross-dataset generalization (training on NarrativeQA and RACE, testing on SQuAD and QuAIL) and a comparison with LLMLingua-2 are included. If the long-context results survive a careful audit of positional encodings and model context windows, CompLLM would be a useful contribution to soft context compression, particularly because the base LLM remains frozen and compressed segments are reusable. However, the headline scalability and LOFT claims currently rest on an undocumented assumption about how CEs beyond the pretrained context window are handled.

major comments (3)
  1. [Section 4.3 / Fig. 5 / Table 1] The paper never states the pretrained context windows of Gemma3-4B and Qwen3-4B, nor how position IDs are assigned to Concept Embeddings when the concatenated CE sequence exceeds that window. Section 4.3 says the model is fed up to "128k effective embeddings"; with C=2, the LOFT 128k-token context becomes ~64k CEs, and Fig. 5 extends to 128k CEs. If either base model has a 32k window (typical for 4B-scale open models), the long-context evaluations must be relying on silent truncation or unvalidated positional extrapolation. This is load-bearing: the near-zero baselines in Table 1 and the "CompLLM improves at long context" pattern in Fig. 5 could be an artifact of the compressed sequence fitting inside the native window while the uncompressed one does not. Please report native context lengths, the exact position-ID assignment (including any RoPE/ALiBi/truncation policy), and include a con
  2. [Table 1 / Section 4.4] The un-compressed LOFT baseline is the raw LLM on 128k tokens. If the base model cannot natively accept 128k, the baseline is not a meaningful "full context" reference: its 0.00-0.07 scores may reflect a context-window failure rather than task difficulty. The paper must state the baseline's maximum context length and the exact way the 128k prompt was fed (truncation, sliding window, or extrapolation). Without this, the improvement attributed to CompLLM is confounded with the compression-induced reduction in sequence length.
  3. [Section 4.3 / Fig. 5] The phrase "128k effective embeddings" is ambiguous. For CompLLM it means the number of embeddings actually fed to the LLM, so at C=2 the original context is ~256k tokens; for the baseline it equals the original token count. If the x-axis is effective embeddings, the two curves correspond to different original context lengths; if it is original tokens, the compressed branch is evaluated at half the effective length. Please define the x-axis precisely and report results with original context length held fixed across with/without compression, in addition to any effective-embedding-matched comparison.
minor comments (5)
  1. [Title / Abstract] The title as typeset reads "COMPRESSION FORLONGCONTEXTQ&A"; please fix the missing spaces.
  2. [Eq. (1)-(2)] The Smooth-L1 beta is fixed to 1; report sensitivity to this choice or at least state how beta was selected. Also clarify how the per-layer normalization by sigma behaves when teacher activations have very small variance.
  3. [Section 3.2] Clarify whether y is the gold answer or an LLM-generated sample, and whether y is included in the student input during training. The current wording ("can be computed online or offline, using the LLM") is ambiguous and matters for interpreting the distillation loss.
  4. [Fig. 4] Provide the exact measurement protocol for the timing plot: model, batch size, input-length range, number of runs, precision, whether the KV cache is pre-filled, and how compression time is included in the green curve.
  5. [Section 4.1] Report the number of training steps/epochs, total compute, and any early-stopping criterion; "until convergence" is not reproducible. Also list the exact HuggingFace model identifiers for Gemma3-4B and Qwen3-4B.

Circularity Check

0 steps flagged

No significant circularity: CompLLM's claims are empirical and complexity-arithmetic, not derived from fitted inputs or self-citations.

full rationale

CompLLM is an empirical systems paper; its claimed speedups and quality results are not derived from a fitted parameter or a self-citation chain. The TTFT speedup (up to 4x at C=2) follows from the stated complexity arithmetic O(N^2) -> O((N/C)^2) = O(N^2/C^2) and is then measured in Fig 4; no predicted quantity is defined in terms of the measured quantity. The quality comparisons (Fig 5, Table 1) are held-out evaluations against an uncompressed baseline on NarrativeQA/SQuAD/RACE/QuAIL test sets and the external LOFT benchmark. Training uses the same frozen LLM as teacher and student, which is self-distillation, but that is a training procedure, not a circular argument: teacher hidden states are computed from the uncompressed context and student states from compressed CEs, and the loss is a matching objective, not a claim that student equals teacher by construction. The only non-standard component, the ability to feed long CE sequences to a base LLM without specifying positional-encoding adaptation, is an unspecified implementation detail and a potential validity/robustness concern, but it is not a circular step: the results are not forced by defining CEs in terms of the output metrics, and no load-bearing self-citation or imported uniqueness theorem appears.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 1 invented entities

The central claim rests on the base LLM's ability to accept arbitrary continuous embeddings (CEs), the sufficiency of answer-position hidden-state distillation as supervision, and an unstated assumption that positional encodings can be extrapolated or interpolated to very long CE sequences. The paper contributes a new method, not a derivation, so the axiom burden is empirical design choices plus background assumptions about frozen LLM behavior.

free parameters (5)
  • compression_rate_C = 2
    Fixed at 2 for all main experiments; controls the trade-off between sequence length reduction and information preservation. Not derived, chosen by hand.
  • segment_size_S = 20
    Chosen so that per-segment attention cost O(S^2) is small while each segment carries enough context. No ablation is shown.
  • smooth_l1_beta = 1
    PyTorch default; not tuned.
  • learning_rate = 1e-4
    Optimizer setting; standard and not central to the claim.
  • batch_size = 4
    Training configuration; not central.
axioms (4)
  • domain assumption The base LLM can be fed continuous vectors ('Concept Embeddings') outside its token embedding table and produce meaningful hidden states.
    Invoked in Section 3.1 (Figure 2) and essential to the method; supported by prior soft-prompt work but not proven for these models.
  • domain assumption Matching teacher hidden states at answer positions is sufficient supervision for the compressor to preserve the information needed to answer arbitrary questions about the context.
    The training objective (Eq. 1-3) assumes answer-token hidden states capture all question-relevant context information.
  • domain assumption The base LLM can process up to 128k embeddings (CEs) even if its pretrained context window may be smaller; no position-encoding adaptation is described.
    Sections 3.1 and 4.3 feed up to 128k effective embeddings without discussing positional encodings or context-window limits.
  • ad hoc to paper Independent per-segment compression does not lose information that spans segment boundaries, or if it does, the loss is acceptable for the evaluated QA tasks.
    The design never lets one segment attend to another during compression; multi-hop reasoning across sentences is therefore handled only by the frozen LLM's ability to combine the compressed segment representations.
invented entities (1)
  • Concept Embeddings (CEs) no independent evidence
    purpose: Compact latent representations of text segments that replace token embeddings for the frozen LLM.
    CEs are trained outputs and are validated only inside the paper's own evaluation. The paper provides no external falsifiable handle (e.g., a pretrained model or a measurable property) separate from its downstream results.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of CompLLM: Compression for Long Context Q&A." pith.science (2026). https://pith.science/paper/3HEMHZAG

@misc{pith2026250919228,
  author       = {Pith},
  title        = {Pith review of: CompLLM: Compression for Long Context Q&A},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3HEMHZAG}},
  note         = {Machine review of arXiv:2509.19228}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Large Language Models (LLMs) face significant computational challenges when processing long contexts due to the quadratic complexity of self-attention. While soft context compression methods, which map input text to smaller latent representations, have shown promise, their real-world adoption is limited. Existing techniques typically compress the context as a single unit, which leads to quadratic compression complexity and an inability to reuse computations across queries with overlapping contexts. In this work, we introduce CompLLM, a soft compression technique designed for practical deployment. Instead of processing the context holistically, CompLLM divides it into segments and compresses each one independently. This simple design choice yields three critical properties: efficiency, as the compression step scales linearly with the context length; scalability, enabling models trained on short sequences (e.g., 1k tokens) to generalize to contexts of 100k tokens; and reusability, allowing compressed segments to be cached and reused across different queries. Our experiments show that with a 2x compression rate, at high context lengths CompLLM speeds up Time To First Token (TTFT) by up to 4x and reduces the KV cache size by 50%. Furthermore, CompLLM achieves performance comparable to that obtained with the uncompressed context, and even surpasses it on very long sequences, demonstrating its effectiveness and practical utility.

Figures

Figures reproduced from arXiv: 2509.19228 by Gabriele Berton, Jayakrishnan Unnikrishnan, Mubarak Shah, Son Tran.

Figure 1
Figure 1. Figure 1: At high context lengths, CompLLM leads to considerable speedup and improved results, [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Conceptualization of Token Embeddings (TEs) (Top) and Concept Embeddings (CEs) (Bottom), and how they can both lead to the same output, using the sentence “golden dogs are called” as an example. TEs are contained in the LLM’s embeddings table and limited to roughly 200k (e.g. 262k for Gemma3 models and 151k for Qwen3 models). CEs lie in the same features space as TEs, but are not limited in number, and can… view at source ↗
Figure 3
Figure 3. Figure 3: Training protocol of CompLLM for context-based Q&A. The CompLLM (made of the [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Inference speed with and without CompLLM, for contexts of different lengths (x axis), for [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Results with and without compression across multiple context lengths for four datasets, [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Results with Gemma3-4B with no compression, with CompLLM, and with LLMLingua-2. [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. ReSum: Synergizing LLM Reasoning and Summarization with Reinforcement Learning

    cs.AI 2026-06 unverdicted novelty 7.0

    ReSum trains LLMs via RLVR to self-summarize reasoning trajectories, yielding 4% average performance gains and 18.6% shorter rollouts through contrastive rollout branches.

  2. SeDeM: Selective Decompression of Hidden-State Memories for Long-Context Question Answering

    cs.CL 2026-07 conditional novelty 6.0

    SeDeM stores long contexts as compact query-independent memory blocks, selects only query-relevant blocks, and decompresses them into intermediate decoder hidden states, beating compression baselines on four long-cont...

  3. ReSum: Synergizing LLM Reasoning and Summarization with Reinforcement Learning

    cs.AI 2026-06 conditional novelty 6.0

    ReSum's contrastive RL branching on self-summarization points improves LLM math reasoning accuracy by about 4% and shortens rollouts by about 18.6% across tested backbones.

  4. When Less is More: The LLM Scaling Paradox in Context Compression

    cs.LG 2026-02 unverdicted novelty 6.0

    Larger LLM compressors in lossy setups often yield less faithful context reconstructions due to knowledge overwriting and semantic drift, with mid-sized models outperforming larger ones across 27 tested configurations.

Reference graph

Works this paper leans on

65 extracted references · 2 canonical work pages · cited by 3 Pith papers

  1. [1]

    Prompt-saw: Leveraging relation-aware graphs for textual prompt compression

    Muhammad Asif Ali, Zhengping Li, Shu Yang, Keyuan Cheng, Yang Cao, Tianhao Huang, Lijie Hu, Lu Yu, and Di Wang. Prompt-saw: Leveraging relation-aware graphs for textual prompt compression. CoRR, abs/2404.00489, 2024. URL https://doi.org/10.48550/arXiv.2404.00489

    Pith/arXiv arXiv doi:10.48550/arxiv.2404.00489 2024

  2. [2]

    QAMPARI : A benchmark for open-domain questions with many answers

    Samuel Amouyal, Tomer Wolfson, Ohad Rubin, Ori Yoran, Jonathan Herzig, and Jonathan Berant. QAMPARI : A benchmark for open-domain questions with many answers. In Sebastian Gehrmann, Alex Wang, Jo \ a o Sedoc, Elizabeth Clark, Kaustubh Dhole, Khyathi Raghavi Chandu, Enrico Santus, and Hooman Sedghamiz (eds.), Proceedings of the Third Workshop on Natural La...

    2023

  3. [3]

    The claude 3 model family: Opus, sonnet, haiku, 2024

    Anthropic . The claude 3 model family: Opus, sonnet, haiku, 2024. Claude-3 Model Card

    2024

  4. [4]

    Retaining key information under high compression ratios: Query-guided compressor for LLM s

    Zhiwei Cao, Qian Cao, Yu Lu, Ningxin Peng, Luyang Huang, Shanbo Cheng, and Jinsong Su. Retaining key information under high compression ratios: Query-guided compressor for LLM s. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 126...

    work page doi:10.18653/v1/2024.acl-long.685 2024

  5. [5]

    Kv-distill: Nearly lossless learnable context compression for llms, 2025

    Vivek Chari, Guanghui Qin, and Benjamin Van Durme. Kv-distill: Nearly lossless learnable context compression for llms, 2025. URL https://arxiv.org/abs/2503.10337

    Pith/arXiv arXiv 2025

  6. [6]

    DAST : Context-aware compression in LLM s via dynamic allocation of soft tokens

    Shaoshen Chen, Yangning Li, Zishan Xu, Yongqin Zeng, Shunlong Wu, Xinshuo Hu, Zifei Shan, Xin Su, Jiwei Tang, Yinghui Li, and Hai-Tao Zheng. DAST : Context-aware compression in LLM s via dynamic allocation of soft tokens. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Findings of the Association for Computational Li...

    2025

  7. [7]

    x RAG : Extreme context compression for retrieval-augmented generation with one token

    Xin Cheng, Xun Wang, Xingxing Zhang, Tao Ge, Si-Qing Chen, Furu Wei, Huishuai Zhang, and Dongyan Zhao. x RAG : Extreme context compression for retrieval-augmented generation with one token. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=6pTlXqrO0p

    2024

  8. [8]

    Selection-p: Self-supervised task-agnostic prompt compression for faithfulness and transferability

    Tsz Ting Chung, Leyang Cui, Lemao Liu, Xinting Huang, Shuming Shi, and Dit-Yan Yeung. Selection-p: Self-supervised task-agnostic prompt compression for faithfulness and transferability. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp.\ 11057--11070, Miami, Florida, USA,...

    doi:10.18653/v1/2024.findings-emnlp.646 2024

  9. [9]

    Finch: Prompt-guided key-value cache compression for large language models

    Giulio Corallo and Paolo Papotti. Finch: Prompt-guided key-value cache compression for large language models. Transactions of the Association for Computational Linguistics, 12: 0 1517--1532, 2024

    2024

  10. [10]

    BERT: pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming - Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAA...

    2019

  11. [11]

    Efficient prompt compression with evaluator heads for long-context transformer inference, 2025

    Weizhi Fei, Xueyan Niu, Guoqing Xie, Yingqing Liu, Bo Bai, and Wei Han. Efficient prompt compression with evaluator heads for long-context transformer inference, 2025. URL https://arxiv.org/abs/2501.12959

    Pith/arXiv arXiv 2025

  12. [12]

    Beam search strategies for neural machine translation

    Markus Freitag and Yaser Al - Onaizan. Beam search strategies for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation, pp.\ 56--60, Vancouver, 2017. Association for Computational Linguistics. doi:10.18653/v1/W17-3207

    doi:10.18653/v1/w17-3207 2017

  13. [13]

    In-context autoencoder for context compression in a large language model

    Tao Ge, Hu Jing, Lei Wang, Xun Wang, Si-Qing Chen, and Furu Wei. In-context autoencoder for context compression in a large language model. In The Twelfth International Conference on Learning Representations, 2024 a . URL https://openreview.net/forum?id=uREj4ZuGJE

    2024

  14. [14]

    In-context autoencoder for context compression in a large language model

    Tao Ge, Hu Jing, Lei Wang, Xun Wang, Si-Qing Chen, and Furu Wei. In-context autoencoder for context compression in a large language model. In The Twelfth International Conference on Learning Representations, 2024 b . URL https://openreview.net/forum?id=uREj4ZuGJE

    2024

  15. [15]

    Lo RA : Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lo RA : Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9

    2022

  16. [16]

    Recurrent context compression: Efficiently expanding the context window of llm, 2024

    Chensen Huang, Guibo Zhu, Xuepeng Wang, Yifei Luo, Guojing Ge, Haoran Chen, Dong Yi, and Jinqiao Wang. Recurrent context compression: Efficiently expanding the context window of llm, 2024. URL https://arxiv.org/abs/2406.06110

    Pith/arXiv arXiv 2024

  17. [17]

    Taeho Hwang, Sukmin Cho, Soyeong Jeong, Hoyun Song, SeungYoon Han, and Jong C. Park. EXIT : Context-aware extractive compression for enhancing retrieval-augmented generation. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Findings of the Association for Computational Linguistics: ACL 2025, pp.\ 4895--4924, Vienna, A...

    doi:10.18653/v1/2025.findings-acl.253 2025

  18. [18]

    Perceiver: General perception with iterative attention, 2021

    Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, and Joao Carreira. Perceiver: General perception with iterative attention, 2021

    2021

  19. [19]

    LLML ingua: Compressing prompts for accelerated inference of large language models

    Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. LLML ingua: Compressing prompts for accelerated inference of large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp.\ 13358--13376, Singapore, December 2023. Association for Co...

    doi:10.18653/v1/2023.emnlp-main.825 2023

  20. [20]

    L ong LLML ingua: Accelerating and enhancing LLM s in long context scenarios via prompt compression

    Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. L ong LLML ingua: Accelerating and enhancing LLM s in long context scenarios via prompt compression. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long ...

    doi:10.18653/v1/2024.acl-long.91 2024

  21. [21]

    Compressed context memory for online language model interaction

    Jang-Hyun Kim, Junyoung Yeom, Sangdoo Yun, and Hyun Oh Song. Compressed context memory for online language model interaction. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=64kSvC4iPg

    2024

  22. [22]

    Unsupervised multilingual sentence boundary detection

    Tibor Kiss and Jan Strunk. Unsupervised multilingual sentence boundary detection. Computational Linguistics, 32 0 (4): 0 485--525, 2006. doi:10.1162/coli.2006.32.4.485. URL https://aclanthology.org/J06-4003/

    doi:10.1162/coli.2006.32.4.485 2006

  23. [23]

    Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transac...

    doi:10.1162/tacl_a_00276 2019

  24. [24]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023

    2023

  25. [25]

    RACE : Large-scale R e A ding comprehension dataset from examinations

    Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. RACE : Large-scale R e A ding comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp.\ 785--794, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi:10.18653/v1/D17-1082. URL http...

    doi:10.18653/v1/d17-1082 2017

  26. [26]

    Jinhyuk Lee, Anthony Chen, Zhuyun Dai, Dheeru Dua, Devendra Singh Sachan, Michael Boratko, Yi Luan, Sébastien M. R. Arnold, Vincent Perot, Siddharth Dalmia, Hexiang Hu, Xudong Lin, Panupong Pasupat, Aida Amini, Jeremy R. Cole, Sebastian Riedel, Iftekhar Naim, Ming-Wei Chang, and Kelvin Guu. Can long-context language models subsume retrieval, rag, sql, and...

    Pith/arXiv arXiv 2024

  27. [27]

    u ttler, Mike Lewis, Wen tau Yih, Tim Rockt \

    Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \"u ttler, Mike Lewis, Wen tau Yih, Tim Rockt \"a schel, Sebastian Riedel, and Douwe Kiela. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks . In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neur...

    2020

  28. [28]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \"u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \"a schel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Advances in Neural Information Processing Systems, volume 33, 2020 b

    2020

  29. [29]

    Prefix-tuning: Optimizing continuous prompts for generation, 2021 a

    Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation, 2021 a

    2021

  30. [30]

    Prefix-tuning: Optimizing continuous prompts for generation

    Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Jungo Kasai, Jinho D. Choi, and Xiang Lorraine Li (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp.\ 4582--4597,...

    2021

  31. [31]

    500x C ompressor: Generalized prompt compression for large language models

    Zongqian Li, Yixuan Su, and Nigel Collier. 500x C ompressor: Generalized prompt compression for large language models. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 25081--25091, Vienna, Austria, Ju...

    doi:10.18653/v1/2025.acl-long.1219 2025

  32. [32]

    Barys Liskavets, Maxim Ushakov, Shuvendu Roy, Mark Klibanov, Ali Etemad, and Shane K. Luke. Prompt compression with context-aware sentence encoding for fast and improved llm inference. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp.\ 24595--24604, 2025

    2025

  33. [33]

    Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

    Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12: 0 151--167, 2024 a

    2024

  34. [34]

    Cachegen: Kv cache compression and streaming for fast large language model serving

    Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Ananthanarayanan, Michael Maire, Henry Hoffmann, Ari Holtzman, and Junchen Jiang. Cachegen: Kv cache compression and streaming for fast large language model serving. In Proceedings of the ACM SIGCOMM Conference. ACM, 2024 b . doi:10.1145/36...

    arXiv 2024

  35. [35]

    Quest: A retrieval dataset of entity-seeking queries with implicit set operations

    Chaitanya Malaviya, Peter Shaw, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Quest: A retrieval dataset of entity-seeking queries with implicit set operations. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 14032--14047, 2023

    2023

  36. [36]

    Learning to compress prompts with gist tokens

    Jesse Mu, Xiang Lisa Li, and Noah Goodman. Learning to compress prompts with gist tokens. In Thirty-seventh Conference on Neural Information Processing Systems, 2023 a . URL https://openreview.net/forum?id=2DtxPCL3T5

    2023

  37. [37]

    Learning to compress prompts with gist tokens

    Jesse Mu, Xiang Lisa Li, and Noah Goodman. Learning to compress prompts with gist tokens. In Thirty-seventh Conference on Neural Information Processing Systems, 2023 b . URL https://openreview.net/forum?id=2DtxPCL3T5

    2023

  38. [38]

    OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Madry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex Paino, Alex Renzin, Alex Tachard Passos, Alexander Kirillov, Alexi Christakis, Alex...

    Pith/arXiv arXiv 2024

  39. [39]

    Vicky Zhao, Lili Qiu, and Dongmei Zhang

    Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Victor Ruhle, Yuqing Yang, Chin-Yew Lin, H. Vicky Zhao, Lili Qiu, and Dongmei Zhang. LLML ingua-2: Data distillation for efficient and faithful task-agnostic prompt compression. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for ...

    2024

  40. [40]

    PyTorch : An imperative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Soumith Chintala, Guillaume Desmaison, Edward Killeen, Zhikang Lin, Naresh Singh, J \'E ric Tauber, Alban Torossian, Vaibhav Chaniot, and Yi Yang. PyTorch : An imperative style, high-performance deep learning library. In Hanna Wallach, Hugo Larochelle, Alina Beygelzimer, ...

    2019

  41. [42]

    Long context in-context compression by getting to the gist of gisting, 2025 b

    Aleksandar Petrov, Mark Sandler, Andrey Zhmoginov, Nolan Miller, and Max Vladymyrov. Long context in-context compression by getting to the gist of gisting, 2025 b . URL https://arxiv.org/abs/2504.08934

    Pith/arXiv arXiv 2025

  42. [43]

    SQ u AD : 100,000+ questions for machine comprehension of text

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQ u AD : 100,000+ questions for machine comprehension of text. In Jian Su, Kevin Duh, and Xavier Carreras (eds.), Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp.\ 2383--2392, Austin, Texas, November 2016. Association for Computational Linguistic...

    doi:10.18653/v1/d16-1264 2016

  43. [44]

    Getting closer to AI complete question answering: A set of prerequisite real tasks

    Anna Rogers, Olga Kovaleva, Matthew Downey, and Anna Rumshisky. Getting closer to AI complete question answering: A set of prerequisite real tasks. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational A...

    2020

  44. [45]

    The NarrativeQA reading comprehension challenge

    Tom\'a s Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, G\'abor Melis, and Edward Grefenstette. The NarrativeQA reading comprehension challenge. Transactions of the Association for Computational Linguistics, TBD: 0 TBD, 2018. URL https://TBD

    2018

  45. [46]

    Codi: Compressing chain-of-thought into continuous space via self-distillation, 2025

    Zhenyi Shen, Hanqi Yan, Linhai Zhang, Zhanghao Hu, Yali Du, and Yulan He. Codi: Compressing chain-of-thought into continuous space via self-distillation, 2025. URL https://arxiv.org/abs/2502.21074

    Pith/arXiv arXiv 2025

  46. [47]

    Perception compressor: A training-free prompt compression framework in long context scenarios

    Jiwei Tang, Jin Xu, Tingwei Lu, Zhicheng Zhang, YimingZhao YimingZhao, LinHai LinHai, and Hai-Tao Zheng. Perception compressor: A training-free prompt compression framework in long context scenarios. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Findings of the Association for Computational Linguistics: NAACL 2025, pp.\ 4093--4108, Albuquerque, New M...

    doi:10.18653/v1/2025.findings-naacl.229 2025

  47. [48]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context , 2024

    Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context , 2024

    2024

  48. [49]

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, Gaël Liu, Francesco Visin, Kathleen Kenealy, Lucas Bey...

    Pith/arXiv arXiv 2025

  49. [50]

    BEIR : A heterogeneous benchmark for zero-shot evaluation of information retrieval models

    Nandan Thakur, Nils Reimers, Andreas R \"u ckl \'e , Abhishek Srivastava, and Iryna Gurevych. BEIR : A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL https://openreview.net/forum?id=wCu6T5xFjeJ

    2021

  50. [51]

    M u S i Q ue: Multihop questions via single-hop question composition

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. M u S i Q ue: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 2022

    2022

  51. [52]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https:...

    2017

  52. [53]

    In-context former: Lightning-fast compressing context for large language model

    Xiangfeng Wang, Zaiyi Chen, Tong Xu, Zheyong Xie, Yongyi He, and Enhong Chen. In-context former: Lightning-fast compressing context for large language model. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp.\ 2445--2460, Miami, Florida, USA, November 2024 a . Association...

    2024

  53. [54]

    In-context former: Lightning-fast compressing context for large language model

    Xiangfeng Wang, Zaiyi Chen, Tong Xu, Zheyong Xie, Yongyi He, and Enhong Chen. In-context former: Lightning-fast compressing context for large language model. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp.\ 2445--2460, Miami, Florida, USA, November 2024 b . Association...

    doi:10.18653/v1/2024.findings-emnlp.138 2024

  54. [55]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pp.\ 24824--24837, 2022

    2022

  55. [56]

    RECOMP : Improving retrieval-augmented LM s with context compression and selective augmentation

    Fangyuan Xu, Weijia Shi, and Eunsol Choi. RECOMP : Improving retrieval-augmented LM s with context compression and selective augmentation. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=mlJLVigNHp

    2024

  56. [57]

    Qwen3 technical report, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, ...

    Pith/arXiv arXiv 2025

  57. [58]

    PRCA : Fitting black-box large language models for retrieval question answering via pluggable reward-driven contextual adapter

    Haoyan Yang, Zhitao Li, Yong Zhang, Jianzong Wang, Ning Cheng, Ming Li, and Jing Xiao. PRCA : Fitting black-box large language models for retrieval question answering via pluggable reward-driven contextual adapter. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL https://openreview.net/forum?id=gI11vXg1W4

    2023

  58. [59]

    Cohen, Ruslan Salakhutdinov, and Christopher D

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA : A dataset for diverse, explainable multi-hop question answering. In Conference on Empirical Methods in Natural Language Processing ( EMNLP ) , 2018

    2018

  59. [60]

    WebVoyager: Building an End-to-End Web Agent that Masters Complex Tasks , 2024

    Hong-Bin Zeng, Chen-Chung Hsieh, Cheng-I Lai, and Pu-Jen Cheng. WebVoyager: Building an End-to-End Web Agent that Masters Complex Tasks , 2024

    2024

  60. [61]

    DAC : A dynamic attention-aware approach for task-agnostic prompt compression

    Yi Zhao, Zuchao Li, Hai Zhao, Baoyuan Qi, and Liu Guoming. DAC : A dynamic attention-aware approach for task-agnostic prompt compression. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 19395--19407, ...

    doi:10.18653/v1/2025.acl-long.952 2025

  61. [62]

    Leveraging attention to effectively compress prompts for long-context llms

    Yunlong Zhao, Haoran Wu, and Bo Xu. Leveraging attention to effectively compress prompts for long-context llms. Proceedings of the AAAI Conference on Artificial Intelligence, 39 0 (24): 0 26048--26056, Apr. 2025 b . doi:10.1609/aaai.v39i24.34800. URL https://ojs.aaai.org/index.php/AAAI/article/view/34800

    work page doi:10.1609/aaai.v39i24.34800 2025

  62. [63]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  63. [64]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  64. [65]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  65. [66]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.