REVIEW 18 cited by
LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
This paper focuses on task-agnostic prompt compression for better generalizability and efficiency. Considering the redundancy in natural language, existing approaches compress prompts by removing tokens or lexical units according to their information entropy obtained from a causal language model such as LLaMa-7B. The challenge is that information entropy may be a suboptimal compression metric: (i) it only leverages unidirectional context and may fail to capture all essential information needed for prompt compression; (ii) it is not aligned with the prompt compression objective. To address these issues, we propose a data distillation procedure to derive knowledge from an LLM to compress prompts without losing crucial information, and meantime, introduce an extractive text compression dataset. We formulate prompt compression as a token classification problem to guarantee the faithfulness of the compressed prompt to the original one, and use a Transformer encoder as the base architecture to capture all essential information for prompt compression from the full bidirectional context. Our approach leads to lower latency by explicitly learning the compression objective with smaller models such as XLM-RoBERTa-large and mBERT. We evaluate our method on both in-domain and out-of-domain datasets, including MeetingBank, LongBench, ZeroScrolls, GSM8K, and BBH. Despite its small size, our model shows significant performance gains over strong baselines and demonstrates robust generalization ability across different LLMs. Additionally, our model is 3x-6x faster than existing prompt compression methods, while accelerating the end-to-end latency by 1.6x-2.9x with compression ratios of 2x-5x. Our code is available at https://aka.ms/LLMLingua-2.
Forward citations
Cited by 18 Pith papers
-
Floor, Ceiling, and the Fusion Gap: How Much of Crowd Reading Attention Can Machines Predict?
Machines recover up to 53% of the crowd-highlight prediction headroom, and fusing five frontier models reaches about 60%, confirmed in a pre-registered replication.
-
Mach-Mind-4-Flash Technical Report
Post-training alone—parallel domain RL experts, Multi-Teacher On-Policy Distillation, and Hybrid Median-length Policy Optimization—lifts a 3B-activated MoE to roughly 100B-class agent and reasoning scores.
-
What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agents
KV-cache eviction, prompt compression, recurrent state bounding, and agent memory consolidation are unified as one rate-distortion problem with a shared lower bound, shared failure mode, and transferable mechanisms.
-
Consistent but Miscalibrated: Evaluating LLM Limitations for Risk Communication in Natural Language
Current LLMs produce consistent but miscalibrated natural-language descriptors of likelihood and uncertainty from probabilistic predictions and are not yet reliable zero-shot risk communicators.
-
Spectral-LSH: Sub-Quadratic Prompt Compression via Krylov-Projected Locality-Sensitive Hashing
At compression ratios above 8x, grouping tokens via a Krylov-projected LSH of an implicit attention kernel preserves LLM output quality far better than block averaging, at a large preprocessing latency cost.
-
Towards Efficient Large Language Reasoning Models via Extreme-Ratio Chain-of-Thought Compression
Extra-CoT trains a semantic compressor on math CoT data, applies mixed-ratio SFT, and uses CHRPO reinforcement learning to achieve over 73% token reduction on MATH-500 with 0.6% accuracy gain on Qwen3-1.7B.
-
When Compression Becomes an Attack Surface: Black-Box Attacks on Prompt-Compressed LLM Agents
The paper claims prompt compression is a new attack surface, but the abstract's COMA attack never appears in the body and the body's SoftCom requires white-box access.
-
ARC-Encoder: learning compressed text representations for large language models
ARC-Encoder pools queries in an encoder's last attention layer to produce compressed continuous representations that a frozen decoder consumes as token embeddings.
-
Can Reasoning Path still be Effective as Input? Bridging Post-Reasoning to Chain-of-Thought Compression
UCoT compresses chain-of-thought reasoning by feeding a large language model a compact continuous 'upfront thought' generated by a small model, cutting output tokens roughly in half on math benchmarks while staying cl...
-
Cartridges: Lightweight and general-purpose long context representations via self-study
A per-corpus trained KV cache, called a Cartridge, matches full-context in-context learning quality on long-document benchmarks while using up to 38.6x less serving memory.
-
EPiC: Towards Lossless Speedup for Reasoning Training through Edge-Preserving CoT Condensation
Keeping only the first and last parts of chain-of-thought training traces preserves math reasoning accuracy on MATH500 at half the sequence length, but not on harder benchmarks.
-
Concise Reasoning, Big Gains: Pruning Long Reasoning Trace with Difficulty-Aware Prompting
Fine-tuning small models on difficulty-adapted, shortened reasoning traces (LiteCoT) yields equal or better benchmark accuracy than training on much longer traces, with far fewer tokens.
-
QwenLong-CPRS: Towards $\infty$-LLMs with Dynamic Context Optimization
QwenLong-CPRS is a 7B instruction-guided compressor that shrinks long contexts to query-relevant spans, boosting downstream LLM accuracy and cutting prefill cost.
-
Beyond Hard and Soft: Hybrid Context Compression for Balancing Local and Global Information Retention
HyCo2 combines soft global compression with hard local token selection, reporting QA performance near uncompressed retrieval while cutting context tokens by about 88.8%.
-
Lightweight Chunk Selection for Mobile Retrieval-Augmented Generation
Combining question hidden states, MoE routing signals, and chunk embeddings in a compact MLP improves rank-1 evidence chunk selection for mobile RAG by 2.49 points on average across TriviaQA, PopQA, and MS MARCO.
-
Not All Errors Are Created Equal: ASCoT Addresses Late-Stage Fragility in Efficient LLM Reasoning
ASCoT claims later reasoning errors are more harmful than early ones and uses a position-weighted verifier to prune and correct CoT steps, but its key evidence is internally inconsistent.
-
Lossless Token Sequence Compression via Meta-Tokens
A new compression scheme replaces repeated token subsequences with learnable placeholder tokens, shrinking prompts by 15-27% with no loss of information, and fine-tuned LLMs perform nearly as well as on uncompressed input.
-
A Survey of LLM $\times$ DATA
A comprehensive survey of the bidirectional links between LLMs and data management, organized as DATA4LLM and LLM4DATA with a new 'IaaS' data-quality framework.
Discussion (0). Sign in to comment.