REVIEW 9 cited by
Long Context Compression with Activation Beacon
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Long context compression is a critical research problem due to its significance in reducing the high computational and memory costs associated with LLMs. In this paper, we propose Activation Beacon, a plug-in module for transformer-based LLMs that targets effective, efficient, and flexible compression of long contexts. To achieve this, our method introduces the following technical designs. 1) We directly compress the activations (i.e. keys and values at every layer), rather than leveraging soft prompts to relay information (which constitute a major bottleneck to encapsulate the complex information within long contexts). 2) We tailor the compression workflow, where each fine-grained input unit is progressively compressed, enabling high-quality compression and efficient computation during both training and inference. 3) We train the model through compression-based auto-regression, making full use of plain texts and instructional data to optimize the model's compression performance. 4) During training, we randomly sample a compression ratio at each step, teaching the model to support a wide range of compression configurations. Extensive evaluations are conducted on various long-context tasks whose lengths (e.g., 128K) may far exceed the maximum training length (20K), such as document understanding, few-shot learning, and Needle-in-a-Haystack. Whilst existing methods struggle to handle these challenging tasks, Activation Beacon maintains a comparable performance to the uncompressed baseline across various scenarios, achieving a 2x acceleration in inference time and an 8x reduction of memory costs for KV cache. Our data, model, and code have been released at \url{https://github.com/FlagOpen/FlagEmbedding/}.
Forward citations
Cited by 9 Pith papers
-
SeDeM: Selective Decompression of Hidden-State Memories for Long-Context Question Answering
SeDeM stores long contexts as compact query-independent memory blocks, selects only query-relevant blocks, and decompresses them into intermediate decoder hidden states, beating compression baselines on four long-cont...
-
Understanding Is Done Early: A Depth Division of Labor in Large Language Models and Its Use for Unbounded-Context Memory
Long-context memory can be organized by caching intermediate residual states and recomputing only query-conditioned upper layers over a bounded retrieved pack.
-
What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agents
KV-cache eviction, prompt compression, recurrent state bounding, and agent memory consolidation are unified as one rate-distortion problem with a shared lower bound, shared failure mode, and transferable mechanisms.
-
SuperWriter: Reflection-Driven Long-Form Generation with Large Language Models
A 7B writing model trained on plan-write-refine thinking data with multi-stage preference optimization matches or beats several larger models on long-form generation benchmarks.
-
FAEDKV: Infinite-Window Fourier Transform for Unbiased KV Cache Compression
FAEDKV compresses the KV cache by keeping the most important Fourier-frequency components of cached keys and values, reporting gains over eviction baselines on LongBench and position-flat retrieval on needle-in-a-hays...
-
Video-XL-2: Towards Very Long-Video Understanding Through Task-Aware KV Sparsification
Video-XL-2 cuts long-video inference cost with chunked pre-filling and query-gated dense-or-sparse KV reloading, reporting half the FLOPs and a third less decoding memory at roughly equal benchmark scores.
-
Less is More Tokens: Efficient Math Reasoning via Difficulty-Aware Chain-of-Thought Distillation
Difficulty-aware compression of CoT traces plus SFT and DPO lets LLMs shorten reasoning on easy math problems, cutting tokens by up to 30% with mixed accuracy effects.
-
HCAttention: Extreme KV Cache Compression via Heterogeneous Attention Computing for LLMs
HCAttention combines key quantization, CPU value offloading, and cumulative-attention eviction to run long-context LLMs with 12.5% to 25% of the GPU KV cache while keeping LongBench accuracy close to full attention.
-
Toward Efficient Agents: Memory, Tool learning, and Planning
A survey that organizes efficiency techniques for LLM agents into memory, tool learning, and planning, and consolidates benchmarks and metrics for measuring cost-performance trade-offs.
Discussion (0). Sign in to comment.