REVIEW 8 cited by
InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
Large language models (LLMs) have emerged as a cornerstone in real-world applications with lengthy streaming inputs (e.g., LLM-driven agents). However, existing LLMs, pre-trained on sequences with a restricted maximum length, cannot process longer sequences due to the out-of-domain and distraction issues. Common solutions often involve continual pre-training on longer sequences, which will introduce expensive computational overhead and uncontrollable change in model capabilities. In this paper, we unveil the intrinsic capacity of LLMs for understanding extremely long sequences without any fine-tuning. To this end, we introduce a training-free memory-based method, InfLLM. Specifically, InfLLM stores distant contexts into additional memory units and employs an efficient mechanism to lookup token-relevant units for attention computation. Thereby, InfLLM allows LLMs to efficiently process long sequences with a limited context window and well capture long-distance dependencies. Without any training, InfLLM enables LLMs that are pre-trained on sequences consisting of a few thousand tokens to achieve comparable performance with competitive baselines that continually train these LLMs on long sequences. Even when the sequence length is scaled to $1,024$K, InfLLM still effectively captures long-distance dependencies. Our code can be found in \url{https://github.com/thunlp/InfLLM}.
Forward citations
Cited by 8 Pith papers
-
LOCKS: Page-Local Compact Key Summaries for Efficient Long-Context Decoding
Page-local rank-8 spectral key summaries let sparse decode selection track the exact mass oracle and match FullKV quality at ~2% attended tokens with 2× latency cut at 1M context.
-
vAttention: Verified Sparse Attention
vAttention is a sparse attention method that mixes heavy-hitter tokens with a statistically sized random sample to provide (ε, δ)-guaranteed approximation of full attention.
-
Rectified Sparse Attention
ReSA keeps sparse decoding accurate by refreshing the KV cache with dense attention every f tokens, achieving near-lossless quality with a large speedup.
-
DeltaLLM: A Training-Free Framework Exploiting Temporal Sparsity for Efficient Edge LLM Inference
DeltaLLM exploits temporal sparsity in key vectors (delta encoding) with a hybrid local-window attention to cut attention computation by up to 60% on 1-2B parameter LLMs with minimal accuracy loss.
-
Task-Aware KV Compression For Cost-Effective Long Video Understanding
Video-X2L uses bi-level KV compression with task-aware selective reloading to improve long-video QA accuracy and reduce decode-time memory versus uniform KV compression.
-
InfiniteHiP: Extending Language Model Context Up to 3 Million Tokens on a Single GPU
A modular sparse-attention framework with CPU KV offloading and layer-wise RoPE adjustment runs Llama-class models on 3M-token contexts on a single 48GB GPU without training.
-
An Overview of Algorithms for Contactless Cardiac Feature Extraction from Radar Signals: Advances and Challenges
The paper claims to be the first review of signal-processing algorithms for radar-based cardiac monitoring, with a new taxonomy and public dataset listings.
-
SeerAttention-R: Sparse Attention Adaptation for Long Reasoning
A learned gate selects the important KV blocks during long decoding, preserving math reasoning accuracy while skipping up to 90% of attention work.
Discussion (0). Continue with ORCID to comment.