REVIEW 15 cited by
LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Today's large language models (LLMs) typically train on short text segments (e.g., <4K tokens) due to the quadratic complexity of their Transformer architectures. As a result, their performance suffers drastically on inputs longer than those encountered during training, substantially limiting their applications in real-world tasks involving long contexts such as encoding scientific articles, code repositories, or long dialogues. Through theoretical analysis and empirical investigation, this work identifies three major factors contributing to this length generalization failure. Our theoretical analysis further reveals that commonly used techniques like truncating the attention window or relative positional encodings are inadequate to address them. Answering these challenges, we propose LM-Infinite, a simple and effective method for enhancing LLMs' capabilities of handling long contexts. LM-Infinite is highly flexible and can be used with most modern LLMs off-the-shelf. Without any parameter updates, it allows LLMs pre-trained with 2K or 4K-long segments to generalize to up to 200M length inputs while retaining perplexity. It also improves performance on downstream tasks such as Passkey Retrieval and Qasper in the zero-shot setting. LM-Infinite brings substantial efficiency improvements: it achieves 2.7x decoding speed up and 7.5x memory saving over the original model. Our codes are released at \url{https://github.com/Glaciohound/LM-Infinite}.
Forward citations
Cited by 15 Pith papers
-
PhyCheck: Fine-Grained Evidence-Grounded Dataset for Physical Law Understanding in Video-LLMs
PhyCheck is a 69,825-pair video QA benchmark that tests and improves Video-LLMs' ability to judge whether events obey physical laws, with fine-grained evidence questions and a context-sensitivity pilot.
-
From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving
KV-cache serving systems concentrate into five archetypes under a four-axis taxonomy, with ownership explaining residual distributed design variance and seven measurement gaps blocking next steps.
-
DeCoRAG: Cognitive Decoupling and Semantic-Aware Cropping for Complex Document Understanding
Decoupling semantic anchoring from high-res cropping fixes VLM visual attention sinks in document Graph RAG, gaining up to 12.5 SPR points and 40.8% fewer prompt tokens.
-
Long-Context Modeling with Dynamic Hierarchical Sparse Attention for Memory-Constrained LLM Inference
DHSA predicts attention sparsity via chunk-level similarity with learned dynamic boundaries and matches dense-attention accuracy on LongBench at a 2k token budget.
-
Guard Vector: Beyond English LLM Guardrails with Task-Vector Composition and Streaming-Aware Prefix SFT
A task vector from an English guard model transfers safety classification to Korean, Chinese, and Japanese models, and a prefix-SFT variant maintains accuracy under streaming with a single-token classifier.
-
SuperWriter: Reflection-Driven Long-Form Generation with Large Language Models
A 7B writing model trained on plan-write-refine thinking data with multi-stage preference optimization matches or beats several larger models on long-form generation benchmarks.
-
Native-Resolution Image Synthesis
A single diffusion transformer trained on native-resolution ImageNet achieves state-of-the-art FID at 256 and 512, and extrapolates to 1024 and 1536 with moderate degradation.
-
SALE : Low-bit Estimation for Efficient Sparse Attention in Long-context LLM Prefilling
SALE is a training-free sparse attention method that uses 4-bit quantized query-key estimates and a relative attention score to skip unimportant blocks, achieving over 3.36x prefill speedup on 64K+ token contexts with...
-
Structured Thoughts For Improved Reasoning And Context Pruning
Structured try/outcome SFT improves math reasoning by up to 8% over standard SFT and enables pruning ~85% of context with ~9% accuracy drop.
-
FAEDKV: Infinite-Window Fourier Transform for Unbiased KV Cache Compression
FAEDKV compresses the KV cache by keeping the most important Fourier-frequency components of cached keys and values, reporting gains over eviction baselines on LongBench and position-flat retrieval on needle-in-a-hays...
-
Docopilot: Improving Multimodal Models for Document-Level Understanding
A new academic-paper dataset and a retrieval-free fine-tuned InternVL2 model improve multi-page document QA accuracy and latency on several benchmarks.
-
Curse of High Dimensionality Issue in Transformer for Long-context Modeling
A group attention mechanism that keeps important tokens and merges the rest cuts long-context decoding cost, but the theoretical case for grouping is only weakly supported.
-
SELF: Self-Extend the Context Length With Logistic Growth Function
SELF replaces Self-Extend's fixed token grouping with a logistic-growth grouping schedule, giving mixed but sometimes large gains on long-context benchmarks.
-
LongMagpie: A Self-synthesis Method for Generating Large-scale Long-context Instructions
By prompting an aligned LLM with a document and the special token that precedes a user query, LongMagpie synthesizes long-context instruction data that outperforms prior datasets when used to fine-tune Llama-3-8B.
-
InfiniteHiP: Extending Language Model Context Up to 3 Million Tokens on a Single GPU
A modular sparse-attention framework with CPU KV offloading and layer-wise RoPE adjustment runs Llama-class models on 3M-token contexts on a single 48GB GPU without training.
Discussion (0). Continue with ORCID to comment.