REVIEW 13 cited by
Landmark Attention: Random-Access Infinite Context Length for Transformers
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
While Transformers have shown remarkable success in natural language processing, their attention mechanism's large memory requirements have limited their ability to handle longer contexts. Prior approaches, such as recurrent memory or retrieval-based augmentation, have either compromised the random-access flexibility of attention (i.e., the capability to select any token in the entire context) or relied on separate mechanisms for relevant context retrieval, which may not be compatible with the model's attention. In this paper, we present a novel approach that allows access to the complete context while retaining random-access flexibility, closely resembling running attention on the entire context. Our method uses a landmark token to represent each block of the input and trains the attention to use it for selecting relevant blocks, enabling retrieval of blocks directly through the attention mechanism instead of by relying on a separate mechanism. Our approach seamlessly integrates with specialized data structures and the system's memory hierarchy, enabling processing of arbitrarily long context lengths. We demonstrate that our method can obtain comparable performance with Transformer-XL while significantly reducing the number of retrieved tokens in each step. Finally, we show that fine-tuning LLaMA 7B with our method successfully extends its context length capacity to over 32k tokens, allowing for inference at the context lengths of GPT-4. We release the implementation of landmark attention and the code to reproduce our experiments at https://github.com/epfml/landmark-attention/.
Forward citations
Cited by 13 Pith papers
-
LOCKS: Page-Local Compact Key Summaries for Efficient Long-Context Decoding
Page-local rank-8 spectral key summaries let sparse decode selection track the exact mass oracle and match FullKV quality at ~2% attended tokens with 2× latency cut at 1M context.
-
Anti-Periodic Positional Encoding: M\"obius Boundary Conditions Make In-Context Retrieval Reliable
An anti-periodic rotary positional encoding (M\u00f6bius RoPE) on 25% of heads collapses seed-to-seed variance in needle-in-a-haystack retrieval at matched perplexity.
-
From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving
KV-cache serving systems concentrate into five archetypes under a four-axis taxonomy, with ownership explaining residual distributed design variance and seven measurement gaps blocking next steps.
-
Understanding Is Done Early: A Depth Division of Labor in Large Language Models and Its Use for Unbounded-Context Memory
Long-context memory can be organized by caching intermediate residual states and recomputing only query-conditioned upper layers over a bounded retrieved pack.
-
Raven: High-Recall Sequence Modeling with Sparse Memory Routing
Raven is a linear-time sequence model whose sparse, input-dependent routing writes tokens into dedicated memory slots, preserving long-context recall and extrapolating 16x beyond training length.
-
Remembering Distinct Items, Not Tokens: A Learnable Dirichlet-Process Cache Between State-Space Models and Attention
A DP-means allocate-on-novelty cache matches full-attention associative recall while storing only distinct items, and a minimal novelty gate recovers the rule end-to-end.
-
What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agents
KV-cache eviction, prompt compression, recurrent state bounding, and agent memory consolidation are unified as one rate-distortion problem with a shared lower bound, shared failure mode, and transferable mechanisms.
-
CoDe-R: Refining Decompiler Output with LLMs via Rationale Guidance and Adaptive Inference
Rationale-guided fine-tuning plus dual-path adaptive inference lifts a 1.3B decompiler refiner to 50% average re-executability, a new lightweight SOTA on HumanEval-Decompile.
-
Balancing Computation Load and Representation Expressivity in Parallel Hybrid Neural Networks
FlowHN splits input tokens between parallel attention and SSM branches to balance compute load, improving throughput and hardware utilization in small autoregressive language models.
-
Longer Context, Deeper Thinking: Uncovering the Role of Long-Context Ability in Reasoning
Enhancing a model's long-context ability before supervised fine-tuning improves math reasoning accuracy, including on short-input benchmarks.
-
Curse of High Dimensionality Issue in Transformer for Long-context Modeling
A group attention mechanism that keeps important tokens and merges the rest cuts long-context decoding cost, but the theoretical case for grouping is only weakly supported.
-
100-LongBench: Are de facto Long-Context Benchmarks Literally Evaluating Long-Context Ability?
100-LongBench and LongScore evaluate LLMs at controlled context lengths and rank models by relative performance drop from a short-context baseline, not by raw accuracy.
-
SELF: Self-Extend the Context Length With Logistic Growth Function
SELF replaces Self-Extend's fixed token grouping with a logistic-growth grouping schedule, giving mixed but sometimes large gains on long-context benchmarks.
Discussion (0). Continue with ORCID to comment.