REVIEW 4 cited by
Attention Sorting Combats Recency Bias In Long Context Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Current language models often fail to incorporate long contexts efficiently during generation. We show that a major contributor to this issue are attention priors that are likely learned during pre-training: relevant information located earlier in context is attended to less on average. Yet even when models fail to use the information from a relevant document in their response, they still pay preferential attention to that document compared to an irrelevant document at the same position. We leverage this fact to introduce ``attention sorting'': perform one step of decoding, sort documents by the attention they receive (highest attention going last), repeat the process, generate the answer with the newly sorted context. We find that attention sorting improves performance of long context models. Our findings highlight some challenges in using off-the-shelf language models for retrieval augmented generation.
Forward citations
Cited by 4 Pith papers
-
Evidence Interfaces Shape How Retrieval-Augmented Readers Use Support
Top-k retrieval scores should be reported with complete-support coverage, because the same score can mix missing-support failures with reader-side interface effects.
-
Mitigating Posterior Salience Attenuation in Long-Context LLMs with Positional Contrastive Decoding
Positional Contrastive Decoding, a training-free method that contrasts standard and over-rotated RoPE logits, improves long-context retrieval and QA by a few points.
-
LISA: Linear-Indexed Sparse Attention for Efficient Long-Context Reasoning
LISA replaces standard attention with linear attention plus indexer-selected sparse attention, claiming O(nM) inference cost and accuracy gains, but the complexity analysis ignores the indexer's full-context scoring.
-
Beyond Isolated Capabilities: Bridging Long CoT Reasoning and Long-Context Understanding
DeepSeek-R1-distilled models show higher multi-document QA accuracy than their base counterparts and flatter position-bias curves, especially with 50-80 documents.
Discussion (0). Sign in to comment.