Pith. sign in

REVIEW 5 cited by

Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2403.04797 v1 pith:27FYMKFB submitted 2024-03-05 cs.CL cs.LG

Zhenyu Zhang , Runjin Chen , Shiwei Liu , Zhewei Yao , Olatunji Ruwase , Beidi Chen , Xiaoxia Wu , Zhangyang Wang This is my paper · ORCID
classification cs.CLcs.LG
keywords llmsms-poecontextlanguagemiddleapproachencodinginformation
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

This paper aims to overcome the "lost-in-the-middle" challenge of large language models (LLMs). While recent advancements have successfully enabled LLMs to perform stable language modeling with up to 4 million tokens, the persistent difficulty faced by most LLMs in identifying relevant information situated in the middle of the context has not been adequately tackled. To address this problem, this paper introduces Multi-scale Positional Encoding (Ms-PoE) which is a simple yet effective plug-and-play approach to enhance the capacity of LLMs to handle the relevant information located in the middle of the context, without fine-tuning or introducing any additional overhead. Ms-PoE leverages the position indice rescaling to relieve the long-term decay effect introduced by RoPE, while meticulously assigning distinct scaling ratios to different attention heads to preserve essential knowledge learned during the pre-training step, forming a multi-scale context fusion from short to long distance. Extensive experiments with a wide range of LLMs demonstrate the efficacy of our approach. Notably, Ms-PoE achieves an average accuracy gain of up to 3.8 on the Zero-SCROLLS benchmark over the original LLMs. Code are available at https://github.com/VITA-Group/Ms-PoE.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Mitigating Posterior Salience Attenuation in Long-Context LLMs with Positional Contrastive Decoding

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Positional Contrastive Decoding, a training-free method that contrasts standard and over-rotated RoPE logits, improves long-context retrieval and QA by a few points.

  2. RoToR: Towards More Reliable Responses for Order-Invariant Inputs

    cs.CL 2025-02 conditional novelty 6.0 of 10

    RoToR makes a frozen LLM order-invariant by circularly rotating a single global sort of segment position IDs, and Selective Routing combines it with the original model for mixed lists.

  3. On the Emergence of Position Bias in Transformers

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Multi-layer causal attention provably drives every token's context toward the first token, while decay masks and RoPE introduce a competing distance bias that trades off against depth.

  4. AdaRoPE: Not All Attention Heads Should Rotate and Scale Equally

    cs.AI 2026-06 conditional novelty 5.0 of 10

    Head-wise learnable rotary frequencies and length-dependent attention scaling (AdaRoPE) beat uniform RoPE and YaRN schedules in pretraining and 8k-to-64k context extension up to 8B scale.

  5. Pause-Tuning for Long-Context Comprehension: A Lightweight Approach to LLM Attention Recalibration

    cs.CL 2025-02 reject novelty 5.0 of 10

    Fine-tuning LLaMA models on pause-token needle-retrieval data improves single-needle long-context scores on average, but the improvement is inconsistent and turns negative in multi-needle tests.

Pith tools