Pith. sign in

REVIEW 3 cited by

Functional Interpolation for Relative Positions Improves Long Context Transformers

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2310.04418 v2 pith:42Y3KP3R submitted 2023-10-06 cs.LG

Shanda Li , Chong You , Guru Guruganesh , Joshua Ainslie , Santiago Ontanon , Manzil Zaheer , Sumit Sanghai , Yiming Yang
show 2 more authors
Sanjiv Kumar Srinadh Bhojanapalli
This is my paper · ORCID
classification cs.LG
keywords longermodelspositionrelativecontextcontextsencodingfire
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Preventing the performance decay of Transformers on inputs longer than those used for training has been an important challenge in extending the context length of these models. Though the Transformer architecture has fundamentally no limits on the input sequence lengths it can process, the choice of position encoding used during training can limit the performance of these models on longer inputs. We propose a novel functional relative position encoding with progressive interpolation, FIRE, to improve Transformer generalization to longer contexts. We theoretically prove that this can represent some of the popular relative position encodings, such as T5's RPE, Alibi, and Kerple. We next empirically show that FIRE models have better generalization to longer contexts on both zero-shot language modeling and long text benchmarks.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ClockRoPE: Random Fourier Rotations for Temporal Routine Modeling

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Random Fourier Rotations let transformer position encodings approximate any positive-definite attention kernel; ClockRoPE applies this to model daily/weekly routines in sequential recommendation.

  2. Extrapolation by Association: Length Generalization Transfer in Transformers

    cs.CL 2025-06 conditional novelty 5.0 of 10

    Length generalization on a short-trained main task can be inherited from a longer-trained related auxiliary task trained jointly with it.

  3. HoPE: Hyperbolic Rotary Positional Encoding for Stable Long-Range Dependency Modeling in Large Language Models

    cs.CL 2025-09 reject novelty 4.0 of 10

    HoPE replaces RoPE's sine/cosine rotations with hyperbolic functions plus an exponential damping term to enforce monotonic attention decay, but the claimed consistent superiority and the 'RoPE as special case' theorem...

Pith tools