Pith. sign in

REVIEW 5 major objections 5 minor 2 cited by

This paper argues that replacing RoPE's sine/cosine rotations with hyperbolic 'boost' rotations plus an exponential penalty makes attention weights decay monotonically with distance, eliminating the oscillation that hinders stable long-rang

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

HoPE replaces RoPE's sine/cosine rotations with hyperbolic functions plus an exponential damping term to enforce monotonic attention decay, but the claimed consistent superiority and the 'RoPE as special case' theorem are not supported.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection The central claim of monotonic decay is false as stated; the paper has a new formula and reasonable experiments but needs major revision. the 5 major comments →

arxiv 2509.05218 v2 pith:SMERQVG2 submitted 2025-09-05 cs.CL cs.AI

HoPE: Hyperbolic Rotary Positional Encoding for Stable Long-Range Dependency Modeling in Large Language Models

Chang Dai , Hongyu Shan , Mingyang Song , Di Liang This is my paper
classification cs.CL cs.AI
keywords hyperbolic rotary positional encodinglength extrapolationrelative position encodingLorentz transformationmonotonic attention decayRoPElong-range dependencyTransformer attention
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the oscillatory attention patterns of Rotary Positional Encoding (RoPE) can be removed by replacing its sine/cosine rotation blocks with hyperbolic rotations drawn from Lorentz boosts, and that the resulting encoding—HoPE—makes attention weights decay monotonically with token distance. The key theoretical claim is that with a damping coefficient larger than every rotation frequency, the attention score contains a factor that forces exponential decay as positions separate, while a separate theorem says the encoding still lets any relative position become the preferred one. On train-short-test-long perplexity (PG19, arXiv) and on the SCROLLS long-context tasks, the paper reports HoPE beating RoPE on extended sequences and besting ALiBi on several downstream tasks. If correct, HoPE is a drop-in positional encoding for decoder-only language models that trades RoPE's periodic curl for a smooth, local-first but long-range-aware decay.

Core claim

HoPE replaces each 2×2 cosine/sine rotation block of RoPE with a hyperbolic 'boost' matrix B(θ,m)=[[cosh mθ, sinh mθ],[sinh mθ, cosh mθ]], applying it to queries at angle +mθ and to keys at −mθ, then premultiplying the query block by e^{−mθ′} and the key block by e^{+mθ′}. When the query–key inner product is computed, the penalty and the boost combine into e^{−(m−n)θ′} cosh((m−n)θ_i); under a causal mask (n≤m) and with θ′>θ_i for every dimension i, this behaves as e^{−(m−n)(θ′−θ_i)} and decays exponentially with distance. That monotonic decay is the paper's central result: it removes the oscillation that RoPE inherits from trigonometric periodicity, and the paper argues it makes long-range a

What carries the argument

The carrying object is the hyperbolic rotation (boost) matrix B(θ,m) with entries cosh(mθ) and sinh(mθ), paired with the exponential penalty e^{∓mθ′} on query and key. The product of the two produces the decay kernel e^{−(m−n)θ′} cosh((m−n)θ_i); the condition θ′>θ_i converts the cosh growth into net exponential decay. For d-dimensional embeddings, the construction is block-diagonal, with each 2D subspace getting its own frequency θ_i and a shared global damping θ′.

Load-bearing premise

The monotonic-decay theorem holds only if the damping coefficient θ′ is larger than every per-dimension rotation frequency θ_i, and the argument quietly assumes a causal mask so that later tokens never attend to earlier ones; the paper gives no rule for choosing θ′ and does not state the causal-only qualification, so the central decay guarantee rests on an unspecified hyperparameter and an unstated masking condition.

What would settle it

Train or run a HoPE model with θ′ chosen smaller than the largest θ_i: the analysis predicts attention weights that grow with distance rather than decay. Alternatively, extract attention-weight curves from a trained HoPE model for relative distances up to the training length; any non-monotonic increase in the smoothed decay curve would contradict the central claim.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If HoPE's decay guarantee holds, Transformer attention over long sequences becomes smooth and local-first: nearby tokens dominate while distant ones still contribute a controlled, exponentially suppressed signal.
  • HoPE replaces RoPE in existing decoder-only LLMs with a drop-in change to the position-encoding kernel, so the stability gain does not require altering the attention mechanism itself.
  • The reported BiPE-HoPE results imply HoPE compounds with interpolation methods: pre-trained HoPE models extrapolate further after BiPE-style fine-tuning than RoPE does.
  • Because the discrimination theorem guarantees any relative position can be made the attention argmax, HoPE retains RoPE's representational power in principle even while biasing toward locality.
  • The generalized Lorentz formulation places RoPE inside a larger family of positional encodings, so future encodings can interpolate between orthogonal rotation and hyperbolic boost behavior.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial: the decay proof requires n≤m, so HoPE as presented is a causal/decoder-only encoding; applying the same penalty to bidirectional attention would amplify far-future tokens instead of suppressing them, so an adaptation (e.g., using |m−n| in the penalty) would be needed for encoders.
  • Editorial: exponential decay is a strong locality prior, and although the paper proves that a suitable key can always make a distant position win, it does not show that gradient descent can learn such keys; a testable extension is to compare HoPE vs RoPE on synthetic copy/retrieval tasks at long range.
  • Editorial: a precision test of the 'RoPE is a special case' claim would be to take the limit of HoPE's boost matrix as θ→0 scaled appropriately and show it converges to the rotation matrix; if the limit fails, the unification claim reduces to a shared Lorentz-group ancestry rather than exact containment.
  • Editorial: the linear penalty e^{−mθ′} depends on absolute position m, not just relative distance, so HoPE's behavior may drift as sequence length grows; a natural variant is to make θ′ itself position-independent or schedule it, and measure whether the perplexity gains persist at 100k+ tokens.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes Hyperbolic Rotary Positional Encoding (HoPE), a positional encoding scheme that replaces RoPE's trigonometric rotations with hyperbolic sine/cosine 'boost' matrices and adds an exponential damping coefficient. The authors claim that HoPE enforces monotonic decay of attention weights with distance, that RoPE is a special case of their formulation, and that experiments on perplexity and SCROLLS benchmarks show consistent superiority over existing methods. The paper includes a theoretical analysis in Section 3.2, implementation in Algorithm 1, and experiments in Section 4.

Significance. If the monotonic-decay guarantee and the empirical superiority claims were both correct, HoPE would be a useful contribution to long-context Transformer design, since it directly targets the oscillatory attention weights of RoPE. The paper also ships an explicit algorithm and makes falsifiable experimental comparisons. However, the central theoretical claim is not established (indeed it is false as stated), the 'RoPE as a special case' claim is unsupported, and the reported experimental results contradict the 'consistently exceeds' assertion. For these reasons the manuscript in its current form does not provide a reliable basis for the claimed contribution.

major comments (5)
  1. [Section 3.2.1, Eq. (10)-(12)] The claimed monotonic decay of attention weights is not guaranteed even under the stated condition θ' > θ_i. For a 2D block, the pre-softmax score is g(d) = e^{-θ'd}[A cosh(θ d) + B sinh(θ d)] with d = m-n (from Eq. 9). Take q=(1,0), k=(0,1), so A=0, B=1, with θ=1, θ'=1.5. Then g(d)=e^{-1.5d} sinh(d), whose derivative is e^{-1.5d}(cosh d - 1.5 sinh d), positive for d < 0.5 ln(5) ≈ 0.80. Thus g increases from 0 to a maximum before decaying; it is not monotonically non-increasing. Moreover, Eq. (9) contains e^{-(m-n)θ'}, which becomes an amplification when n>m, so the guarantee also requires an unstated causal-mask assumption (n≤m). Neither the condition θ' > max_i θ_i nor the causal-only qualification appears in the main theorem or Algorithm 1, and no selection rule or concrete value for θ' is given. The central mechanism is therefore unproven.
  2. [Abstract and Section 4, Tables 1-3] The statement that HoPE 'consistently exceeds existing positional encoding methods' is contradicted by the paper's own tables. In Table 2 (arXiv PPL at 6144), ALiBi achieves 5.01 while HoPE achieves 82.04; in Table 3 (QuALITY EM), ALiBi achieves 0.87 while HoPE achieves 0.45. Even in the BiPE comparisons, BiPE-HoPE is worse than BiPE-RoPE at length 1024 in Table 2 (4.83 vs. 4.74). The fine-tuning results in Table 3 show HoPE winning on four of seven tasks, not 'consistently.' The abstract and Section 4.2 overstate the empirical findings.
  3. [Section 3.2.2 and Appendix 8.1] The proof of Positional Discrimination Capacity is incomplete and does not establish the theorem. Equation (20) writes 'kr > B'(θ,t)/B'(θ,r) kt', which is not a well-formed scalar inequality for vectors; the construction appears to modify only k_r, but changing k_r also changes the score for that position, and the proof does not show that the argmax is unique or that the 2D construction carries to the sum over subspaces in Eq. (22). The theorem is essentially asserted rather than proved.
  4. [Abstract and Section 3.1] The claim that 'RoPE is a special case of our generalized formulation' is not substantiated and appears incorrect. The RoPE matrix is an orthogonal rotation (cos θ, sin θ), while the HoPE matrix in Eq. (5) uses cosh θ and sinh θ with a separate damping coefficient e^{-θ'}. Setting θ'=0 does not recover RoPE, because cosh θ and sinh θ do not satisfy cos^2 + sin^2 = 1 and do not preserve norms. No limiting argument or parameter choice is provided to show RoPE as a special case.
  5. [Section 7 and Section 4.4] The Limitations section states that the method's effectiveness 'hinges on careful tuning of the damping coefficient θ'', but the paper provides no ablation results or guidance for choosing θ'. Section 4.4 refers to a 'scaling factor' ablation and Figure 3, but Figure 3 is not described with axis labels or numerical values, and no θ' values are reported anywhere. Since the monotonic-decay property itself depends on θ' > θ_i (and on an unstated masking assumption), this missing hyperparameter analysis is load-bearing, not merely a presentation gap.
minor comments (5)
  1. [Abstract and Section 1] Typo: 'slation issues' should likely be 'oscillation issues' or similar. The abstract also refers to 'Alibi' inconsistently with the body's 'ALiBi'.
  2. [Section 1] References are missing in several places, e.g., '(())' after Llama/Gemini/DeepSeek and after 'numerous efforts have been made'. This needs to be completed.
  3. [Section 4 and Figure 3] Figure 3 is mentioned but no data or axis labels are provided; the ablation description is qualitative and does not report the exact scaling factors or θ' values. The paper should include a table or plot with numeric results.
  4. [Throughout] Inconsistent capitalization: 'Hope' appears in several places (e.g., Section 4.3, Figure 4, Section 4.5) instead of 'HoPE'; 'Bipe' vs. 'BiPE' also inconsistent.
  5. [Appendix 8.4] The Lorentz group background is neither used in the derivation nor connected to the HoPE construction; the phrase 'refer to Masaki Notation' appears to be an incomplete reference.

Circularity Check

1 steps flagged

HoPE's decay guarantee is largely the design goal re-derived from the penalty coefficient inserted into the definition; external benchmarks keep the overall contribution partly independent.

specific steps
  1. self definitional [Section 3.1, Eqs. (10)-(11) and Section 3.2.1, Eq. (12)]
    "To address this issue, we introduce a penalty coefficient e±mθ′ ... Specifically, the penalty ensures that as the positional difference m−n increases, the dot product of q and k decreases, thereby enforcing the intended behaviour of the attention mechanism. ... For dimension-pair (2i,2i+1), consider the asymptotic behavior: lim|m−n|→∞ e−|m−n|θ′ cosh(|m−n|θ_i)∝e−|m−n|(θ′−θ_i). When θ′ > θ_i,∀i, the attention weights exhibit exponential decay concerning positional distance."

    The decay presented as the central theoretical result is inserted by construction: Eqs. (10)-(11) multiply q and k by e^{-mθ'} and e^{mθ'}, so the attention score carries the factor e^{-(m-n)θ'} by definition. Section 3.2.1 then takes the asymptotic limit of this same inserted factor and concludes exponential decay; the conclusion restates the design goal rather than deriving an independent consequence. The only added condition, θ'>θ_i, is an input to Algorithm 1 with no stated value or selection rule, and the paper's Limitations section concedes that performance hinges on tuning θ'. This is partial circularity; the empirical PPL and SCROLLS comparisons are external and not fitted to HoPE's claimed decay.

full rationale

The central 'monotonic decay' claim is not independently derived; it is the intended effect of the e^{±mθ'} penalty that HoPE introduces. Section 3.1 states that the penalty 'ensures' decreasing dot product, and Section 3.2.1's asymptotic analysis only re-expresses that inserted exponential. This is a construction-based, self-definitional element rather than a prediction. However, the paper does not rely on self-citations in a load-bearing way: the related-work self-citations (Song et al.) support prior hyperbolic applications but are not used to justify HoPE's construction. The experimental evidence (perplexity on PG19/arXiv, fine-tuning on SCROLLS, comparisons with BiPE variants) is external and independent, so the paper has substantial non-circular content. Separately, Eq. (12) is asymptotic and does not establish true monotonicity at short distances; that is a correctness gap, not the basis for the circularity score. The Limitations section's admission that θ' must be carefully tuned further confirms that the decay guarantee is conditional on an unstated hyperparameter, lowering the strength of the theoretical claim but not negating the independent empirical evaluation.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 0 invented entities

The method depends on unstated hyperparameters (θ', θ_i, scaling factor), on the locality prior that close tokens should attend more, on the standard Lorentz-group algebra, and on an implicit causal mask so that the exponential penalty is always a decay factor rather than an amplifier.

free parameters (3)
  • θ' (global damping coefficient) = not reported
    Introduced in Eq. (10)-(11) to enforce monotonic decay. The theoretical guarantee in Eq. (12) requires θ' > θ_i for all i, but the paper never states the value or selection rule; the ablation says a scaling factor matters but gives no numbers.
  • θ_i (per-dimension hyperbolic frequencies) = not reported
    Used in the block-diagonal rotations in Eq. (14) and Algorithm 1. No frequency schedule (e.g., base wavelength analogous to RoPE's 10000) is specified in the experiments, so the decay exponent is not reproducible.
  • scaling factor for positional magnitude = not reported
    Section 3.1 Eq. (5) and the ablation in Section 4.4 refer to a scaling factor that adjusts the positional transformation's magnitude, but the main experiments do not state its value.
axioms (3)
  • standard math Lorentz boost matrices preserve the Minkowski spacetime interval and are non-orthogonal transformations.
    Preliminaries Section 2.2; used to justify calling B(θ,m) a hyperbolic rotation, even though the paper applies it to Euclidean query/key coordinates.
  • domain assumption Attention weights should decay with token distance (locality prior).
    Section 3.1: 'this contradicts the assumption of positional encoding, which posits that tokens closer to each other should be assigned higher attention weights.' This prior is assumed, not derived.
  • domain assumption The model is causal/decoder-only, so every attended key has position n ≤ m, making e^{-(m-n)θ'} a decay factor.
    Eq. (10)-(12) use e^{-(m-n)θ'} as a decay; for n > m the factor grows. The paper never states the causal-only qualification, yet all experiments use a decoder-only Transformer (Section 4.2).

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of HoPE: Hyperbolic Rotary Positional Encoding for Stable Long-Range Dependency Modeling in Large Language Models." pith.science (2026). https://pith.science/paper/SMERQVG2

@misc{pith2026250905218,
  author       = {Pith},
  title        = {Pith review of: HoPE: Hyperbolic Rotary Positional Encoding for Stable Long-Range Dependency Modeling in Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SMERQVG2}},
  note         = {Machine review of arXiv:2509.05218}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Positional encoding mechanisms enable Transformers to model sequential structure and long-range dependencies in text. While absolute positional encodings struggle with extrapolation to longer sequences due to fixed positional representations, and relative approaches like Alibi exhibit performance degradation on extremely long contexts, the widely-used Rotary Positional Encoding (RoPE) introduces oscillatory attention patterns that hinder stable long-distance dependency modelling. We address these limitations through a geometric reformulation of positional encoding. Drawing inspiration from Lorentz transformations in hyperbolic geometry, we propose Hyperbolic Rotary Positional Encoding (HoPE), which leverages hyperbolic functions to implement Lorentz rotations on token representations. Theoretical analysis demonstrates that RoPE is a special case of our generalized formulation. HoPE fundamentally resolves RoPE's slation issues by enforcing monotonic decay of attention weights with increasing token distances. Extensive experimental results, including perplexity evaluations under several extended sequence benchmarks, show that HoPE consistently exceeds existing positional encoding methods. These findings underscore HoPE's enhanced capacity for representing and generalizing long-range dependencies. Data and code will be available.

Figures

Figures reproduced from arXiv: 2509.05218 by Chang Dai, Di Liang, Hongyu Shan, Mingyang Song.

Figure 1
Figure 1. Figure 1: Illustration of attention scores. For the same [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Implementation of Hyperbolic Rotary Position Embedding. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Ablation Experiment long-range dependencies within sequences, result￾ing in higher perplexity. Moderate Scaling Factors: Moderate scaling fac￾tors strike a balance by maintaining positional in￾formation while avoiding noise amplification or unnecessary details, typically leading to lower per￾plexity. Larger Scaling Factors: Substantial scaling fac￾tors can amplify noise or other non-ideal character￾istics … view at source ↗
Figure 5
Figure 5. Figure 5: Attention Weight Values under Different Posi [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Attention weight decay trend with rope can derive the following matrix. ρ(gk) =  cosh(gk) sinh(gk) sinh(gk) cosh(gk)  , (25) However, the rotation matrix of RoPE is an orthog￾onal matrix, which means it will not change the modulus of any vector:  cos(θ) − sin(θ) sin(θ) cos(θ)  q1 q2  =  q1 cos(θ) − q2 sin(θ) q1 sin(θ) + q2 cos(θ)  (26) The length after rotation remains unchanged. However, our hyper… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Multi-Granularity Reasoning for Natural Language Inference

    cs.CL 2026-04 conditional novelty 3.5

    Stacking element-wise multi-layer BERT interactions and DenseNet yields modest NLI gains over BERT/RoBERTa baselines on standard benchmarks.

  2. Efficient Task Adaptation in Large Language Models via Selective Parameter Optimization

    cs.CL 2026-04 unverdicted novelty 3.0

    The paper claims a selective fine-tuning method that identifies and freezes core parameters to mitigate catastrophic forgetting in LLMs while improving domain adaptation, shown in experiments with GPT-J and LLaMA-3.

Reference graph

Works this paper leans on

55 extracted references · 21 canonical work pages · cited by 2 Pith papers · 6 internal anchors

  1. [1]

    Aaron B Adcock, Blair D Sullivan, and Michael W Mahoney. 2013. Tree-like structure in large social and information networks. In IEEE International Conference on Data Mining, pages 1--10. IEEE

    work page 2013

  2. [2]

    Joshua Ainslie, Tao Lei, Michiel de Jong, Santiago Ontañón, Siddhartha Brahma, Yury Zemlyanskiy, David Uthus, Mandy Guo, James Lee-Thorp, Yi Tay, Yun-Hsuan Sung, and Sumit Sanghai. 2023. https://arxiv.org/abs/2303.09752 Colt5: Faster long-range transformers with conditional computation . Preprint, arXiv:2303.09752

    Pith/arXiv arXiv 2023

  3. [3]

    Cem Anil, Yuhuai Wu, Anders Andreassen, Aitor Lewkowycz, Vedant Misra, Vinay Ramasesh, Ambrose Slone, Guy Gur-Ari, Ethan Dyer, and Behnam Neyshabur. 2022. https://arxiv.org/abs/2207.04901 Exploring length generalization in large language models . Preprint, arXiv:2207.04901

    Pith/arXiv arXiv 2022

  4. [4]

    Federico Barbero, Alex Vitvitskyi, Christos Perivolaropoulos, Razvan Pascanu, and Petar Veličković. 2024. https://arxiv.org/abs/2410.06205 Round and round we go! what makes rotary positional encodings useful? Preprint, arXiv:2410.06205

    Pith/arXiv arXiv 2024

  5. [5]

    Ines Chami, Zhitao Ying, Christopher R \'e , and Jure Leskovec. 2019. Hyperbolic graph convolutional neural networks. In Advances in Neural Information Processing Systems, pages 4868--4879

    work page 2019

  6. [6]

    Guanzheng Chen, Xin Li, Zaiqiao Meng, Shangsong Liang, and Lidong Bing. 2024. https://arxiv.org/abs/2310.16450 Clex: Continuous length extrapolation for large language models . Preprint, arXiv:2310.16450

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  7. [7]

    Mingda Chen, Zewei Chu, Sam Wiseman, and Kevin Gimpel. 2022. https://arxiv.org/abs/2104.07091 Summscreen: A dataset for abstractive screenplay summarization . Preprint, arXiv:2104.07091

    Pith/arXiv arXiv 2022

  8. [8]

    Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. 2023. https://arxiv.org/abs/2306.15595 Extending context window of large language models via positional interpolation . Preprint, arXiv:2306.15595

    Pith/arXiv arXiv 2023

  9. [9]

    Ta-Chung Chi, Ting-Han Fan, Peter J Ramadge, and Alexander Rudnicky. 2022 a . Kerple: Kernelized relative positional embedding for length extrapolation. Advances in Neural Information Processing Systems, 35:8386--8399

    work page 2022

  10. [10]

    KERPLE: Kernelized Relative Positional Embedding for Length Extrapolation

    Ta-Chung Chi, Ting-Han Fan, Peter J. Ramadge, and Alexander I. Rudnicky. 2022 b . https://arxiv.org/abs/2205.09921 Kerple: Kernelized relative positional embedding for length extrapolation . Preprint, arXiv:2205.09921

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  11. [11]

    Ta-Chung Chi, Ting-Han Fan, Peter J Ramadge, et al. 2023. Dissecting transformer length extrapolation via the lens of receptive field analysis. In The 61st Annual Meeting Of The Association For Computational Linguistics

    work page 2023

  12. [12]

    Jishnu Ray Chowdhury and Cornelia Caragea. 2023. https://arxiv.org/abs/2305.20019 Monotonic location attention for length generalization . Preprint, arXiv:2305.20019

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  13. [13]

    Smith, and Matt Gardner

    Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. 2021. https://arxiv.org/abs/2105.03011 A dataset of information-seeking questions and answers anchored in research papers . Preprint, arXiv:2105.03011

    Pith/arXiv arXiv 2021

  14. [14]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://arxiv.org/abs/1810.04805 Bert: Pre-training of deep bidirectional transformers for language understanding . Preprint, arXiv:1810.04805

    Pith/arXiv arXiv 2019

  15. [15]

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. https://arxiv.org/abs/2101.00027 The pile: An 800gb dataset of diverse text for language modeling . Preprint, arXiv:2101.00027

    Pith/arXiv arXiv 2020

  16. [16]

    Olga Golovneva, Tianlu Wang, Jason Weston, and Sainbayar Sukhbaatar. 2024. https://arxiv.org/abs/2405.18719 Contextual position encoding: Learning to count what's important . Preprint, arXiv:2405.18719

    Pith/arXiv arXiv 2024

  17. [17]

    Brian C. Hall. 2000. https://arxiv.org/abs/math-ph/0005032 An elementary introduction to groups and representations . Preprint, arXiv:math-ph/0005032

    work page internal anchor Pith review Pith/arXiv arXiv 2000

  18. [18]

    Adi Haviv, Ori Ram, Ofir Press, Peter Izsak, and Omer Levy. 2022. https://arxiv.org/abs/2203.16634 Transformer language models without positional encodings still learn positional information . Preprint, arXiv:2203.16634

    Pith/arXiv arXiv 2022

  19. [19]

    Zhenyu He, Guhao Feng, Shengjie Luo, Kai Yang, Liwei Wang, Jingjing Xu, Zhi Zhang, Hongxia Yang, and Di He. 2024. https://arxiv.org/abs/2401.16421 Two stones hit one bird: Bilevel positional encoding for better length extrapolation . Preprint, arXiv:2401.16421

    Pith/arXiv arXiv 2024

  20. [20]

    Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. 2021. https://arxiv.org/abs/2104.02112 Efficient attentions for long document summarization . Preprint, arXiv:2104.02112

    Pith/arXiv arXiv 2021

  21. [21]

    Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. 2023. https://arxiv.org/abs/2305.19466 The impact of positional encoding on length generalization in transformers . Preprint, arXiv:2305.19466

    Pith/arXiv arXiv 2023

  22. [22]

    Yuta Koreeda and Christopher D. Manning. 2021. https://arxiv.org/abs/2110.01799 Contractnli: A dataset for document-level natural language inference for contracts . Preprint, arXiv:2110.01799

    Pith/arXiv arXiv 2021

  23. [23]

    Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2017. https://arxiv.org/abs/1712.07040 The narrativeqa reading comprehension challenge . Preprint, arXiv:1712.07040

    Pith/arXiv arXiv 2017

  24. [24]

    Dmitri Krioukov, Fragkiskos Papadopoulos, Maksim Kitsak, Amin Vahdat, and Mari \'a n Bogun \'a . 2010. Hyperbolic geometry of complex networks. Physical Review E, 82(3):036106

    work page 2010

  25. [25]

    Shanda Li, Chong You, Guru Guruganesh, Joshua Ainslie, Santiago Ontanon, Manzil Zaheer, Sumit Sanghai, Yiming Yang, Sanjiv Kumar, and Srinadh Bhojanapalli. 2024. https://arxiv.org/abs/2310.04418 Functional interpolation for relative positions improves long context transformers . Preprint, arXiv:2310.04418

    Pith/arXiv arXiv 2024

  26. [26]

    Pascal Mettes, Mina Ghadimi Atigh, Martin Keller-Ressel, Jeffrey Gu, and Serena Yeung. 2023. Hyperbolic deep learning in computer vision: A survey. arXiv preprint arXiv:2305.06611

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  27. [27]

    Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He, and Samuel R. Bowman. 2022. https://arxiv.org/abs/2112.08608 Quality: Question answering with long input texts, yes! Preprint, arXiv:2112.08608

    Pith/arXiv arXiv 2022

  28. [28]

    Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2023. https://arxiv.org/abs/2309.00071 Yarn: Efficient context window extension of large language models . Preprint, arXiv:2309.00071

    Pith/arXiv arXiv 2023

  29. [29]

    Wei Peng, Tuomas Varanka, Abdelrahman Mostafa, Henglin Shi, and Guoying Zhao. 2021. Hyperbolic deep neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence

    work page 2021

  30. [30]

    Smith, and Mike Lewis

    Ofir Press, Noah A. Smith, and Mike Lewis. 2022. https://arxiv.org/abs/2108.12409 Train short, test long: Attention with linear biases enables input length extrapolation . Preprint, arXiv:2108.12409

    Pith/arXiv arXiv 2022

  31. [31]

    Rae, Anna Potapenko, Siddhant M

    Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, and Timothy P. Lillicrap. 2019. https://arxiv.org/abs/1911.05507 Compressive transformers for long-range sequence modelling . Preprint, arXiv:1911.05507

    Pith/arXiv arXiv 2019

  32. [32]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2023. https://arxiv.org/abs/1910.10683 Exploring the limits of transfer learning with a unified text-to-text transformer . Preprint, arXiv:1910.10683

    Pith/arXiv arXiv 2023

  33. [33]

    Anian Ruoss, Grégoire Delétang, Tim Genewein, Jordi Grau-Moya, Róbert Csordás, Mehdi Bennani, Shane Legg, and Joel Veness. 2023. https://arxiv.org/abs/2305.16843 Randomized positional encodings boost length generalization of transformers . Preprint, arXiv:2305.16843

    Pith/arXiv arXiv 2023

  34. [34]

    Rik Sarkar. 2011. Low distortion delaunay embedding of trees in hyperbolic plane. In International Symposium on Graph Drawing, pages 355--366. Springer

    work page 2011

  35. [35]

    Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, and Omer Levy. 2022. https://arxiv.org/abs/2201.03533 Scrolls: Standardized comparison over long language sequences . Preprint, arXiv:2201.03533

    Pith/arXiv arXiv 2022

  36. [36]

    Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. https://arxiv.org/abs/1803.02155 Self-attention with relative position representations . Preprint, arXiv:1803.02155

    Pith/arXiv arXiv 2018

  37. [37]

    Mingyang Song, Yi Feng, and Liping Jing. 2022 a . https://doi.org/10.18653/V1/2022.NAACL-MAIN.419 Hyperbolic relevance matching for neural keyphrase extraction . In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-...

    work page doi:10.18653/v1/2022.naacl-main.419 2022

  38. [38]

    Mingyang Song, Yi Feng, and Liping Jing. 2022 b . https://doi.org/10.1145/3511808.3557538 A preliminary exploration of extractive multi-document summarization in hyperbolic space . In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, Atlanta, GA, USA, October 17-21, 2022 , pages 4505--4509. ACM

    work page arXiv 2022

  39. [39]

    Mingyang Song, Yi Feng, and Liping Jing. 2023 a . https://doi.org/10.1145/3543507.3583197 Hisum: Hyperbolic interaction model for extractive multi-document summarization . In Proceedings of the ACM Web Conference 2023, WWW 2023, Austin, TX, USA, 30 April 2023 - 4 May 2023 , pages 1427--1436. ACM

    work page arXiv 2023

  40. [40]

    Mingyang Song, Huafeng Liu, Yi Feng, and Liping Jing. 2023 b . https://doi.org/10.18653/V1/2023.FINDINGS-ACL.66 Improving embedding-based unsupervised keyphrase extraction by incorporating structural information . In Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023 , pages 1041--1048. Association for Co...

    work page doi:10.18653/v1/2023.findings-acl.66 2023

  41. [41]

    Mingyang Song, Huafeng Liu, and Liping Jing. 2023 c . https://doi.org/10.18653/V1/2023.EMNLP-MAIN.997 Hyperrank: Hyperbolic ranking model for unsupervised keyphrase extraction . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023 , pages 16070--16080. Association for Computa...

    work page doi:10.18653/v1/2023.emnlp-main.997 2023

  42. [42]

    Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. 2023. https://arxiv.org/abs/2104.09864 Roformer: Enhanced transformer with rotary position embedding . Preprint, arXiv:2104.09864

    Pith/arXiv arXiv 2023

  43. [43]

    Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, and Furu Wei. 2022. https://arxiv.org/abs/2212.10554 A length-extrapolatable transformer . Preprint, arXiv:2212.10554

    Pith/arXiv arXiv 2022

  44. [44]

    Atsushi Suzuki, Atsushi Nitanda, Jing Wang, Linchuan Xu, Kenji Yamanishi, and Marc Cavazza. 2021 a . Generalization error bound for hyperbolic ordinal embedding. In International Conference on Machine Learning, pages 10011--10021. PMLR

    work page 2021

  45. [45]

    Atsushi Suzuki, Atsushi Nitanda, Linchuan Xu, Kenji Yamanishi, Marc Cavazza, et al. 2021 b . Generalization bounds for graph embedding using negative sampling: Linear vs hyperbolic. Advances in Neural Information Processing Systems, 34:1243--1255

    work page 2021

  46. [46]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Harts...

    Pith/arXiv arXiv 2023

  47. [47]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2023. https://arxiv.org/abs/1706.03762 Attention is all you need . Preprint, arXiv:1706.03762

    Pith/arXiv arXiv 2023

  48. [48]

    Yu-An Wang and Yun-Nung Chen. 2020. https://arxiv.org/abs/2010.04903 What do position embeddings learn? an empirical study of pre-trained language model positional encoding . Preprint, arXiv:2010.04903

    Pith/arXiv arXiv 2020

  49. [49]

    Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang, and Hao Ma. 2023. https://arxiv.org/abs/2309.16039 Effective long-co...

    Pith/arXiv arXiv 2023

  50. [50]

    Haoran Yang, Hongxu Chen, Lin Li, Philip S Yu, and Guandong Xu. 2021. Hyper meta-path contrastive learning for multi-behavior recommendation. arXiv preprint arXiv:2109.02859

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  51. [51]

    Menglin Yang, Zhihao Li, Min Zhou, Jiahong Liu, and Irwin King. 2022. HICF : Hyperbolic informative collaborative filtering. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 2212--2221

    work page 2022

  52. [52]

    Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev. 2021. https://arxiv.org/abs/2104.05938 Qmsum: A new benchmark for query-based multi-domain meeting summarization . Preprint, arXiv:2104.05938

    Pith/arXiv arXiv 2021

  53. [53]

    Min Zhou, Bisheng Li, Menglin Yang, and Lujia Pan. 2022. Telegraph: A benchmark dataset for hierarchical link prediction. arXiv preprint arXiv:2204.07703

    Pith/arXiv arXiv 2022

  54. [54]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  55. [55]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.