Pith. sign in

REVIEW 2 major objections 4 minor 31 references

Standard RoPE applies one frequency schedule and attention scaling to every head; AdaRoPE claims each head needs its own learnable frequencies and scaling, and shows consistent gains in pretraining quality and 8k-to-64k context extension.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 12:12 UTC pith:WKP4WYQR

load-bearing objection Useful drop-in RoPE variant with real pretraining evidence; the extrapolation claim needs one tuned-baseline control before the head-wise story holds. the 2 major comments →

arxiv 2607.19363 v2 pith:WKP4WYQR submitted 2026-06-05 cs.AI cs.CL

AdaRoPE: Not All Attention Heads Should Rotate and Scale Equally

Shaowen Wang , Yuke Zheng , Tansheng Zhu , Shuang Chen , Shaofan Liu , Suncong Zheng , Jian Li This is my paper
classification cs.AI cs.CL
keywords rotary position embeddinghead-wise frequenciesattention scalingcontext extensionlength generalizationeffective sequence lengthretrieval headslearnable position parameters
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that the uniform frequency schedule and attention scaling of standard rotary position embedding (RoPE) is fundamentally misaligned with how attention heads actually behave: heads differ in effective attention span (some local, some global) and in which rotation frequencies they rely on. To fix this, it introduces AdaRoPE, which gives each head its own learnable rotation frequencies and a per-head, length-dependent attention temperature. Using simplified retrieval tasks and probabilistic analysis, the paper derives why head heterogeneity exists — retrieval heads need constant effective sequence length, aggregation heads need it to grow linearly — and why uniform scaling cannot satisfy both. Empirically, models pretrained with AdaRoPE consistently outperform uniform RoPE, partial RoPE, ALiBi, and no-position baselines across scales up to 2.7B, and context extension from 8k to 64k improves by tuning only tens of thousands of positional parameters with the backbone frozen. A sympathetic reader should care because AdaRoPE is a drop-in replacement for RoPE and, if the head-wise structure is the real cause of the gains, makes long-context adaptation remarkably cheap.

Core claim

On the paper's own terms, the central discovery is that attention heads in RoPE-based transformers are spectrally heterogeneous, and a uniform frequency schedule wastes embedding dimensions: heads with small target windows need high rotation frequencies, heads with large windows need low frequencies, and a single global schedule forces most heads into the wrong band (Theorem 1). Likewise, length extension requires head-specific attention scaling: retrieval heads must keep their effective sequence length constant to preserve sharp focus, while global aggregation heads must let effective length grow linearly with context (Theorems 2–3), so any single length-dependent temperature either over-sh

What carries the argument

The two trainable components are AdaFreq, per-head per-dimension rotation frequencies θ_f^(h)=exp(ξ_f^(h)) replacing the fixed geometric schedule b^{-2f/d}, and AdaScale, a per-head length-dependent inverse temperature λ^(h)(L) = (1/τ^(h))[ln(1 + max{L,Lref}/Lref)]^{γ^(h)} applied to the query before attention. The theoretical core is a frequency-mass bound (Theorem 1) — the weighted average frequency M must lie roughly between C/(rW) and π/(W+1) for a window-retrieval task to be feasible — and an effective-sequence-length analysis using the generalized effective sample size E_β(α): retrieval heads need E_β constant in L, aggregation heads need E_β = Θ(L), and Theorems 4–5 derive the form of

Load-bearing premise

The load-bearing premise is that the gains come from head-wise structure itself, not from the simple act of fine-tuning positional parameters: the extrapolation experiments compare AdaRoPE against a YaRN baseline that is applied without any tuning, so a tuned global schedule on the same 100 samples could plausibly capture much of the improvement.

What would settle it

Run the same frozen-backbone extrapolation protocol but fine-tune a single global (shared) YaRN-style frequency and temperature schedule — the same first 100 PG19 samples, same optimizer, same parameter budget — and evaluate RULER at 64k on Llama-3-8B; if a tuned global schedule matches AdaRoPE's roughly 52 accuracy, the head-wise mechanism claim collapses to 'fine-tuning positional parameters helps.' A second check: fine-tune YaRN's global schedule, then continue with low-rank adapters, mirroring the paper's two-stage recipe, and compare against AdaRoPE→Joint.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If AdaRoPE is right, any RoPE-pretrained model can be extended to 64k context by fine-tuning only roughly 10–50K positional parameters with the backbone frozen, outperforming YaRN at all evaluated lengths while mostly preserving short-context accuracy.
  • Pretraining with head-wise learnable frequencies and scaling becomes a directly better drop-in default than uniform RoPE across model scales up to 2.7B parameters, with no architectural change.
  • The theoretical bound linking retrieval window W to feasible frequency mass implies that fixed frequency schedules are not just suboptimal but provably capacity-limited: they leave embedding dimensions unused for the window the head needs.
  • The two-stage recipe — warm up AdaRoPE on 100 samples, then jointly train low-rank adapters — is a concrete, transferable protocol for long-context continued pretraining.
  • The E_β(α) analysis gives a quantitative diagnostic — measuring each head's effective length — for deciding which heads need sharper versus broader attention during context extension.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: a testable extension the paper leaves implicit is to estimate each head's target effective length E_β from a few long-context samples and set the temperature schedule analytically from the paper's Theorem 4, removing the need to learn γ^(h) and τ^(h).
  • Editorial inference: an untested corollary of the frequency-mass bound is that head-wise learnable frequencies could also help non-uniform retrieval (e.g., mid-context 'lost in the middle') because each head could specialize its own window; combining AdaFreq with position-dependent routing is a natural next step.
  • Editorial inference: the 100-sample fine-tuning result raises a question the paper does not settle — how much of the context-extension gain is generic positional re-adaptation versus head-specific specialization; a tuned global baseline would decide whether the per-head structure is load-bearing, and that control is missing from the experiments.
  • Editorial inference: if the heavy-tailed head-influence results generalize, head-level pruning or quantization should preserve heads with extreme learned frequencies, since the ablations show a small minority of retrieval-like heads are indispensable for long-context behavior.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper argues that standard RoPE's uniform frequency schedule and uniform attention scaling across heads are suboptimal, because heads with different functional roles require different frequency bands and length-dependent scaling. It proposes AdaRoPE, consisting of AdaFreq (per-head, per-dimension learnable log-frequencies) and AdaScale (per-head, length-aware inverse temperatures of the form lambda(h)(L)=(1/tau(h))[ln(1+max{L,L_ref}/L_ref)]^{gamma(h)}). The authors support the proposal with (i) theoretical analyses of simplified retrieval and aggregation tasks (Theorems 1-5, proofs in Appendix F), (ii) pretraining experiments on four architectures up to 2.7B parameters comparing AdaRoPE against RoPE, PRoPE, ALiBi, and NoPE (Table 1), and (iii) context-extension experiments on SmolLM-2-1.7B and Llama-3-8B extending 8k to 64k via extrapolation (Table 2) and continued pretraining with LoRA (Table 3). The headline claims are that AdaRoPE consistently outperforms RoPE variants in pretraining and enables better context extension while preserving short-context performance.

Significance. If the claims are correct, AdaRoPE is a lightweight, plug-and-play replacement for RoPE that improves both pretraining and long-context extrapolation, adding only ~10-50K parameters. The paper includes formal theorems with proofs, a wide range of backbones, and internally consistent ablations. The main risk is that the context-extension comparisons are confounded by training protocol, so the central mechanism claim ('head-wise is why it works') is not yet established. The pretraining results are more solid but would benefit from variance reporting. Overall, the contribution is potentially significant for positional-embedding design, but the current evidence does not fully support the strongest claims.

major comments (2)
  1. [§5.2, Table 2] The extrapolation comparison is confounded. AdaRoPE is fine-tuned on the first 100 PG19 samples with the backbone frozen, while Backbone+YaRN is applied without any tuning. Consequently, the reported gains may reflect the mere act of fitting 10-50K positional parameters to 100 samples rather than the head-wise structure. The ablations (w/o AdaFreq, w/o AdaScale, Share Freq.) all tune a head-specific or partially head-specific set of parameters, so they do not control for tuned global alternatives. A control that tunes a global (shared across heads) frequency schedule and scaling on the same 100 samples is necessary to attribute the gains to head-specificity. Without it, the central mechanism claim collapses to 'fine-tuning position parameters helps'.
  2. [§5.3, Table 3] The two-stage benefit is confounded with warm-up. AdaRoPE→LoRA and AdaRoPE→Joint both include a 100-sample warm-up of AdaRoPE parameters before LoRA training, whereas YaRN→LoRA uses fixed YaRN without any warm-up. A control that performs a 100-sample warm-up of a global (non-head-wise) YaRN-style schedule before LoRA is missing. This is needed to separate the effect of positional warm-up from the effect of head-wise adaptation. As reported, the table cannot distinguish between 'AdaRoPE's head-wise structure helps' and 'any fine-tuning of positional parameters helps'.
minor comments (4)
  1. [Tables 1-3] No seeds or error bars are reported. Some per-benchmark entries are negative versus RoPE (e.g., LLaMA 430M Lambada and WinoGrande, LLaMA 1.3B PIQA), so the 'consistently outperforms' claim rests on average improvements. Reporting standard deviations over at least 3 seeds would strengthen the pretraining claim.
  2. [§3.3, Theorems 4-5] The stated motivation for the AdaScale log-polynomial form lambda = (1/tau)[ln(1+L/L_ref)]^gamma is not a direct derivation: Theorem 4(i) yields lambda ~ sqrt(log L) for constant E*, and Theorems 4-5 leave the head-specific E* unspecified. The chosen parameterization is a reasonable flexible family, but the text overstates how tightly the theorems constrain it. Please clarify the sense in which the theorems motivate this form.
  3. [Equation (12)] The upper bound in Theorem 1 contains an indicator function of theta_max > pi/(W+1). This is unusual for a bound; please make explicit whether the indicator is a constant or a function of W, and how C2 depends on it.
  4. [General] Minor typographical issues: Eq. (33) has an exponent '2^{1/max{beta,beta-1}-1}' that appears to be a formatting artifact; in Theorem 2, 'then' in the statement should be 'then'. Also, several appendix references (e.g., 'Section D', 'Section F.1') could use hyperlinks for readability.

Circularity Check

0 steps flagged

No significant circularity; the empirical claims rest on external benchmarks and the theoretical schedule leaves its free constants explicitly learnable.

full rationale

The derivation chain is self-contained. AdaFreq and AdaScale are new parameterizations evaluated against independent baselines (RoPE, PRoPE, ALiBi, NoPE, YaRN) on held-out NLU and RULER benchmarks, so the pretraining and context-extension gains are not definitional. The theoretical results (Theorems 1–5) derive frequency-mass bounds and scaling relations for toy retrieval/aggregation tasks rather than assuming the head-wise conclusion; the move to per-head parameters is additionally supported by direct head-level measurements. Remark 6 candidly states that the head-specific β and Ê_β(L) forms are unknown a priori and are absorbed into the learnable τ(h) and γ(h), meaning AdaScale is a fitted parametric family motivated by the theorems, not a predicted constant masquerading as a first-principles output—this weakens the theory-verification narrative but does not constitute a by-construction reduction. The main validity concern, that Table 2 compares fine-tuned AdaRoPE with untuned YaRN and Table 3 lacks a YaRN-with-warm-up control, is an experimental-fairness issue about attributing gains to head-wise structure; it is not a case where the fitted parameters and the reported prediction coincide by construction. No load-bearing self-citation chain appears: the only overlapping-author reference (Li et al., 2025a, HARoPE) is cited as related work and is not needed for any theorem or benchmark claim.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

No invented physical entities. The method adds trainable scalars (xi, gamma, tau) and reuses the importance-sampling effective-sample-size metric E_beta (Huggins & Roy 2019; Martino et al. 2017) as 'effective sequence length'—a defined statistic without an independent falsifiable handle. The main ledger concern is the free-parameter load: all of AdaFreq's xi and AdaScale's (gamma,tau) are fitted, and the theory only motivates their existence rather than constraining their values, so the paper's contribution is the learnable parameterization plus the empirical demonstration, not a parameter-free prediction.

free parameters (4)
  • xi(h)_f (AdaFreq log-frequencies) = learned; not tabulated—visualized as effective base Theta and per-layer spectra (Figs. 3, 9)
    Per-head, per-dimension-pair log rotation frequencies; the core fitted quantity of AdaFreq (~H·d/2 numbers).
  • gamma(h) (AdaScale growth exponent) = learned; observed mostly ~0.1-0.37, init 0 (Fig. 3 Right, Fig. 11)
    Controls how fast head h's inverse temperature grows with context length; fitted to downstream loss/RULER, not derived from Theorems 4-5 (Remark 6).
  • tau(h) (AdaScale base temperature) = learned; init 1, distribution centered near 1 with a right tail (Fig. 11 Middle)
    Global per-head logit scale; fitted.
  • L_ref (AdaScale reference length) = 64 (pretraining), 8192 (context extension)
    Hand-set hyperparameter defining the knee of the polylog schedule; chosen as the model's native context length in extension.
axioms (5)
  • standard math Standard Fourier/analytic tools: Dirichlet-kernel bounds, Lipschitz continuity, Riemann-integrability facts, CLT/Hanson-Wright/Borel-Cantelli/Mills-ratio/von Bahr-Esseen inequalities
    Used throughout proofs F.1-F.7; unproved background results.
  • domain assumption Correlated Gaussian query-key logit model: q ~ N(0,I_d), k_i = (rho/sqrt(d))q + sigma z_i with z_i i.i.d. N(0,I_d)
    Postulated in Section E.4 as the infinite-width model under which Propositions 1-2 and Theorems 4-5 are proved; the paper says it 'captures effects beyond the initialization regime' but provides no validation that real trained logits obey it.
  • domain assumption Real LLM heads can be mapped to the retrieval/aggregation dichotomy with well-defined window W and effective length E_beta
    Bridges the single-layer toy theory (NRW, Section D; E.2-E.3) to the claim that heads in Llama-3-8B 'require' different frequencies/scaling (Sections 3.1-3.2, Figure 2). The measured heterogeneity is real, but its causal role is assumed, not shown.
  • ad hoc to paper The functional form lambda(h)(L) = (1/tau)[ln(1+max{L,L_ref}/L_ref)]^gamma contains the per-head optimal schedule
    Introduced in Section 3.3. Theorems 4-5 yield implicit lambda(L) involving E*_beta and O(e^{rho lambda}) corrections that this family cannot represent; Remark 6 states the unknown beta, E-hat_beta 'degrees of freedom' are absorbed into fitted tau,gamma. The form is a heuristic, not a derivation.
  • domain assumption 100 PG19 samples suffice to fit head-wise positional parameters that transfer to RULER-64k
    Section 5.2's headline extrapolation result assumes optimization of xi,gamma,tau on the first 100 samples of one book corpus generalizes to synthetic retrieval at 64k; overfitting to that tiny set is a real risk and no validation curve is shown.

pith-pipeline@v1.3.0-alltime-deepseek · 47770 in / 28024 out tokens · 269900 ms · 2026-08-02T12:12:02.880861+00:00 · methodology

0 comments
read the original abstract

Rotary Position Embedding (RoPE) is widely adopted in Transformers to encode positional information, yet standard implementations enforce a uniform frequency schedule and scaling across all attention heads. Using simplified retrieval tasks and length generalization scenarios, we show -- both empirically and theoretically -- that heads with different functional roles require distinct frequency ranges and attention scaling factors to operate effectively. Ignoring this structure leads to suboptimal utilization of embedding dimensions and degraded performance, particularly under long-context settings. To address these limitations, we propose AdaRoPE, which equips each attention head with learnable rotation frequencies and attention scaling factors. Pretrained LLMs with AdaRoPE consistently outperform existing RoPE variants, including partial RoPE and NoPE baselines. For context extension, we further show that uniform frequency and attention scaling, used in methods such as YaRN, are suboptimal. By applying head-specific scaling, AdaRoPE enables better context extension while better preserving short-context performance in both the extrapolation setting and the long-context continued pretraining setting. These results highlight the importance of optimizing rotary position embedding at the level of individual attention heads.

Figures

Figures reproduced from arXiv: 2607.19363 by Jian Li, Shaofan Liu, Shaowen Wang, Shuang Chen, Suncong Zheng, Tansheng Zhu, Yuke Zheng.

Figure 1
Figure 1. Figure 1: Window-dependent sparse utilization of RoPE frequencies. We analysis per-dimension-pair contributions to the attention logit between a query q and a key k at different relative distances ∆, where color indicates P i qiki for each RoPE dimension pair. Darker red or blue denotes larger-magnitude logit contributions. It clearly exhibits distinct periodic patterns across different relative distances. We plot t… view at source ↗
Figure 2
Figure 2. Figure 2: Head-wise RoPE utilization and effective length in LLaMA-3-8B. Each subfigure shows one representative head with effective length (E2; left panel) and RoPE dimension-pair utilization (right panel), measured on 1,024 FineWeb-Edu samples. a: E2 grows with context length and the frequency usage is relatively uniform. b: E2 stays nearly constant across context lengths and usage is highly sparse/spiky. Together… view at source ↗
Figure 3
Figure 3. Figure 3: Left: AdaRoPE yields a lower coefficient of variation (CoV) across dimensions compared to RoPE and PRoPE, indicating a more balanced utilization of frequencies. The box plot represents the min, max, 25th and 75th percentiles, and the mean across all heads. Middle: The effective RoPE base Θ exhibits an increasing trend with network depth, suggesting that deeper layers prioritize learning lower frequencies. … view at source ↗
Figure 4
Figure 4. Figure 4: Analysis of extrapolation performance and learned frequency patterns. Left: NLL (negative log-likelihood) versus token position comparing AdaRoPE and YaRN during the extrapolation stage. Middle: NLL comparison during the long-context continued pretraining stage (AdaRoPE vs. YaRN+LoRA). Right: Visualization of log-frequency modifications (difference between the YaRN and learned log inverse frequency) across… view at source ↗
Figure 5
Figure 5. Figure 5: displays the attention logit contributions from different RoPE frequency pairs as the target window size W varies. (a) W = 1 (b) W = 32 (c) W = 128 (d) W = 512 [PITH_FULL_IMAGE:figures/full_fig_p015_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Effective Length Specialization. Left: AdaRoPE enables a wider distribution of effective lengths across heads compared to RoPE. Middle: Distribution of learned γ (h) values, which control the E2 scaling. Right: Heads that learn a larger γ (h) value maintain a smaller mean effective length [PITH_FULL_IMAGE:figures/full_fig_p016_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Per-head effective length scaling. Left: A head with a large learned γ (h) = 0.371 (Layer 7, Head 1) maintains a small and constant E2. Right: A head with a small learned γ (h) = 0.019 (Layer 3, Head 7) allows its E2 to grow with position, acting as a global head. B.2. Analysis on Pretraining AdaScale enables diverse attention spans. As context length grows, attention can dilute, as the same query attends … view at source ↗
Figure 8
Figure 8. Figure 8: AdaRoPE enables frequency specialization. Left: The distribution of weighted-average frequencies across heads is broader for AdaRoPE than RoPE. Right: High correlation (r=0.935) between the learned mean log-frequency and the utilized weighted mean log-frequency in AdaRoPE. B.3. Analysis on Extrapolation Setting. We conduct a head-level analysis on three SmolLM2-135M variants: the original backbone, YaRN, a… view at source ↗
Figure 9
Figure 9. Figure 9: Learned spectral adaptation in AdaRoPE. Top plots show log-frequency; bottom plots show mean absolute dot product. Left: A high-frequency head (Layer 2, Head 2) learns to use even higher frequencies (orange) than standard RoPE (blue). Right: A low-frequency head (Layer 4, Head 3) learns to use even lower frequencies [PITH_FULL_IMAGE:figures/full_fig_p018_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: AdaFreq ablations. Left: Histogram of head influence ∆h when reverting AdaFreq parameters one head at a time; most heads have negligible effect, but a small set forms a heavy tail that includes the top retrieval head. Right: Learned RoPE spectrum for the most influential head. AdaRoPE reshapes its frequencies non-uniformly, emphasizing mid–high-frequency bands beyond what global schemes like YaRN prescrib… view at source ↗
Figure 11
Figure 11. Figure 11: AdaScale ablations. Left: Histogram of head influence ∆h when reverting AdaScale parameters (γ (h) , τ (h) ) for one head at a time, revealing a sparse set of critical temperature heads. Middle: Distribution of learned base temperatures τ (h) , centered near 1 with a right tail corresponding to heads that sharpen more aggressively with context length. Right: Distribution of learned growth coefficients γ (… view at source ↗
Figure 12
Figure 12. Figure 12: Effective attention length per head. Each point is an attention head; the horizontal axis is log(E base 2 ) and the vertical axis is log(E method 2 ), with the red dashed line indicating y = x. Left: YaRN without scaling; short-range heads are preserved but long-range heads are not sufficiently extended. Middle: YaRN with a global scale; long-range heads are better matched but many short-range heads are d… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

31 extracted references · 14 linked inside Pith

  1. [1]

    At the window boundary, the Lipschitz property together with Eqs

    Lower bound ofM.Note that |T ′(x)|= X f (−af θf ) sin(θf x+φ f ) ≤ X f af θf =M, 29 AdaRoPE: Not All Attention Heads Should Rotate and Scale Equally soTisM-Lipschitz. At the window boundary, the Lipschitz property together with Eqs. (11) and (42) implies l r 2 W m M≥T c− W 2 −T c− (1 +r) W 2 ≥T L c− W 2 −T U c− (1 +r) W 2 ≥T ∗ L −T ∗ U = 2 A⋆ ln 1−ε ε − ∥...

  2. [2]

    Upper bound ofM.Define the average ofT(x)over[c−W/2, c+W/2]as (AW T)(c) = 1 W+ 1 c+W/2X t=c−W/2 T(t). Then Eq. (11) implies (AW TL)(c)≤(A W T)(c).(44) By Lagrange’s trigonometric identity, (AW T)(c) = 1 W+ 1 c+W/2X t=c−W/2 d/2−1X f=0 af cos(θf t+φ f ) = 1 W+ 1 d/2−1X f=0 af W/2X k=−W/2 cos(θf (c+k) +φ f ) = 1 W+ 1 d/2−1X f=0 af W/2X k=−W/2 h cos(θf k) cos...

  3. [3]

    By the continuity ofx(t),tis an interior point ofI, soIis open

    Openness ofI.For anyt∈I, substitutingx(t)≤2tAintox≤A+Bx γ gives x(t) =tA+B(x(t)) γ ≤tA+B(2tA) γ =tA 1 +B2 γtγ−1Aγ−1 ≤tA 1 +γB2 γAγ−1 <2tA, where the second inequality uses γ >1 and t∈[0,1] , and the last inequality follows from 0< A <(γB2γ) 1 1−γ . By the continuity ofx(t),tis an interior point ofI, soIis open. 4.Since I is nonempty, open and closed in th...

    2026

  4. [4]

    org/CorpusID:258840844

    URL https://api.semanticscholar. org/CorpusID:258840844. Chiang, D. and Cholak, P. A. Overcoming a theo- retical limitation of self-attention. InAnnual Meet- ing of the Association for Computational Linguistics,

  5. [6]

    org/CorpusID:268357635

    URL https://api.semanticscholar. org/CorpusID:268357635. Haviv, A., Ram, O., Press, O., Izsak, P., and Levy, O. Trans- former language models without positional encodings still learn positional information.ArXiv, abs/2203.16634,

    Pith/arXiv arXiv

  6. [7]

    org/CorpusID:247839823

    URL https://api.semanticscholar. org/CorpusID:247839823. Heo, B., Park, S., Han, D., and Yun, S. Rotary position embedding for vision transformer. InEuropean Conference on Computer Vision, 2024. URL https: //api.semanticscholar.org/CorpusID: 268536717. Hsieh, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., and Ginsburg, B. Ruler: What’s the ...

    Pith/arXiv arXiv 2024

  7. [10]

    org/CorpusID:252532078

    URL https://api.semanticscholar. org/CorpusID:252532078. Oppenheim, A. V ., Willsky, A. S., and Nawab, S. H.Signals & Systems. Pearson Educaci ´on, 2 edition, 1997. Ostmeier, S., Axelrod, B., Varma, M., Moseley, M. E., Chaudhari, A. S., and Langlotz, C. P. Liere: Lie rotational positional encodings. InInternational Conference on Machine Learning, 2024. UR...

    Pith/arXiv arXiv 1997

  8. [12]

    org/CorpusID:207930593

    URL https://api.semanticscholar. org/CorpusID:207930593. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y ., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer.J. Mach. Learn. Res., 21:140:1–140:67,

  9. [13]

    org/CorpusID:204838007

    URL https://api.semanticscholar. org/CorpusID:204838007. R´enyi, A. On measures of entropy and information. InPro- ceedings of the fourth Berkeley symposium on mathemati- cal statistics and probability, volume 1: contributions to the theory of statistics, volume 4, pp. 547–562. University of California Press, 1961. Sakaguchi, K., Bras, R. L., Bhagavatula,...

    1961

  10. [14]

    org/CorpusID:199370376

    URL https://api.semanticscholar. org/CorpusID:199370376. Shang, N., Zhang, L. L., Wang, S., Zhang, G., Lopez, G., Yang, F., Chen, W., and Yang, M. Lon- grope2: Near-lossless llm context window scal- ing.ArXiv, abs/2502.20082, 2025. URL https: //api.semanticscholar.org/CorpusID: 276647831. Shaw, P., Uszkoreit, J., and Vaswani, A. Self-attention with relati...

    Pith/arXiv arXiv 2025

  11. [16]

    org/CorpusID:269330144

    URL https://api.semanticscholar. org/CorpusID:269330144. Xiong, W., Liu, J., Molybog, I., Zhang, H., Bhargava, P., Hou, R., Martin, L., Rungta, R., Sankararaman, K. A., Oguz, B., et al. Effective long-context scaling of founda- tion models. InProceedings of the 2024 Conference of the North American Chapter of the Association for Com- putational Linguistic...

    Pith/arXiv arXiv 2024

  12. [17]

    org/CorpusID:159041722

    URL https://api.semanticscholar. org/CorpusID:159041722. Zhang, X., Chang, X., Li, M., Roy-Chowdhury, A., Chen, J., and Oymak, S. Selective attention: Enhancing trans- former through principled context control.Advances in Neural Information Processing Systems, 37:11061– 11086, 2024a. Zhang, Y ., Chen, Z., Liu, Y ., Qin, Z., Yuan, H., Xu, K., Yuan, Y ., Gu...

    Pith/arXiv arXiv

  13. [18]

    train short, test long

    URL https://api.semanticscholar. org/CorpusID:283694164. Zhang, Z. A., Chen, R., Liu, S., Yao, Z., Ruwase, O., Chen, B., Wu, X., and Wang, Z. Found in the middle: How language models use long contexts better via plug- and-play positional encoding.ArXiv, abs/2403.04797, 2024b. URL https://api.semanticscholar. org/CorpusID:268296885. 13 AdaRoPE: Not All Att...

    Pith/arXiv arXiv 2018

  14. [21]

    Sincex(t)≤2tA≤2Aand by the hypothesis0< A <(γB2 γ) 1 1−γ we have 1−γBx(t) γ−1 ≥1−γB(2A) γ−1 >1−γB(2γB) −1 = 1 2 >0, hencex ′(t)>0

    Monotonicity ofx(t).For anyτ∈Iandt∈[0, τ], differentiateΨ(x(t), t) = 0to obtain x′(t) 1−γBx(t) γ−1 =A. Sincex(t)≤2tA≤2Aand by the hypothesis0< A <(γB2 γ) 1 1−γ we have 1−γBx(t) γ−1 ≥1−γB(2A) γ−1 >1−γB(2γB) −1 = 1 2 >0, hencex ′(t)>0. Thusx(t)is strictly increasing on[0, τ]

  15. [22]

    Then x(τn) is increasing and bounded by 2A, so by the monotone convergence theorem the limitx ∗ := limn→∞ x(τn)exists

    Closedness of I.Let τn ∈I with τn ↑τ ∗. Then x(τn) is increasing and bounded by 2A, so by the monotone convergence theorem the limitx ∗ := limn→∞ x(τn)exists. Clearlyx ∗ ≤2τ ∗A. Hence ∂xΨ(x∗, τ∗) = 1−γB(x ∗)γ−1 ≥1−γB(2A) γ−1 >0. By the implicit function theorem the solution extends uniquely to x(τ ∗) :=x ∗. Therefore the property holds at τ ∗, and I is closed

  16. [24]

    By the union bound, P max 1≤i≤L zi > τ σ ≤ LX i=1 P(zi > τ σ) =LΦ(−τ σ)≤L· 1 τ σ √ 2π e− 1 2 τ 2σ2

    Bounding S> L (τ).For any x >0, the Mills’ ratio gives Φ(−x)≤ϕ(x)/x , where Φ and ϕ denote the CDF and PDF of the standard normal distribution, respectively (see, e.g., Vershynin (2026, Proposition 2.1.2)). By the union bound, P max 1≤i≤L zi > τ σ ≤ LX i=1 P(zi > τ σ) =LΦ(−τ σ)≤L· 1 τ σ √ 2π e− 1 2 τ 2σ2 . 40 AdaRoPE: Not All Attention Heads Should Rotate...

    2026

  17. [25]

    Then EY i = Z τ σ −∞ eτ σx 1√ 2π e− x2 2 dx=e 1 2 τ 2σ2 Z τ σ −∞ 1√ 2π e− (x−τ σ)2 2 dx= 1 2 e 1 2 τ 2σ2

    BoundingS ≤ L (τ).SetY i :=e τ σzi 1{zi≤τ σ}. Then EY i = Z τ σ −∞ eτ σx 1√ 2π e− x2 2 dx=e 1 2 τ 2σ2 Z τ σ −∞ 1√ 2π e− (x−τ σ)2 2 dx= 1 2 e 1 2 τ 2σ2 . Hence ES ≤ L (τ) = LX i=1 eτ µi · 1 2 Ee τ σzi = 1 2 LX i=1 Ee τ σsi = 1 2 ES L(τ). Moreover, EY 2 i = Z τ σ −∞ e2τ σx 1√ 2π e− x2 2 dx=e 2τ 2σ2 Z τ σ −∞ 1√ 2π e− (x−2τ σ)2 2 dx=e 2τ 2σ2 Φ(−τ σ), By Mills...

    2019

  18. [26]

    Let F(L)(z) := QL i=1 Fi(z) be the CDF of s(L)

    Bounding s(L). Let F(L)(z) := QL i=1 Fi(z) be the CDF of s(L). Since F(L)(·) is non-decreasing, choose γL := min lnL, τ / √ lnL 1/2 and define s∗ := inf z:F (L)(z) =e −γL . Then P(s(L) < s∗) =e −γL →0 as L→ ∞. Moreover, γL =− LX i=1 lnF i(s∗) =− LX i=1 ln Φ s∗ −µ i σ ≥ −Lln Φ s∗ +∥µ∥ ∞ σ . 41 AdaRoPE: Not All Attention Heads Should Rotate and Scale Equall...

  19. [27]

    Thus, by Mills’ ratio, ln Φ s∗ − ∥µ∥∞ σ ≥ −2 1−Φ s∗ − ∥µ∥∞ σ ≥ −2 1 √ 2π s∗−∥µ∥∞ σ exp − 1 2 s∗ − ∥µ∥∞ σ 2! ≥ −exp − 1 2 s∗ − ∥µ∥∞ σ 2!

    For x∈ 1 2 ,1 , we have lnx≥1− 1 x =− 1−x x ≥ −2(1−x). Thus, by Mills’ ratio, ln Φ s∗ − ∥µ∥∞ σ ≥ −2 1−Φ s∗ − ∥µ∥∞ σ ≥ −2 1 √ 2π s∗−∥µ∥∞ σ exp − 1 2 s∗ − ∥µ∥∞ σ 2! ≥ −exp − 1 2 s∗ − ∥µ∥∞ σ 2! . Therefore, forL≥3, 1≤γ L ≤Lexp − 1 2 s∗ − ∥µ∥∞ σ 2! , which implies s∗ ≤σ √ 2 lnL+∥µ∥ ∞. To obtain an upper bound for s(L), set s∗ := 2σ √ lnL+∥µ∥ ∞. By the union b...

  20. [28]

    By Markov’s inequality, P ζL > δ s(L) =z ≤ 1 δ E ζL s(L) =z

    Boundingζ L.For anyδ >0, write P ζL > δ)≤P(ζ L > δ, s(L) ∈[s ∗, s∗]) +P(s (L) /∈[s∗, s∗] .(57) By the law of total probability, P(ζL > δ, s(L) ∈[s ∗, s∗]) = Z s∗ s∗ P ζL > δ s(L) =z f(L)(z) dz, wheref (L)(z)is the PDF ofs (L). By Markov’s inequality, P ζL > δ s(L) =z ≤ 1 δ E ζL s(L) =z . 43 AdaRoPE: Not All Attention Heads Should Rotate and Scale Equally ...

  21. [29]

    Sinceg(x, y) =x β β−1 /y 1 β−1 is continuous at(1,1), the continuous mapping theorem gives Eβ(α) ˆEβ(L) =g Z(s;λ) EZ(s;λ) , Z(s;βλ) EZ(s;βλ) P − →g(1,1) = 1asL→ ∞

    By Lemma 2(i), Z(s;λ) EZ(s;λ) P − →1, Z(s;βλ) EZ(s;βλ) P − →1,asL→ ∞. Sinceg(x, y) =x β β−1 /y 1 β−1 is continuous at(1,1), the continuous mapping theorem gives Eβ(α) ˆEβ(L) =g Z(s;λ) EZ(s;λ) , Z(s;βλ) EZ(s;βλ) P − →g(1,1) = 1asL→ ∞. (ii)If Λ = √ 2 min{1/β,1} , then by Lemma 2(i) and (ii), there are two cases. If Λ = √ 2 and βΛ< √ 2, so that 0< β <1, then...

  22. [30]

    We estimate Ik using the cosine integral functionCi(z), defined by Ci(z) =− Z ∞ z cost t dt.(59) We first prove that, for any z∈[1,∞) , |Ci(z)| ≤1/z

    Bounding Ik.We first show that, for any k≥b , |Ik| ≤(b+ 1)/(klnb) . We estimate Ik using the cosine integral functionCi(z), defined by Ci(z) =− Z ∞ z cost t dt.(59) We first prove that, for any z∈[1,∞) , |Ci(z)| ≤1/z. For any t >0, we have t−1 = R ∞ 0 e−ut du. Applying Fubini’s theorem on truncated intervals and then taking the limit gives Ci(z) =−lim K→∞...

  23. [31]

    Hence τ→ ∞and τlnL=o(e τ )for bothτ∈ {ρλ, ρβλ}

    Bounding SL(τ).By assumption, λ(L)→ ∞and λ(L) lnL=o(e ρmin{1,β}λ(L) ) as L→ ∞. Hence τ→ ∞and τlnL=o(e τ )for bothτ∈ {ρλ, ρβλ}. Decompose the partial sum as SL(τ) = L−1X k=0 1 + L−1X k=0 eτ Ik −1 =L+ (e τ −1) + L−1X k=1 eτ Ik −1 . We bound the remaining sum by splitting the index set atk=⌊τ⌋. Recall that, for anyk∈N, Ik = 1 lnb Z 1 1/b cos(ku) u du. ThusI ...

    2048

  24. [2016]

    org/CorpusID:2381275

    URL https://api.semanticscholar. org/CorpusID:2381275. Penedo, G., Kydl´ıˇcek, H., Lozhkov, A., Mitchell, M., Raffel, C. A., V on Werra, L., Wolf, T., et al. The fineweb datasets: Decanting the web for the finest text data at scale.Ad- vances in Neural Information Processing Systems, 37: 30811–30849, 2024. Peng, B., Quesnelle, J., Fan, H., and Shippole, E...

    Pith/arXiv arXiv 2024

  25. [2018]

    org/CorpusID:3725815

    URL https://api.semanticscholar. org/CorpusID:3725815. Su, J., Ahmed, M., Lu, Y ., Pan, S., Bo, W., and Liu, Y . Roformer: Enhanced transformer with rotary position embedding, 2021. URL https://arxiv.org/abs/ 2104.09864. Team, G., Kamath, A., Ferret, J., Pathak, S., Vieillard, N., Merhej, R., Perrin, S., Matejovicova, T., Ram ´e, A., Rivi`ere, M., et al. ...

    Pith/arXiv arXiv 2021

  26. [2019]

    org/CorpusID:208290939

    URL https://api.semanticscholar. org/CorpusID:208290939. Capp´e, O., Douc, R., Guillin, A., Marin, J.-M., and Robert, C. P. Adaptive importance sampling in general mixture classes.Statistics and Computing, 18(4):447–459, 2008. Chen, S., Wong, S., Chen, L., and Tian, Y . Extending con- text window of large language models via positional inter- polation.arX...

    Pith/arXiv arXiv 2008

  27. [2021]

    org/CorpusID:235458009

    URL https://api.semanticscholar. org/CorpusID:235458009. Huggins, J. H. and Roy, D. M. Sequential Monte Carlo as approximate sampling: bounds, adaptive resampling via ∞-ESS, and an application to particle Gibbs.Bernoulli, 25(1):584–622, 2019. Kong, A. A note on importance sampling using standardized weights. Technical Report 348, Uni- versity of Chicago, ...

    arXiv 2019

  28. [2022]

    org/CorpusID:247084324

    URL https://api.semanticscholar. org/CorpusID:247084324. Chiang, T.-R. and Yogatama, D. The rotary position embedding may cause dimension inefficiency in atten- tion heads for long-distance retrieval.arXiv preprint arXiv:2502.11276, 2025. URL https://arxiv. org/abs/2502.11276. Submitted 16 February 2025. Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., ...

    Pith/arXiv arXiv 2025

  29. [2023]

    org/CorpusID:263134555

    URL https://api.semanticscholar. org/CorpusID:263134555. Barbero, F., Vitvitskyi, A., Perivolaropoulos, C., Pas- canu, R., and Veli ´ckovi´c, P. Round and round we go! what makes rotary positional encodings useful? In International Conference on Learning Representations (ICLR), 2025. URL https://arxiv.org/abs/ 2410.06205. Published as a conference paper a...

    Pith/arXiv arXiv 2025

  30. [2024]

    org/CorpusID:270845965

    URL https://api.semanticscholar. org/CorpusID:270845965. Madaan, L., Singh, A. K., Schaeffer, R., Poulton, A., Koyejo, O., Stenetorp, P., Narang, S., and Hupkes, D. Quantifying variance in evaluation bench- marks.ArXiv, abs/2406.10229, 2024. URL https: //api.semanticscholar.org/CorpusID: 270521534. Martino, L., Elvira, V ., and Louzada, F. Effective sam- ...

    Pith/arXiv arXiv 2024

  31. [2025]

    org/CorpusID:276116722

    URL https://api.semanticscholar. org/CorpusID:276116722. Anson, B., Wang, X., and Aitchison, L. Scale-invariant 9 AdaRoPE: Not All Attention Heads Should Rotate and Scale Equally attention. InAdvances in Neural Information Processing Systems, 2025. Bai, J., Bai, S., Chu, Y ., Cui, Z., Dang, K., Deng, X., Fan, Y ., Ge, W., Han, Y ., Huang, F., Hui, B., Ji,...

    Pith/arXiv arXiv 2025