Pith. sign in

REVIEW 4 major objections 6 minor 49 references

Effective Length Extrapolation via Dimension-Wise Positional Embeddings Manipulation

T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Selectively rescaling 48 RoPE dimensions pushes an 8k-trained Llama to 128k with no training.

desk verdict Genuinely new per-dimension RoPE manipulation with large observed gains, but the headline result is partly fitted and the cross-model transfer of effective lengths is unverified. read the letter →

arxiv 2504.18857 v1 pith:NUW2BC63 submitted 2025-04-26 cs.CL cs.AI

Yi Lu , Wanxu Zhao , Xin Zhou , Chenxin An , Chenglong Wang , Shuo Li , Yuming Yang , Jun Zhao
show 4 more authors
Tao Ji Tao Gui Qi Zhang Xuanjing Huang
This is my paper · ORCID
classification cs.CLcs.AI
keywords lengthextrapolationRoPEpositionalembeddingstraining-freecontextextensionlong-contextbenchmarksdimension-wisescalingattentioncontributionwindow
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the failure of rotary-position models on long contexts is not uniform across hidden dimensions: different RoPE frequency groups have different "effective relative distances," and only a handful of dimensions actually carry the burden of length extrapolation. Its proposal, DPE, rescales the relative-position matrix of just the top 48 dimensions per head to their detected effective lengths, leaving other dimensions untouched. This is claimed to push an 8k-trained Llama-3-8B to 128k contexts without any continued training, beating YaRN and Self-Extend, and to raise Llama-3.1-70B's RULER average by more than 18 points within its trained window. If right, long-context capability becomes a cheap inference-time parameter that any RoPE model could tune.

What carries the argument

The load-bearing object is the per-dimension relative-position matrix $P_j$, built from RoPE's rotational embeddings $R(\theta_j, n-m)$ with frequencies $\theta_j = b^{-2j/d}$. DPE detects an effective length $e_i$ for each of $C=8$ dimension groups by sweeping the detection length $t$ in Eq. (7) on a needle-in-a-haystack task, then identifies key dimensions per head by ranking the 2-norm attention contribution $\|q^{(j)}_m\|\|k^{(j)}_n\|$, which bounds $|\langle q^{(j)}_m, k^{(j)}_n\rangle|$ by Cauchy-Schwarz. Only the top 48 dimensions have their position matrices rescaled by $s_i = \lfloor L/e_i \rfloor$ outside a local window $w=1\text{k}$ (Eq. (9)), so all relative positions land inside each group's detected comfort zone. The mechanism's work is to show that selective rescaling, not uniform rescaling, is what determines extrapolation success.

What would settle it

Run DPE on a model not in the paper (e.g., another RoPE model family at 8k training length) twice: once with the paper's Llama-3-8B effective-length table fixed, and once with effective lengths re-detected by the same NIAH sweep on that model. If the re-detected lengths differ substantially from the fixed table and the fixed-table version loses most of the gain, the transfer assumption is falsified; likewise, evaluating Llama-3-8B+DPE at 256k should show collapse beyond the largest detected effective length if the mechanism is truly length-bounded.

Watch

Extended reading notes

Core claim

DPE's central discovery is that RoPE dimensions are not interchangeable for length generalization. By sweeping the maximum relative position of each of eight dimension groups from 1k to 128k on a needle-in-a-haystack task, the paper finds that low- and high-frequency groups tolerate very long positions while middle frequencies collapse past the training length. A 2-norm attention-contribution ranking then identifies, for each attention head, the 48 dimensions whose positions matter most; scaling only those dimensions' position matrices to their per-group effective lengths (e.g., 65536 for dimensions 0-7, 4096 for dimensions 32-47 on Llama-3-8B) restores retrieval at 128k. The paper reports that this selective manipulation outperforms scaling all dimensions, lifts Llama-3-8B from 0 to 56.08 on RULER 128k, and lifts Llama-3.1-70B from 66.41 to 86.39, above the reported GPT-4-1106-preview score of 81.2.

Load-bearing premise

The load-bearing premise is that each group of RoPE dimensions has one stable effective relative distance, measured once on Llama-3-8B-Instruct with a needle-in-a-haystack sweep, and that this distance transfers to other models and settings; if that transfer fails, DPE's fixed "C=8, w=1k, top-48 for all models" recipe loses its anchor.

Editorial extensions

If this is right

  • An 8k-trained Llama-3-8B-Instruct can answer needle-in-a-haystack queries at 128k with 92.5% accuracy, and 56.08 average on RULER's 13 tasks, without continued training.
  • The same recipe lifts models already trained to 128k inside their training length: Llama-3.1-70B goes from 66.41 to 86.39 on RULER, beating the reported GPT-4-1106-preview score of 81.2.
  • Only the top 48 of 64 RoPE dimension pairs need rescaling; scaling all dimensions is slightly worse, so the method is a model of dimensional sparsity, not a blanket adjustment.
  • Because the change lives in the position matrix, it composes with FlashAttention-2 and adds negligible inference overhead at 128k.
  • Effective lengths differ across dimension groups, so the method turns context extension into a per-group hyperparameter search rather than a global interpolation or exclusion choice.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported effective-length table (dimensions 0-7 at 64k, 8-15 at 16k, etc.) might double as a diagnostic: it exposes which RoPE frequencies a model actually learned to trust, so one could compare training distributions across base models.
  • Re-detecting effective lengths on each target model, rather than reusing the Llama-3-8B table for all models, is an obvious testable upgrade; if rankings shift across scales or families, per-model detection should outperform the paper's fixed settings.
  • DPE's gains inside the training window suggest position-matrix conditioning, not just out-of-distribution avoidance; it may be improvable by stacking with frequency-based rescaling such as base changes.
  • The 256k frontier is unexplored: nothing in the mechanism stops effective lengths from being extrapolated further, but the paper stops at 128k.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Dimension-Wise Positional Embeddings Manipulation (DPE), a training-free method for extrapolating the context length of RoPE-based LLMs. DPE divides head dimensions into groups, detects a maximum 'effective relative distance' per group via Needle-in-a-Haystack (NIAH) accuracy on Llama-3-8B-Instruct, identifies per-head 'key dimensions' using a 2-norm attention-contribution metric, and rescales only those dimensions' relative position matrices during inference. Experiments on NIAH, RULER, and InfiniteBench report consistent gains over existing training-free baselines, including large improvements on Llama-3-8B (8K→128K), Mistral-7B, Qwen-2.5-7B, and Llama-3.1-70B, with the latter surpassing GPT-4-1106-preview on RULER.

Significance. If the cross-model generality is confirmed, DPE is a notable contribution: it is simple, requires no training, adds negligible overhead with FlashAttention-2, and is backed by ablations showing that selective dimension-wise manipulation outperforms uniform position scaling. The paper includes a useful analysis of differing effective lengths across RoPE dimensions and demonstrates that per-head 2-norm attention contribution can identify dimensions whose rescaling restores long-context performance. The release of code further supports reproducibility. However, the strength of the claims depends on unverified assumptions about the transferability of the effective lengths across models with different RoPE bases.

major comments (4)
  1. [Section 3.1 / Appendix D.1 / Table 1] The effective lengths E reported in Appendix D.1 are selected by ranking NIAH accuracy on Llama-3-8B-Instruct (Figure 1b), and the same benchmark's score for that same model is then reported as a DPE outcome in Table 1. The Llama-3-8B NIAH row is therefore partly a selection result rather than an independent prediction, which inflates the apparent performance of DPE on that row. Please either provide a held-out evaluation (e.g., detect E on a different model or using perplexity, then evaluate NIAH on Llama-3-8B) or explicitly separate selection results from predictions when discussing Table 1.
  2. [Section 3.3 / Appendix D.1] The manuscript does not state whether the per-group effective lengths E are re-detected for each model or reused from Llama-3-8B. Because RoPE bases differ across the evaluated models (e.g., Llama-3 uses base 500000, Mistral-v0.2 uses 10000, and Qwen-2.5 uses approximately 1e6), the same dimension index corresponds to different angular frequencies across models; thus the transferability of E is not self-evident. If E is reused without re-detection, a transfer experiment (e.g., comparing fixed E with per-model re-detected E on Mistral or Qwen-2.5) is required to justify the cross-model claims; if E is re-detected per model, the per-model E values must be reported and the detection procedure must be described for each model. This is load-bearing for the claim that DPE generalizes across model families.
  3. [Section 3.2] The procedure for computing the 2-norm attention contribution used to select key dimensions is underspecified. The text defines D_h using 'all the queries and keys of the sequence with length L' but does not say whether the 2-norm scores are computed on the evaluation sequence at inference time, on a separate calibration set, or on a short sequence within the training length, nor how many tokens are used. Since the top-48 selection is fixed for all models, please specify the exact data, sequence length, and aggregation procedure used to determine D_h; this is necessary for reproducibility and for interpreting the overhead of the method.
  4. [Tables 1 and 2] No error bars, confidence intervals, or significance tests are reported anywhere in the paper, even though several headline comparisons are based on small margins (e.g., Llama-3-8B NIAH DPE 92.50 vs. Self-Extend 89.50, and Llama-3.1-70B RULER DPE 86.39 vs. GPT-4 81.2). Given that Table 1 uses only 100 test cases per model, the reported differences may be within sampling noise for some rows. Please report variance across repeated runs or bootstrap intervals, or at least the number of test items per cell, and state whether differences are statistically reliable.
minor comments (6)
  1. [Appendix D.2] The baseline name 'Rerope' should be 'ReRoPE', and 'trucated length' should be 'truncated length'.
  2. [Abstract and Section 4] The abstract refers to 'GPT-4-128K' while Table 2 reports 'GPT-4-1106-preview'; please harmonize the terminology.
  3. [Section 6 (Related Work)] There is a typo: 'demons trate' should be 'demonstrate'.
  4. [Equations (7) and (9)] The notation using floor brackets is ambiguous; please define the floor division explicitly near Equation (7).
  5. [Section 4] The sentence 'it achieves 56 point' should read '56 points' or '56.08'.
  6. [Appendix A] The table header spells 'Claude2' while the text says 'Claud2'; please correct the spelling.

Circularity Check

1 steps flagged · score 4.0 of 10

Per-group effective lengths are fit to Llama-3-8B NIAH accuracy, so the same model's Table 1 NIAH score is partly a selection result; the RULER/InfiniteBench cross-model results remain independent predictions.

  1. fitted input called prediction [Appendix D.1; Section 3.1; Table 1]
    "The final effective length for Llama3-8B-Instruct was determined based on the ranked NIAH accuracy results shown in Figure 1b. For each 8-dimension segment, we selected the configuration yielding Rank 1 performance."

    The per-group effective lengths E are selected by ranking NIAH accuracy on Llama-3-8B-Instruct, and Table 1 then reports DPE's NIAH accuracy on that same model (92.50) as an experimental outcome. E is therefore a hyperparameter fitted to maximize NIAH accuracy on this exact model and task; the reported score is a selection result, not a held-out prediction. The same NIAH task is also used in Section 5 to ablate these hyperparameters, reinforcing the selection loop. The RULER and InfiniteBench results, and the other models' NIAH rows, are not fitted this way and remain external evidence, so this is a confined partial circularity rather than a collapse of the whole derivation.

full rationale

DPE's key-dimension identification via the 2-norm attention contribution (Section 3.2) is independent of NIAH: it is computed from query/key norms and is not fit to the target benchmark. The cross-model RULER and InfiniteBench numbers for Mistral, Llama-3.1, and Qwen use the Llama-3-8B-detected effective lengths as fixed hyperparameters and are evaluated on held-out tasks and models, so they are genuine predictions. The circular component is the Llama-3-8B NIAH row in Table 1 (and the Section 5 ablation of the same quantities), where the per-group effective lengths were chosen to maximize NIAH accuracy on that model and then the same model's NIAH accuracy is reported as DPE's result. This fits the 'fitted input called prediction' pattern but does not by itself force the paper's main extrapolation claims, which rest on RULER/InfiniteBench results. A separate robustness concern, not a circularity finding, is that the paper fixes C=8, w=1k, and top-48 'for all the models' (Section 3.3) while Appendix D.1 reports effective lengths only for Llama-3-8B-Instruct; because RoPE bases differ across Llama-3, Mistral, and Qwen, identical dimension indices map to different physical frequencies, so the cross-model transfer of E is unverified. No load-bearing self-citation chain is present: the 2-norm metric is attributed to Barbero et al. and Ji et al., and baseline citations are not used to justify DPE's central premise. Overall score 4 reflects one confined fitted-input-as-prediction with substantial independent external evaluation.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

DPE introduces no new physical or architectural entities; all freedom is in hyperparameters (E, k, w, C) and in the heuristic selection of key dimensions.

free parameters (4)
  • Per-group effective lengths E = 65536, 16384, 65536, 16384, 4096, 4096, 8192, 32768 (Llama-3-8B-Instruct)
    Selected as the rank-1 NIAH accuracy length per 8-dimension group in Appendix D.1; this is a fit to the evaluation benchmark that Table 1 then reports.
  • Key dimension count k = 48 (ablation peak reported at k=40 in Fig 5b)
    Chosen to maximize NIAH accuracy in Fig 3 and Fig 5b; discrepancy with ablation peak is not explained.
  • Local window w = 1024
    Set by hand to preserve local context, following previous methods like Self-Extend and ReRoPE (Section 3.3).
  • Group count C = 8
    Arbitrary split of the 64-dim head into groups of 8; sensitivity to grouping not reported.
assumptions (5)
  • domain assumption Dimensions within one frequency group share the same effective relative distance.
    Stated in Section 3.1 before the detection procedure; it is an approximation, since within a group the frequencies differ by a factor of b^(2/d).
  • ad hoc to paper The 2-norm attention contribution identifies the dimensions whose position scaling matters for extrapolation.
    Section 3.2 uses |<q,k>| <= ||q|| ||k|| to rank dimensions; the link between this static norm and OOD error is heuristic, not derived.
  • ad hoc to paper Effective lengths E detected on Llama-3-8B-Instruct transfer to Mistral-7B, Llama-3.1, and Qwen-2.5 without re-detection.
    Appendix D.1 gives E only for Llama-3-8B; Section 3.3 applies the same constants ('all the models') with no per-model detection results.
  • domain assumption Leaving non-key dimensions at their raw out-of-distribution positions does not hurt overall accuracy.
    Empirical finding in Figure 3; only top-48 dims are scaled, the rest see position indices beyond 8K or 32K.
  • standard math Standard RoPE rotation and FlashAttention-2 kernels compute as specified.
    Background in Section 2.1 and implementation in Section 3.3.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Effective Length Extrapolation via Dimension-Wise Positional Embeddings Manipulation." pith.science (2026). https://pith.science/paper/NUW2BC63

@misc{pith2026250418857,
  author       = {Pith},
  title        = {Pith review of: Effective Length Extrapolation via Dimension-Wise Positional Embeddings Manipulation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NUW2BC63}},
  note         = {Machine review of arXiv:2504.18857}
}
read the original abstract

Large Language Models (LLMs) often struggle to process and generate coherent context when the number of input tokens exceeds the pre-trained length. Recent advancements in long-context extension have significantly expanded the context window of LLMs but require expensive overhead to train the large-scale models with longer context. In this work, we propose Dimension-Wise Positional Embeddings Manipulation (DPE), a training-free framework to extrapolate the context window of LLMs by diving into RoPE's different hidden dimensions. Instead of manipulating all dimensions equally, DPE detects the effective length for every dimension and finds the key dimensions for context extension. We reuse the original position indices with their embeddings from the pre-trained model and manipulate the key dimensions' position indices to their most effective lengths. In this way, DPE adjusts the pre-trained models with minimal modifications while ensuring that each dimension reaches its optimal state for extrapolation. DPE significantly surpasses well-known baselines such as YaRN and Self-Extend. DPE enables Llama3-8k 8B to support context windows of 128k tokens without continual training and integrates seamlessly with Flash Attention 2. In addition to its impressive extrapolation capability, DPE also dramatically improves the models' performance within training length, such as Llama3.1 70B, by over 18 points on popular long-context benchmarks RULER. When compared with commercial models, Llama 3.1 70B with DPE even achieves better performance than GPT-4-128K.

Figures

Figures reproduced from arXiv: 2504.18857 by the authors.

Figure 1
Figure 1. Detecting the effective relative distance across different dimension groups. We [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. 2-norm Attention Contribution for different heads and layers of Llama3- 8b. We select top-k dimensions as the key dimensions for extrapolation. For example, the top-1 dimension is selected in the red dashed line. Since different dimensions contribute differ￾ently to length extrapolation, we hypothesize that some dimensions play a crucial role in context extension. Since OOD issue is closely related to the attention … view at source ↗
Figure 3
Figure 3. NIAH Accuracy on Llama3-8b. Only top-k dimensions’ position indices are scaled. Experiment Setup After identifying the key dimensions, we scale the position in￾dices only for key dimensions Dh while keeping the position indices of other dimen￾sions unchanged. We set the scaled length t from 2k to 8k. We also use 100 test samples of NIAH with 128k length and evaluate the accuracy under different values of k to vali￾d… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: An illustrative example of DPE with three main procedures. (a) We detect the effective length ei = 4,ej = 5 for dimension group gi , gj . (b) Dimensions 29 and 47 are identified as key dimensions, and the corresponding effective length is obtained from the detection re…
Figure 5
Figure 5. Figure 5: Ablation study on the rank of effective length and key dimension selection. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Efficiency Test of DPE and the standard Flash Attention based on Llama3.1 8B. [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: NIAH Accuracy ranking(left), NIAH Accuracy(middle) and perplexity value(right) [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

49 extracted references · 11 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Mistral OCR , 2025

    Mistral AI. Mistral OCR , 2025. URL https://mistral.ai/news/mistral-ocr

    work page 2025

  3. [3]

    Llama 3 model card

    AI@Meta. Llama 3 model card. 2024. URL https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md

    2024

  4. [4]

    Training-free long-context scaling of large language models, 2024 a

    Chenxin An, Fei Huang, Jun Zhang, Shansan Gong, Xipeng Qiu, Chang Zhou, and Lingpeng Kong. Training-free long-context scaling of large language models, 2024 a

    work page 2024

  5. [5]

    Why does the effective context length of llms fall short?, 2024 b

    Chenxin An, Jun Zhang, Ming Zhong, Lei Li, Shansan Gong, Yao Luo, Jingjing Xu, and Lingpeng Kong. Why does the effective context length of llms fall short?, 2024 b . URL https://arxiv.org/abs/2410.18745

    arXiv 2024

  6. [6]

    Round and round we go! what makes rotary positional encodings useful?, 2025

    Federico Barbero, Alex Vitvitskyi, Christos Perivolaropoulos, Razvan Pascanu, and Petar Veličković. Round and round we go! what makes rotary positional encodings useful?, 2025. URL https://arxiv.org/abs/2410.06205

    arXiv 2025

  7. [7]

    NTK-Aware Scaled RoPE allows LLaMA models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation , 2023 a

    bloc97. NTK-Aware Scaled RoPE allows LLaMA models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation , 2023 a . URL https://www.reddit.com/r/LocalLLaMA/comments/14lz7j5/ntkaware_scaled_rope_allows_llama_models_to_have/

    work page 2023

  8. [8]

    by parts

    bloc97. Add NTK-Aware interpolation "by parts" correction , 2023 b . URL https://github.com/jquesnelle/scaled-rope/pull/1

    work page 2023

Show all 49 references
  1. [9]

    Clex: Continuous length extrapolation for large language models, 2024

    Guanzheng Chen, Xin Li, Zaiqiao Meng, Shangsong Liang, and Lidong Bing. Clex: Continuous length extrapolation for large language models, 2024. URL https://arxiv.org/abs/2310.16450

    2024 arXiv

  2. [10]

    Extending context window of large language models via positional interpolation, 2023

    Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation, 2023

    2023

  3. [11]

    Transformer-xl: Attentive language models beyond a fixed-length context

    Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Jan 2019. doi:10.18653...

    2019 doi

  4. [12]

    Flashattention-2: Faster attention with better parallelism and work partitioning, 2023

    Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning, 2023. URL https://arxiv.org/abs/2307.08691

    2023 arXiv

  5. [13]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei F...

    2025 arXiv

  6. [14]

    Longrope: Extending llm context window beyond 2 million tokens, 2024

    Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. Longrope: Extending llm context window beyond 2 million tokens, 2024. URL https://arxiv.org/abs/2402.13753

    2024 arXiv

  7. [16]

    Dynamically Scaled RoPE further increases performance of long context LLaMA with zero fine-tuning , 2023

    emozilla. Dynamically Scaled RoPE further increases performance of long context LLaMA with zero fine-tuning , 2023. URL https://www.reddit.com/r/LocalLLaMA/comments/14mrgpr/dynamically_scaled_rope_further_increases/

    2023

  8. [17]

    Data engineering for scaling language models to 128k context, 2024

    Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannaneh Hajishirzi, Yoon Kim, and Hao Peng. Data engineering for scaling language models to 128k context, 2024

    2024

  9. [18]

    Needle in a haystack - pressure testing llms, 2023

    gkamradt. Needle in a haystack - pressure testing llms, 2023. URL https://github.com/gkamradt/LLMTest_NeedleInAHaystack

    2023

  10. [19]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Art...

    2024 arXiv

  11. [20]

    Lm-infinite: Simple on-the-fly length generalization for large language models, 2023

    Chi Han, Qifan Wang, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. Lm-infinite: Simple on-the-fly length generalization for large language models, 2023

    2023

  12. [21]

    On the token distance modeling ability of higher rope attention dimension, 2024

    Xiangyu Hong, Che Jiang, Biqing Qi, Fandong Meng, Mo Yu, Bowen Zhou, and Jie Zhou. On the token distance modeling ability of higher rope attention dimension, 2024. URL https://arxiv.org/abs/2410.08703

    2024 arXiv

  13. [22]

    Ruler: What's the real context size of your long-context language models?, 2024

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What's the real context size of your long-context language models?, 2024

    2024

  14. [23]

    Towards economical inference: Enabling deepseek's multi-head latent attention in any transformer-based llms, 2025

    Tao Ji, Bin Guo, Yuanbin Wu, Qipeng Guo, Lixing Shen, Zhan Chen, Xipeng Qiu, Qi Zhang, and Tao Gui. Towards economical inference: Enabling deepseek's multi-head latent attention in any transformer-based llms, 2025. URL https://arxiv.org/abs/2502.14837

    2025

  15. [24]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  16. [25]

    Abdi, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu

    Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H. Abdi, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention, 2024. URL https://a...

    2024 arXiv

  17. [26]

    Llm maybe longlm: Self-extend llm context window without tuning, 2024

    Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zirui Liu, Chia-Yuan Chang, Huiyuan Chen, and Xia Hu. Llm maybe longlm: Self-extend llm context window without tuning, 2024

    2024

  18. [27]

    Thus spake long-context large language model, 2025

    Xiaoran Liu, Ruixiao Li, Mianqiu Huang, Zhigeng Liu, Yuerong Song, Qipeng Guo, Siyang He, Qiqi Wang, Linlin Li, Qun Liu, Yaqian Zhou, Xuanjing Huang, and Xipeng Qiu. Thus spake long-context large language model, 2025. URL https://arxiv.org/abs/2502.17129

    2025

  19. [28]

    Zhang, Zhilin Yang, Xinyu Zhou, Mingxing Zhang, and Jiezhong Qiu

    Enzhe Lu, Zhejun Jiang, Jingyuan Liu, Yulun Du, Tao Jiang, Chao Hong, Shaowei Liu, Weiran He, Enming Yuan, Yuzhi Wang, Zhiqi Huang, Huan Yuan, Suting Xu, Xinran Xu, Guokun Lai, Yanru Chen, Huabin Zheng, Junjie Yan, Jianlin Su, Yuxin Wu, Neo Y. Zhang, Zhilin Yang, Xinyu Zhou, M...

    2025 arXiv

  20. [29]

    Chiu, Siyu Ren, Fei Yuan, Wenting Zhao, Zhiyong Wu, and Alexander M

    Yi Lu, Jing Nathan Yan, Songlin Yang, Justin T. Chiu, Siyu Ren, Fei Yuan, Wenting Zhao, Zhiyong Wu, and Alexander M. Rush. A controlled study on long context extension and generalization in llms, 2024 a . URL https://arxiv.org/abs/2409.12181

    2024 arXiv

  21. [30]

    Longheads: Multi-head attention is secretly a long context processor

    Yi Lu, Xin Zhou, Wei He, Jun Zhao, Tao Ji, Tao Gui, Qi Zhang, and Xuanjing Huang. Longheads: Multi-head attention is secretly a long context processor. arXiv preprint arXiv:2402.10685, 2024 b

    2024 arXiv

  22. [31]

    Learning to reason with llms, 2024

    OpenAI. Learning to reason with llms, 2024. URL https://openai.com/index/learning-to-reason-with-llms/

    2024

  23. [32]

    Yarn: Efficient context window extension of large language models, 2023

    Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models, 2023

    2023

  24. [33]

    Qwen2.5 technical report, 2025

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, ...

    2025 arXiv

  25. [34]

    Rae, Anna Potapenko, Siddhant M

    Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, and Timothy P. Lillicrap. Compressive transformers for long-range sequence modelling, 2019

    2019

  26. [35]

    Rectified rotary position embeddings

    Jianlin Su. Rectified rotary position embeddings. https://github.com/bojone/rerope, 2023

    2023

  27. [36]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Cornell University - arXiv,Cornell University - arXiv, Apr 2021

    2021

  28. [37]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 0 127063, 2024

    2024

  29. [38]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurip...

    2017

  30. [39]

    Infllm: Unveiling the intrinsic capacity of llms for understanding extremely long sequences with training-free memory

    Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, Song Han, and Maosong Sun. Infllm: Unveiling the intrinsic capacity of llms for understanding extremely long sequences with training-free memory. arXiv preprint arXiv:2402.04617, 2024 a

    2024 arXiv

  31. [40]

    Efficient streaming language models with attention sinks, 2023

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks, 2023

    2023

  32. [41]

    Efficient streaming language models with attention sinks, 2024 b

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks, 2024 b

    2024

  33. [42]

    Effective long-context scaling of foundation models, 2023

    Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, S...

    2023 arXiv

  34. [43]

    Helmet: How to evaluate long-context language models effectively and thoroughly, 2025

    Howard Yen, Tianyu Gao, Minmin Hou, Ke Ding, Daniel Fleischer, Peter Izsak, Moshe Wasserblat, and Danqi Chen. Helmet: How to evaluate long-context language models effectively and thoroughly, 2025. URL https://arxiv.org/abs/2410.02694

    2025 arXiv

  35. [44]

    Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Y. X. Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Wenfeng Liang, and Wangding Zeng. Native sparse attention: Hardware-aligned and natively trainable sparse attention...

    2025 arXiv

  36. [45]

    Anygpt: Unified multimodal llm with discrete sequence modeling, 2024

    Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, Hang Yan, Jie Fu, Tao Gui, Tianxiang Sun, Yugang Jiang, and Xipeng Qiu. Anygpt: Unified multimodal llm with discrete sequence modeling, 2024. URL https://arxiv...

    2024 arXiv

  37. [46]

    Hirope: Length extrapolation for code models using hierarchical position, 2024 a

    Kechi Zhang, Ge Li, Huangzhao Zhang, and Zhi Jin. Hirope: Length extrapolation for code models using hierarchical position, 2024 a . URL https://arxiv.org/abs/2403.19115

    2024 arXiv

  38. [47]

    bench: Extending long context evaluation beyond 100k tokens, 2024 b

    Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, and Maosong Sun. bench: Extending long context evaluation beyond 100k tokens, 2024 b . URL https://arxiv.org/abs/2402.13718

    2024 arXiv

  39. [48]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  40. [49]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  41. [50]

    https://www.reddit.com/r/LocalLLaMA/comments/14lz7j5/ntkaware_scaled_rope_allows_llama_models_to_have/

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2025 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.