Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

A Training-Free Length Extrapolation Approach for LLMs: Greedy Attention Logit Interpolation (GALI)

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read GALI: a training-free method that extends LLM context windows by interpolating attention logits, not embeddings.

desk verdict Genuinely new logit-level interpolation idea, but the 'no tuning' claim is undercut by the authors' own per-setting grid search; worth reviewing with major revisions. read the letter →

arxiv 2502.02659 v2 pith:52X73VVI submitted 2025-02-04 cs.CL cs.AI

Yan Li , Tianyi Zhang , Zechuan Li , Soyeon Caren Han This is my paper · ORCID
classification cs.CLcs.AI
keywords GALIGreedyAttentionLogitInterpolationlengthextrapolationtraining-freeRoPEpositionalout-of-distributionlong-contextLLMs
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

GALI is a training-free method for extending the usable context of LLMs that use rotary position embeddings (RoPE). The paper's claim is that an LLM can generalize far beyond its training window if, instead of rescaling all positions with a global factor, you keep the trained position IDs for the first window and only interpolate the tail, chunk by chunk. The key move is to interpolate the attention logit for fractional relative positions from the logits of neighboring trained integer positions, adding Gaussian noise scaled by the interval to imitate RoPE's oscillation; no position embedding is ever evaluated at an unseen integer. If the claim holds, GALI gives long-context capability without fine-tuning, without per-input-length scaling, and without degrading short-context performance, and it would remove a major barrier to using LLMs on long documents.

What carries the argument

The central mechanism is logit-level interpolation at fractional relative positions combined with greedy, chunk-wise position-ID reuse. RoPE makes the attention logit a function $a(x_m, x_n, m-n)$ of the relative position, so GALI never feeds an unseen integer position to the rotary embedding. Instead, when the relative position $r$ is fractional, it computes the two neighboring trained logits and forms $a(x_m,x_n,r) = a(x_m,x_n,\lfloor r\rfloor) - (a(x_m,x_n,\lfloor r\rfloor) - a(x_m,x_n,\lceil r\rceil)) \cdot (r \bmod 1) + \mathcal{N}(0, (r/L_{tr})^2)$. The implementation approximates $r$ by $\lceil m\rceil - n$ so the operation can be carried out with matrix multiplications on query and key states. The greedy part assigns each over-length chunk only the minimum number of new position IDs while reusing the full pretrained interval, which is what avoids global scaling and preserves short-context behavior.

What would settle it

Run a small RoPE model with exact fractional position IDs, bypassing the $\lceil m\rceil - n$ shortcut, and compare the true attention logit against GALI's interpolation at, say, $r = k + 0.5$ across many layer-head pairs; if the median absolute error is not small relative to the logit scale, or if the error has heavy tails where Gaussian noise would not help, the core approximation fails. A second decisive check is to replace the ceiling-based implementation with exact fractional distances and re-run the LongBench comparison; if GALI's gains vanish, the reported behavior is an artifact of the approximation.

Watch

Extended reading notes

Core claim

The paper's central claim is that pretrained attention logits are a sufficient substrate for length extrapolation: for a fractional relative position $r$, the true attention logit is well approximated by $a(x_m,x_n,r) = a(x_m,x_n,\lfloor r\rfloor) - [a(x_m,x_n,\lfloor r\rfloor) - a(x_m,x_n,\lceil r\rceil)] \cdot (r \bmod 1) + \mathcal{N}(0, (r/L_{tr})^2)$. Combined with greedy chunk-wise position-ID assignment, this removes the logit outliers that appear when RoPE embeddings are evaluated at unseen positions. The paper reports that on Llama-3-8B-instruct with 4k and 8k windows, GALI achieves the highest LongBench average among training-free methods at 16k and 32k targets, stable perplexity on PG19 out to 32k, and closer attention-distribution alignment with the original model than NTK, Dyn-NTK, YaRN, SelfExtend, and ChunkLlama. It also reports the narrower-interval finding: mapping inputs into a smaller positional range improves scores even on short-context tasks.

Load-bearing premise

The method assumes that at fractional relative positions RoPE attention logits are close enough to the linear interpolation of the two neighboring integer-position logits (plus Gaussian noise) that the approximation never changes which tokens attend to which; the ceiling-based distance used in the code is also a stand-in for the true fractional distance, and the paper gives no error bound for either step.

Editorial extensions

If this is right

  • GALI's short-context behavior is protected by construction: since tokens inside the training window keep their original position IDs, performance on ordinary inputs should remain at the backbone level with no hyperparameter change.
  • The method removes the need to choose an input-length-specific scaling factor, because each chunk receives only the minimal number of interpolated IDs, so the same configuration works for 16k and 32k target windows.
  • If the attention-logit approximation is valid across models, GALI should transfer to any RoPE-based LLM with strong positional understanding; the paper's Llama-2-7B results show the gain shrinks when the backbone's positional understanding is weaker.
  • The paper's narrower-interval result implies that evaluations of extrapolation methods should report the effective positional range, not just the target context length, because a wider target window can inflate or deflate scores depending on the scaling factor.
  • The attention-distribution analysis indicates GALI leaves the model's native attention pattern nearly unchanged, which is why it inherits the pretrained model's strengths rather than inserting a new positional prior.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if RoPE logits are indeed locally linear in relative position, the same interpolation scheme could be applied to other position encodings with long-term decay, such as ALiBi; the paper notes this compatibility but does not test it.
  • Editorial inference: the implementation's ceiling-based distance $r = \lceil m\rceil - n$ is not identical to the exact fractional distance $m - n$ in Eq. (3); comparing the two head-to-head would reveal whether the approximation is a harmless implementation detail or a load-bearing part of the method.
  • Editorial inference: the paper's finding that narrower positional intervals improve performance suggests a simpler baseline, compressing all inputs into the shortest well-understood interval, might capture part of GALI's benefit, and GALI's added value would show up mainly where long-range dependencies matter.
  • Editorial inference: because GALI needs two attention-logit passes, its practical reach depends on a flash-attention-compatible implementation; the paper lists this as a limitation and future work, and without it the method's memory cost grows with sequence length.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes GALI, a training-free method for extending the context window of RoPE-based LLMs. The method keeps original position IDs inside the training window, chunks the extrapolated portion, and assigns interpolated position IDs per chunk; attention logits for fractional relative positions are linearly interpolated between integer-position logits with Gaussian noise. Experiments compare GALI with NTK, Dyn-NTK, YaRN, SelfExtend, and ChunkLlama on LongBench, L-Eval, and PG19 using Llama3-8B-ins and Llama2-7B-Chat backbones. The central claims are that GALI is stable and superior without input-length-specific tuning, and that narrower positional intervals improve performance even on short-context tasks.

Significance. If substantiated, GALI would be a practically useful addition to the training-free length-extrapolation toolbox. The paper ships an open-source implementation, evaluates against five baselines across three benchmarks, and includes attention-distribution analyses; these are genuine strengths. However, the key 'no tuning' claim is not supported by the reported protocol because hyperparameters are searched per target-length setting. The algorithm-text mismatch and the unvalidated linearity assumption further weaken the mechanism story. The contribution is promising, but the headline claims need a corrected experimental protocol before acceptance.

major comments (4)
  1. [Appendix E, Table 10] The abstract and Section 1 state that GALI requires no input-length-specific tuning, but for every target-length setting Table 10 searches chunk_size in {1000,2000,3000} and local_window in {128,256,512,1024} and does not report the selected values. Tables 1 and 2 therefore appear to report best-of-grid results, so GALI's advantage over baselines cannot be separated from per-configuration selection. The no-tuning claim is not established; the authors should either fix a single (s, Lw) across all target lengths or explicitly reframe the contribution.
  2. [Section 3.2, Eq. (3); Appendix D, Algorithm 3] Eq. (3) defines the interpolated logit as a linear combination of a(floor(r)) and a(ceil(r)) for r=m-n, with the fractional part (r mod 1) as the mixing coefficient. Algorithm 3 instead uses r=ceil(m)-n and computes attn_floor and attn_ceil from query_states_ceil with key_states_ceil and key_states_floor, with rel_coef=(ceil(position_ids)-position_ids) mod 1. When key positions are integers, attn_floor equals attn_ceil, so no interpolation occurs and the fractional query is effectively snapped upward; when key positions are fractional, rel_coef is not r mod 1 from Eq. (3). The paper should provide an equivalence proof or correct the algorithm.
  3. [Section 4.1, 'Llama3-4k backbone'] The paper obtains Llama3-4k by 'modifying its max_position_embedding parameter.' This config change does not retrain or otherwise make the model equivalent to a model trained with a 4k context window. All 4k-to-16k and 4k-to-32k comparisons, and the inference that narrower intervals are better, rest on this proxy. The authors should justify the equivalence or use a genuinely 4k-trained model.
  4. [Section 4.2 and Appendix C.1] The claim that GALI is superior also on short-context tasks is weakened by the appendix: on GSM, QuALITY, and TOEFL with the Llama3-8k backbone, GALI is described as performing almost identically to the backbone, whereas SelfExtend, NTK, and YaRN outperform the backbone. Given that the average in Table 2 excludes these short datasets, the statement that GALI is 'stable and superior ... even on short-context tasks' is not supported by the reported evidence.
minor comments (5)
  1. [Table 2 and Table 5] There are typos in benchmark names: 'TOFEL' should be 'TOEFL' and 'SFCition' should be 'SFiction'.
  2. [Table 8] The table heading says 'Llama3-7b-chat-4k' but the experiments and text use Llama2-7B-Chat-4K.
  3. [Appendix C.3] The text refers to 'Figures 5 and 6' for the row-by-row attention plots, but those plots are actually Figures 8 and 9.
  4. [Section 3.1, Eq. (2)] The cases block contains a stray 'j >1;' and does not clearly specify the range of j; it should be cleaned up and defined.
  5. [Section 4.3] The sentence 'We tested PPL using a 16k contest window' should say 'context window'.

Circularity Check

1 steps flagged · score 6.0 of 10

GALI's headline 'no input-length-specific tuning' is undercut by per-experiment hyperparameter search reported in Appendix E; reported benchmark scores are best-of-grid selections.

  1. fitted input called prediction [Appendix E, Table 10; abstract and Section 4.2 (Tables 1-2)]
    "GALI achieves stable and superior performance ... without requiring input-length-specific tuning. ... For GALI, the reported hyperparameters represent the combinations we search for each experiment. ... GALI chunk_size=[1000,2000,3000], local_window=[128, 256, 512, 1024]"

    The abstract claims GALI 'achieves stable and superior performance ... without requiring input-length-specific tuning,' with Tables 1-2 as evidence. Appendix E shows that for every setting (2k-to-8k, 4k-to-8k, 4k-to-16k, 4k-to-32k, 8k-to-16k, 8k-to-32k) the GALI numbers were produced after a grid search over chunk_size in {1000,2000,3000} and local_window in {128,256,512,1024}, and the winning configuration is not reported. Thus the reported scores are per-experiment best-of-grid values, not the output of a fixed tuning-free method. The 'stable without tuning' conclusion is constructed from the very search it denies, so the claimed prediction reduces to the hyperparameter fit rather than to GALI's interpolation mechanism.

full rationale

The only load-bearing reduction I can exhibit is the Appendix E selection protocol described above. This is not a derivation-level circularity in Eq. (3): GALI defines attention logit interpolation as a convex combination of pretrained integer-interval logits plus noise, which is a genuine algorithmic construction, and its comparisons against NTK, Dyn-NTK, YaRN, SelfExtend, and ChunkLlama on LongBench, L-Eval, and PG19 are external benchmarks. There is no self-citation chain, no imported uniqueness theorem, and no ansatz smuggled in via citation. The Eq. (3) vs. Algorithm 3 mismatch (r=m-n versus r=ceil(m)-n) is an implementation inconsistency rather than circularity. The circularity score is elevated to 6 because the paper's central advertised property--no input-length-specific tuning--is supported only by numbers that were selected per target-context setting on the test benchmarks; the 'prediction' of stable, tuning-free performance therefore reduces, for the submitted evidence, to a per-experiment grid search.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on two hyperparameters (chunk size and local window) that are searched on the evaluation benchmarks, on an assumed smoothness of RoPE attention logits, on an assumption that the model fully understands its training context window, on a questionable construction of the 4k backbone from an 8k model, and on the standard RoPE properties cited from prior work. No invented physical or architectural entities are introduced.

free parameters (2)
  • chunk_size s = The best value is selected per experiment from the grid [1000, 2000, 3000].
    Controls the number of tokens grouped into one interpolation block; Appendix E Table 10 shows the grid searched for every setting, and Figure 4 shows performance depends on it.
  • local_window Lw = The best value is selected per experiment from the grid [128, 256, 512, 1024].
    Controls the preserved pretrained positional neighborhood; selected per experiment, and Figure 4 and Table 10 show its influence on the reported results.
assumptions (4)
  • domain assumption Attention logits at fractional relative positions can be approximated by linear interpolation between logits at adjacent integer relative positions plus Gaussian noise.
    Used in Eq. (3) and Algorithm 3 without error analysis or a smoothness bound for RoPE logits.
  • domain assumption The model fully understands all positional intervals within its training context window.
    Appendix C.1 states GALI assumes the model fully understands its entire training context window and admits this assumption fails for Llama2, where GALI underperforms.
  • ad hoc to paper Setting max_position_embedding to 4096 produces a model equivalent to one trained with a 4k context window.
    Section 4.1 obtains the Llama3-4k backbone this way; no training is performed, so the model's pretrained positional distribution is still 8k. Comparisons based on this construction drive the 'narrower intervals help' conclusion.
  • standard math RoPE attention depends only on relative position m-n and exhibits long-term decay.
    Quoted from Su et al. (2024) in Section 2 and used to justify logit-level interpolation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Training-Free Length Extrapolation Approach for LLMs: Greedy Attention Logit Interpolation (GALI)." pith.science (2026). https://pith.science/paper/52X73VVI

@misc{pith2026250202659,
  author       = {Pith},
  title        = {Pith review of: A Training-Free Length Extrapolation Approach for LLMs: Greedy Attention Logit Interpolation (GALI)},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/52X73VVI}},
  note         = {Machine review of arXiv:2502.02659}
}
read the original abstract

Transformer-based Large Language Models (LLMs) struggle with inputs exceeding their training context window due to positional out-of-distribution (O.O.D.) issues that disrupt attention. Existing solutions, including fine-tuning and training-free methods, face challenges like inefficiency, redundant interpolation, logit outliers, or loss of local positional information. We propose Greedy Attention Logit Interpolation (GALI), a training-free method that improves length extrapolation by greedily reusing pretrained positional intervals and interpolating attention logit to eliminate outliers. GALI achieves stable and superior performance across a wide range of long-context tasks without requiring input-length-specific tuning. Our analysis further reveals that LLMs interpret positional intervals unevenly and that restricting interpolation to narrower ranges improves performance, even on short-context tasks. GALI represents a step toward more robust and generalizable long-text processing in LLMs. Our implementation of GALI, along with the experiments from our paper, is open-sourced at https://github.com/adlnlp/Gali.

Figures

Figures reproduced from arXiv: 2502.02659 by the authors.

Figure 1
Figure 1. The overall procedure of the proposed GALI framework. The green dashed line illustrates [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. The trend of average scores across different [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Differences in attention score metrics and row-wise entropy across methods compared to the original [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Impact of local window and chunk size on per [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Visualization of long-term decay in attention [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Length distributions using Llama2 and Llama3 tokenizers. The left figure shows the distribution with [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: This is a comparison of the attention score matrices obtained using Llama3-2k and Llama3-4k for length [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Attention score distribution using Llama3-2k backbone. We omitted attention scores outside the 1st [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Attention score distribution using Llama3-4k backbone. We omitted attention scores outside the 1st [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Jet-Long: Efficient Long-Context Extension with Dynamic Bifocal RoPE

    cs.LG 2026-07 accept novelty 6.0 of 10

    Jet-Long is a tuning-free bifocal RoPE method that dynamically sets remote group size from sequence length, recovering the base model within the pretrained window and beating prior zero-shot extenders on RULER, HELMET...

Reference graph

Works this paper leans on

29 extracted references · 10 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    L -eval: Instituting standardized evaluation for long context language models

    An, C., Gong, S., Zhong, M., Zhao, X., Li, M., Zhang, J., Kong, L., and Qiu, X. L -eval: Instituting standardized evaluation for long context language models. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 14388--14411, Bangkok, Thail...

    doi:10.18653/v1/2024.acl-long.776 2024

  3. [3]

    Training-free long-context scaling of large language models

    An, C., Huang, F., Zhang, J., Gong, S., Qiu, X., Zhou, C., and Kong, L. Training-free long-context scaling of large language models. arXiv preprint arXiv:2402.17463, 2024 b

    arXiv 2024

  4. [4]

    L ong B ench: A bilingual, multitask benchmark for long context understanding

    Bai, Y., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z., Du, Z., Liu, X., Zeng, A., Hou, L., Dong, Y., Tang, J., and Li, J. L ong B ench: A bilingual, multitask benchmark for long context understanding. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Lo...

    doi:10.18653/v1/2024.acl-long.172 2024

  5. [5]

    Clex: Continuous length extrapolation for large language models

    Chen, G., Li, X., Meng, Z., Liang, S., and Bing, L. Clex: Continuous length extrapolation for large language models. arXiv preprint arXiv:2310.16450, 2023 a

    arXiv 2023

  6. [6]

    Extending context window of large language models via positional interpolation

    Chen, S., Wong, S., Chen, L., and Tian, Y. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595, 2023 b

    arXiv 2023

  7. [7]

    Longlora: Efficient fine-tuning of long-context large language models

    Chen, Y., Qian, S., Tang, H., Lai, X., Liu, Z., Han, S., and Jia, J. Longlora: Efficient fine-tuning of long-context large language models. ArXiv, abs/2309.12307, 2023 c . URL https://api.semanticscholar.org/CorpusID:262084134

    arXiv 2023

  8. [8]

    L., Zhang, C., Xu, Y., Shang, N., Xu, J., Yang, F., and Yang, M

    Ding, Y., Zhang, L. L., Zhang, C., Xu, Y., Shang, N., Xu, J., Yang, F., and Yang, M. Longrope: Extending llm context window beyond 2 million tokens. arXiv preprint arXiv:2402.13753, 2024

    arXiv 2024

Show all 29 references
  1. [9]

    Position information in transformers: An overview

    Dufter, P., Schmitt, M., and Sch \"u tze, H. Position information in transformers: An overview. Computational Linguistics, 48: 0 733--763, 2021. URL https://api.semanticscholar.org/CorpusID:231986066

    2021

  2. [10]

    Detecting hallucinations in large language models using semantic entropy

    Farquhar, S., Kossen, J., Kuhn, L., and Gal, Y. Detecting hallucinations in large language models using semantic entropy. Nature, 630 0 (8017): 0 625--630, 2024

    2024

  3. [11]

    Lm-infinite: Zero-shot extreme length generalization for large language models

    Han, C., Wang, Q., Peng, H., Xiong, W., Chen, Y., Ji, H., and Wang, S. Lm-infinite: Zero-shot extreme length generalization for large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human La...

    2024

  4. [12]

    Ruler: What's the real context size of your long-context language models? ArXiv, abs/2404.06654, 2024

    Hsieh, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., and Ginsburg, B. Ruler: What's the real context size of your long-context language models? ArXiv, abs/2404.06654, 2024. URL https://api.semanticscholar.org/CorpusID:269032933

    2024 arXiv

  5. [13]

    H., Li, D., Lin, C.-Y., et al

    Jiang, H., Li, Y., Zhang, C., Wu, Q., Luo, X., Ahn, S., Han, Z., Abdi, A. H., Li, D., Lin, C.-Y., et al. Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention. arXiv preprint arXiv:2407.02490, 2024

    2024 arXiv

  6. [14]

    Llm maybe longlm: Self-extend llm context window without tuning

    Jin, H., Han, X., Yang, J., Jiang, Z., Liu, Z., yuan Chang, C., Chen, H., and Hu, X. Llm maybe longlm: Self-extend llm context window without tuning. ArXiv, abs/2401.01325, 2024. URL https://api.semanticscholar.org/CorpusID:266725385

    2024 arXiv

  7. [15]

    The impact of positional encoding on length generalization in transformers

    Kazemnejad, A., Padhi, I., Natesan Ramamurthy, K., Das, P., and Reddy, S. The impact of positional encoding on length generalization in transformers. Advances in Neural Information Processing Systems, 36, 2024

    2024

  8. [16]

    Quickllama: Query-aware inference acceleration for large language models

    Li, J., Shi, H., Jiang, X., Li, Z., Xu, H., and Jia, J. Quickllama: Query-aware inference acceleration for large language models. arXiv preprint arXiv:2406.07528, 2024 a

    2024 arXiv

  9. [17]

    Extending context window in large language models with segmented base adjustment for rotary position embeddings

    Li, R., Xu, J., Cao, Z., Zheng, H.-T., and Kim, H.-G. Extending context window in large language models with segmented base adjustment for rotary position embeddings. Applied Sciences, 14 0 (7): 0 3076, 2024 b

    2024

  10. [18]

    Functional interpolation for relative positions improves long context transformers

    Li, S., You, C., Guruganesh, G., Ainslie, J., Ontanon, S., Zaheer, M., Sanghai, S., Yang, Y., Kumar, S., and Bhojanapalli, S. Functional interpolation for relative positions improves long context transformers. arXiv preprint arXiv:2310.04418, 2023

    arXiv

  11. [19]

    Dynamically Scaled NTK-Aware RoPE , 2023 a

    LocalLLaMA. Dynamically Scaled NTK-Aware RoPE , 2023 a . URL https://www.reddit.com/r/LocalLLaMA/comments/14mrgpr/dynamically_scaled_rope_further_increases/

    2023

  12. [20]

    NTK-Aware Scaled RoPE , 2023 b

    LocalLLaMA. NTK-Aware Scaled RoPE , 2023 b . URL https://www.reddit.com/r/LocalLLaMA/comments/14lz7j5/ntkaware_scaled_rope_allows_llama_models_to_have/

    2023

  13. [21]

    Yarn: Efficient context window extension of large language models

    Peng, B., Quesnelle, J., Fan, H., and Shippole, E. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.00071, 2023

    2023 arXiv

  14. [22]

    W., Potapenko, A., Jayakumar, S

    Rae, J. W., Potapenko, A., Jayakumar, S. M., and Lillicrap, T. P. Compressive transformers for long-range sequence modelling. ArXiv, abs/1911.05507, 2019. URL https://api.semanticscholar.org/CorpusID:207930593

    1911 arXiv

  15. [23]

    Roformer: Enhanced transformer with rotary position embedding

    Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 0 127063, 2024

    2024

  16. [24]

    Never miss a beat: An efficient recipe for context window extension of large language models with consistent" middle" enhancement

    Wu, T., Zhao, Y., and Zheng, Z. Never miss a beat: An efficient recipe for context window extension of large language models with consistent" middle" enhancement. arXiv preprint arXiv:2406.07138, 2024

    2024 arXiv

  17. [25]

    Efficient streaming language models with attention sinks

    Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453, 2023

    2023 arXiv

  18. [26]

    A., Oğuz, B., Khabsa, M., Fang, H., Mehdad, Y., Narang, S., Malik, K., Fan, A., Bhosale, S., Edunov, S., Lewis, M., Wang, S., and Ma, H

    Xiong, W., Liu, J., Molybog, I., Zhang, H., Bhargava, P., Hou, R., Martin, L., Rungta, R., Sankararaman, K. A., Oğuz, B., Khabsa, M., Fang, H., Mehdad, Y., Narang, S., Malik, K., Fan, A., Bhosale, S., Edunov, S., Lewis, M., Wang, S., and Ma, H. Effective long-context scaling o...

    2023

  19. [27]

    Base of rope bounds context length

    Xu, M., Men, X., Wang, B., Zhang, Q., Lin, H., Han, X., et al. Base of rope bounds context length. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  20. [28]

    Attention entropy is a key factor: An analysis of parallel context encoding with full-attention-based pre-trained language models

    Zhang, Z., Wang, Y., Huang, X., Fang, T., Zhang, H., Deng, C., Li, S., and Yu, D. Attention entropy is a key factor: An analysis of parallel context encoding with full-attention-based pre-trained language models. arXiv preprint arXiv:2412.16545, 2024

    2024 arXiv

  21. [29]

    Pose: Efficient context window extension of llms via positional skip-wise training

    Zhu, D., Yang, N., Wang, L., Song, Y., Wu, W., Wei, F., and Li, S. Pose: Efficient context window extension of llms via positional skip-wise training. ArXiv, abs/2309.10400, 2023. URL https://api.semanticscholar.org/CorpusID:262053659

    2023 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.