Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

SELF: Self-Extend the Context Length With Logistic Growth Function

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Logistic grouping stretches usable LLM context by up to 12%

desk verdict A real but incremental idea—logistic group sizing for Self-Extend—is undermined by benchmark comparisons that change group size and context capacity at the same time. read the letter →

arxiv 2505.17296 v1 pith:Q6MVIQO5 submitted 2025-05-22 cs.CL cs.LG

Phat Thanh Dang , Saahil Thoppay , Wang Yang , Qifan Wang , Vipin Chaudhary , Xiaotian Han This is my paper · ORCID
classification cs.CLcs.LG
keywords contextwindowextensionlogisticgrowthfunctiongroupattentionRoPEpositionencodingSelf-ExtendLongBenchLEval
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a training-free trick can push a pretrained LLM to handle contexts longer than its training window: instead of grouping distant tokens into fixed-size chunks as Self-Extend does, group them into chunks whose size grows with distance according to a logistic function, while keeping nearby tokens at exact positions. The authors argue that token relevance falls with distance, so progressively larger groups lose little information while freeing positional capacity. On LEval and LongBench they report gains over Self-Extend of up to 12% on Qwen-7B, 6.4% on Llama-2-7B summarization, and 5.4% on reading comprehension, with no fine-tuning and roughly unchanged runtime. If correct, the method gives a cheap, model-agnostic context-extension option that also behaves predictably as group size increases.

What carries the argument

The load-bearing object is the logistic grouping function $f(x)=\lfloor C e^{rx}/(C+e^{rx}-1)\rfloor$, which maps a group index $x$ to a group size, combined with the neighbor window. Inside the window, tokens keep their true positions; outside, consecutive tokens are assigned group position indices according to cumulative sums of $f$, and the query-side index is defined so that the relative position immediately after the window remains $W$. Because $f$ is monotone with bounded maximum $C$, SELF creates many small early groups and only reaches large groups far from the query, letting far tokens share positional indices while nearby tokens stay exact. A closed-form inverse $f^{-1}(y)=\lfloor (\ln(Cy-y)-\ln(C-y))/r\rfloor$ lets the group-index array be built in $O(n+C)$ work with $O(C)$ parallel span, so the extension costs almost no extra runtime.

What would settle it

In a passkey retrieval test, place the key just outside the neighbor window and again deep in the tail where SELF's groups have reached capacity; if SELF's retrieval accuracy at the tail is not better than fixed-size Self-Extend's at the same maximum group size, the monotone-decay assumption that motivates the logistic schedule is not doing the work.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that the position-remapping idea behind Self-Extend does not require a fixed group size: a smooth, bounded growth curve for group sizes is more compatible with how language models use context. The authors introduce SELF, which assigns group sizes by $f(x)=\lfloor C e^{rx}/(C+e^{rx}-1)\rfloor$, with capacity $C$ and growth rate $r$, and pairs it with a neighbor window of width $W$ where tokens keep their original positions. They report that with the same maximum group size SELF achieves lower perplexity than Self-Extend on PG19 and better or comparable accuracy on most LongBench and LEval tasks, with the largest margins on Qwen-7B and Llama-2-7B. They conclude that SELF works better than Self-Extend at equal capacity and that its accuracy changes more predictably as group size grows, because early groups overlap and the jump from exact positions to maximum group size is gradual.

Load-bearing premise

The load-bearing premise is that the relevance of a token to the current query declines with distance, so tokens far away can be pooled into progressively larger groups that share one positional index without losing what the task needs.

Editorial extensions

If this is right

  • A pretrained model can process prompts several times longer than its training window without fine-tuning, with reported gains up to 12% over fixed-size grouping on LEval.
  • For the same maximum group size, SELF yields lower perplexity than Self-Extend on PG19 across many context lengths, so users can afford larger group sizes and hence longer context extension before quality drops.
  • On tasks where the context length is near the training window, such as MultiFieldQA, 2WikiMultihopQA, HotpotQA, and TREC, SELF tends to match or beat fixed grouping, extending the practical range of the original model.
  • Because the group-index construction is parallel, the method preserves roughly the same runtime as the original model, so the accuracy gain is not bought with additional inference cost.
  • The paper's own results show the benefit is model- and task-dependent: Llama-2-13B and DeepSeek-R1-Distill-Qwen-7B do not improve, and CodeU remains unsolved by all methods.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial: the paper motivates SELF with monotone distance decay but never isolates that mechanism; a direct test would compare SELF with a logistic curve whose growth rate is high enough to mimic fixed chunks, checking whether smoothness alone reproduces the gains.
  • Editorial: if relevance decay is the true driver, SELF should also soften 'lost-in-the-middle' failures, since middle-distance tokens get finer granularity than far-tail tokens; the paper does not report any middle-position analysis.
  • Editorial: because SELF only remaps positions in the attention layer, it could be composed with other position-extension methods such as NTK-aware scaling or YaRN, which change the base frequency of RoPE rather than the index grouping; that combination is untested.
  • Editorial: the degradation on reasoning models suggests an open extension: identify how reinforcement-learned exact-position dependence reacts to grouped positions, either by ablating with non-reasoning checkpoints or by training with group indices during RL.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes SELF, a training-free context-length extension method that modifies the Self-Extend (LongLM) grouping scheme. Instead of fixed-size groups beyond a neighbor window, group sizes grow according to a logistic growth function f(x)=floor(C e^{rx}/(C+e^{rx}-1)), with the capacity C as the maximum group size. The authors evaluate SELF on PG19 perplexity (Table 1), LongBench (Table 2), and LEval (Table 3) using Llama-2-7B-chat, Llama-2-13B-chat, Qwen-7B, and DeepSeek-R1-Distill-Qwen-7B, and report improvements of up to 12% on LEval, 6.4% on LongBench summarization, and 5.4% on reading comprehension. They conclude that SELF works better than SE at the same capacity and that its behavior is more predictable as group size increases.

Significance. If the central claim were established, SELF would be a modest but useful incremental contribution: a drop-in, training-free replacement for Self-Extend's grouping schedule with no change to runtime asymptotics. The paper has real strengths: an honest Limitations section, released code, a sensible perplexity-based investigation of grouping behavior (Figure 2 and Table 1), and a clear parallel-construction analysis in Section 4.3 with O(n+C) work and O(C) span. However, the central claim is not established by the benchmark experiments, because the SELF runs use a fourfold larger maximum group size and a much longer context window than the SE baselines, while the only same-capacity evidence shows small and partially reversed differences. The paper's own negative results for Llama-2-13B and DeepSeek-R1-Distill-Qwen-7B further undermine the generality of the conclusion as stated.

major comments (4)
  1. [§5.2.1, §5.2.2, Conclusion] The benchmark comparisons in Table 2 (LongBench) and Table 3 (LEval) confound the grouping schedule with a fourfold increase in group capacity: SELF is run with C=32 and a 100k context window, while the SE baselines use group sizes 6 and 8 with 16k/25k windows, as the paper itself states in §5.2.1 ('we used a much bigger group size (C = 32)'). Consequently, the claimed gains of up to 12% on LEval (Qwen-7B), 6.4% on LongBench summarization, and 5.4% on reading comprehension cannot be attributed to the logistic grouping schedule rather than to the larger capacity. The Conclusion's statement that 'SELF works better than SE when dealing with the same capacity' is not tested by these tables; it is tested only in Table 1, where the differences are small. A same-capacity comparison (e.g., SELF with C=6/8 at 16k/25k, or SE with C=32 at 100k, all other hyperparameters fixed) is necessary to support the central claim.
  2. [§5.1, Table 1] The only same-capacity evidence, Section 5.1/Table 1, does not support the strength of the conclusion: at C=32 and length 4096 the perplexity difference is 7.119 (SELF) versus 7.141 (SE), and at C=64 and length 16384 the ordering reverses (SE 8.041, SELF 8.068). The paper's own fourth bullet ('the scores are basically the same for SE and SELF' for very long sequences) concedes that the advantage vanishes in the extended-context regime. Given single runs and no significance testing, the PG19 results establish at most a marginal and regime-limited effect, and they do not justify the unqualified capacity-based claim in the Conclusion.
  3. [§5.2.2, Table 3] Table 3 contradicts the general claim that SELF outperforms SE: on Llama-2-13B-chat, SELF is worse than SE on every LEval task (average 39.68 versus 41.49), and on DeepSeek-R1-Distill-Qwen-7B the average drops from 47.07 to 37.68. Section 5.2.2 acknowledges the 13B result but offers no explanation ('We could not come up with the reason'), so the reader is left without conditions under which the method applies. The abstract and conclusion should be qualified to the models and tasks where the effect holds, or the paper should provide an analysis of when the underlying distance-decay premise holds.
  4. [§2, §4.2] The load-bearing premise in Section 2, that token relevance decreases monotonically with distance and that progressively larger groups therefore lose little information, is never directly tested: the experiments vary C, W, and model, but no experiment isolates whether logistic grouping preserves task-relevant information better than fixed grouping at the same number of effective positions. A within-capacity ablation that varies the growth rate r (including the limit r tending to infinity, which recovers SE) on a retrieval or multi-document QA task would test the smoothness assumption directly and would also substantiate the 'more predictable' trade-off claimed in Figure 5.
minor comments (6)
  1. [Abstract vs. Introduction] The Abstract reports up to 12% improvement on LEval, while the Introduction reports 'up to 8% (specifically Qwen2-7B)'; the discrepancy should be reconciled and the numbers tied to specific rows of Table 3.
  2. [Throughout] There are numerous typos and incomplete reference entries, including 'long-conetxt' (§5), 'orginial' (§5.1), duplicated bibliography entries (Jin et al. 2024a/b; Chen et al. 2023b/c; Peng et al. 2023a/b; Xiao et al. 2023/2024b; Han et al. 2024a/b), and a reference for 'How long can open-source llms truly promise on context length?' that lacks venue and full author list.
  3. [§4.2, Eq. (3)] Equation (3) is unclear: the condition 'if i ≤ W' and the claim that the constant c is 'completely covered by the neighbor window' do not precisely specify which positions receive the group-query index; the intended range of the piecewise definition should be reformulated.
  4. [§4.2, extended-length formula] The formula L′ = Σ_{i=1}^{L+max(F)-W-F_W} f(i) lacks parentheses in the upper limit and is introduced without a derivation, so the reader cannot verify the claimed total context capacity without reconstructing the argument.
  5. [Tables 2 and 3] Hyperparameters for the benchmark runs are incomplete: r and W are not reported for Tables 2 and 3, although Table 1 fixes r=0.02 and Figure 5 uses W=1024, which prevents reproduction of the main results from the text alone.
  6. [Figure 1] The footnote to Figure 1 ('This is an oversimplification of how the method works. More details will be explained in Our Proposal section') is an unusual self-referential disclaimer; it should be removed or the figure corrected so that the caption itself is accurate.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: SELF's gains are measured against external benchmarks, and the logistic grouping schedule is not fit to the reported results.

full rationale

The paper's derivation chain is self-contained in the relevant sense: SELF defines a grouping function f(x)=floor(C e^{rx}/(C+e^{rx}-1)) from an explicit distance-decay intuition, and then tests that definition on external benchmarks (PG19, LongBench, LEval). No benchmark outcome is used to define f, and no parameter is fitted to the tasks on which improvements are claimed. The only self-citation is to Self-Extend (Jin et al., 2024a), which includes current co-author Xiaotian Han, but that citation supplies the baseline method and its reported numbers, not a uniqueness theorem or an unverified premise that forces SELF's design. The more substantial concern is experimental control, not circularity: Section 5.2.1 states 'we decided instead of using small group size of 6 and 8 like in LongLM, we used a much bigger group size (C = 32)', so the LongBench and LEval comparisons in Tables 2 and 3 vary both the grouping schedule and the maximum group size, confounding the claimed 12% and 6.4% gains. That is a fairness-of-comparison problem, not a case of the prediction being equivalent to the input by construction. The same-capacity evidence in Table 1 is perplexity-only and shows only modest differences that partly reverse at long contexts, which further weakens the conclusion but does not make it circular. The paper's own Limitations section acknowledges the limited model and benchmark coverage, the CodeU failure, and the degraded behavior on reasoning models; these admissions are consistent with an empirical claim that is falsifiable rather than a renaming of known results. Overall, no specific reduction of a predicted result to a fitted parameter, self-citation, or definitional identity can be exhibited, so the circularity score is low.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The method introduces a grouping schedule, not a new entity, force, dimension, or conserved quantity. The main free parameters are the logistic capacity C, growth rate r, and neighbor window W. The design rests on the distance-relevance assumption and on the inherited grouped-attention assumption from Self-Extend.

free parameters (3)
  • C (logistic capacity, maximum group size) = 16, 32, 64 in PG19; C=32 in LongBench and LEval
    Chosen after perplexity analysis on PG19; not derived from theory.
  • r (logistic growth rate) = 0.02
    Fixed at 0.02 for all experiments; chosen by hand with no tuning protocol reported.
  • W (neighbor window size) = not reported for main experiments; W=1024 shown in Figure 5
    Neighbor window width is a free hyperparameter that controls how many tokens use exact positions; its value is not given for LongBench and LEval runs.
assumptions (3)
  • domain assumption Token relevance decreases monotonically with distance
    Section 2 states this intuition; it justifies progressively larger groups and is not independently validated.
  • domain assumption Grouped attention preserves enough information for the task
    Inherited from Self-Extend; SELF assigns a single position to groups of far tokens and relies on a neighbor window for close tokens (Section 4.1).
  • ad hoc to paper Logistic function is a suitable grouping schedule
    Section 2 picks logistic because it is monotone and bounded; no comparison with other bounded monotone functions is given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SELF: Self-Extend the Context Length With Logistic Growth Function." pith.science (2026). https://pith.science/paper/Q6MVIQO5

@misc{pith2026250517296,
  author       = {Pith},
  title        = {Pith review of: SELF: Self-Extend the Context Length With Logistic Growth Function},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q6MVIQO5}},
  note         = {Machine review of arXiv:2505.17296}
}
read the original abstract

Large language models suffer issues when operated on long contexts that are larger than their training context length due to the standard position encoding for tokens in the attention layer. Tokens a long distance apart will rarely have an effect on each other and long prompts yield unexpected results. To solve this problem, we propose SELF (Self-Extend the Context Length With Logistic Growth Function): a solution of grouping consecutive tokens at varying group sizes using a logistic capacity equation combined with a constant group size at smaller relative distances. Our model had an increase in performance of up to 12% compared to the LongLM extension method in LEval (specifically on the Qwen model). On summarization related tasks in LongBench, our model performed up to 6.4% better than LongLM (specifically on the Llama-2-7b model). On reading comprehension tasks from LEval, our model performed up to 5.4% better than the LongLM. Our code is available at https://github.com/alexeipc/SELF-LLM.

Figures

Figures reproduced from arXiv: 2505.17296 by the authors.

Figure 1
Figure 1. Illustration of our method in extending context [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Illustration about relation between neighbor [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Illustration of the relation between GK and GQ knowing that the relative position right after the neighbor window has to be W. of the neighbor window, because the relative posi￾tions inside the neighbor window will range from 0 to W − 1 (see illustration in [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Illustration of the algorithm grouping the indices using the function [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Compare the trade off with different group [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Jet-Long: Efficient Long-Context Extension with Dynamic Bifocal RoPE

    cs.LG 2026-07 accept novelty 6.0 of 10

    Jet-Long is a tuning-free bifocal RoPE method that dynamically sets remote group size from sequence length, recovering the base model within the pretrained window and beating prior zero-shot extenders on RULER, HELMET...

Reference graph

Works this paper leans on

44 extracted references · 5 canonical work pages · cited by 1 Pith paper

  1. [1]

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, and 1 others. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219

    arXiv 2024

  2. [2]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  3. [3]

    Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. 2023. https://arxiv.org/abs/2307.11088 L-eval: Instituting standardized evaluation for long context language models . Preprint, arXiv:2307.11088

    arXiv 2023

  4. [4]

    Tao Bai, Jinqi Luo, Jun Zhao, Bihan Wen, and Qian Wang. 2021. https://arxiv.org/abs/2102.01356 Recent advances in adversarial training for adversarial robustness . Preprint, arXiv:2102.01356

    arXiv 2021

  5. [5]

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024 a . https://arxiv.org/abs/2308.14508 Longbench: A bilingual, multitask benchmark for long context understanding . Preprint, arXiv:2308.14508

    arXiv 2024

  6. [6]

    Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, and 1 others. 2024 b . Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks. arXiv preprint arXiv:2412.15204

    arXiv 2024

  7. [7]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, and 12 others. 2020. https://arxiv.org/abs/2005.14165 Lan...

    arXiv 2020

  8. [8]

    Guanzheng Chen, Xin Li, Zaiqiao Meng, Shangsong Liang, and Lidong Bing. 2023 a . https://arxiv.org/abs/2310.16450 Clex: Continuous length extrapolation for large language models . arXiv preprint arXiv:2310.16450

    arXiv 2023

Show all 44 references
  1. [10]

    Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. 2023 c . Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595

    2023 arXiv

  2. [11]

    Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. 2023 d . Longlora: Efficient fine-tuning of long-context large language models. arXiv preprint arXiv:2309.12307

    2023 arXiv

  3. [12]

    Le, and Ruslan Salakhutdinov

    Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V. Le, and Ruslan Salakhutdinov. 2019. https://arxiv.org/abs/1901.02860 Transformer-xl: Attentive language models beyond a fixed-length context . Preprint, arXiv:1901.02860

    2019 arXiv

  4. [13]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, and 181 others. 2025. https://arxiv.org/abs/2501.129...

    2025 arXiv

  5. [14]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  6. [15]

    Chi Han, Qifan Wang, Hao Peng, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. 2024 a . https://arxiv.org/abs/2308.16137 Lm-infinite: Zero-shot extreme length generalization for large language models . Preprint, arXiv:2308.16137

    2024 arXiv

  7. [16]

    Chi Han, Qifan Wang, Hao Peng, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. 2024 b . Lm-infinite: Zero-shot extreme length generalization for large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Lin...

    2024

  8. [17]

    Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2023. Longllmlingua: Accelerating and enhancing llms in long context scenarios via prompt compression. arXiv preprint arXiv:2310.06839

    2023 arXiv

  9. [19]

    Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zirui Liu, Chia-Yuan Chang, Huiyuan Chen, and Xia Hu. 2024 b . https://arxiv.org/abs/2401.01325 Llm maybe longlm: Self-extend llm context window without tuning . Preprint, arXiv:2401.01325

    2024 arXiv

  10. [20]

    Gonzalez, Ion Stoica, Xuezhe Ma, and Hao Zhang

    Dacheng Li, Rulin Shao, Anze Xie, Ying Sheng, Lianmin Zheng, Joseph E. Gonzalez, Ion Stoica, Xuezhe Ma, and Hao Zhang. 2023 a . https://lmsys.org/blog/2023-06-29-longchat How long can open-source llms truly promise on context length?

    2023

  11. [21]

    Yucheng Li, Bo Dong, Chenghua Lin, and Frank Guerin. 2023 b . Compressing context to enhance inference efficiency of large language models. arXiv preprint arXiv:2310.06201

    2023 arXiv

  12. [22]

    Jiashuo Liu, Zheyan Shen, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui. 2023. https://arxiv.org/abs/2108.13624 Towards out-of-distribution generalization: A survey . Preprint, arXiv:2108.13624

    2023 arXiv

  13. [23]

    Amirkeivan Mohtashami and Martin Jaggi. 2023. Landmark attention: Random-access infinite context length for transformers. arXiv preprint arXiv:2305.16300

    2023 arXiv

  14. [24]

    Bowen Peng and Jeffrey Quesnelle. 2023. Ntk-aware scaled rope allows llama models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation

    2023

  15. [26]

    Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2023 b . Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.00071

    2023 arXiv

  16. [27]

    Smith, and Mike Lewis

    Ofir Press, Noah A. Smith, and Mike Lewis. 2022. https://arxiv.org/abs/2108.12409 Train short, test long: Attention with linear biases enables input length extrapolation . Preprint, arXiv:2108.12409

    2022 arXiv

  17. [28]

    Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, Chloe Hillier, and Timothy P Lillicrap. 2019. https://arxiv.org/abs/1911.05507 Compressive transformers for long-range sequence modelling . arXiv preprint

    2019 arXiv

  18. [29]

    Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong...

    2024 arXiv

  19. [30]

    Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. 2023. https://arxiv.org/abs/2104.09864 Roformer: Enhanced transformer with rotary position embedding . Preprint, arXiv:2104.09864

    2023 arXiv

  20. [31]

    Qwen Team. 2025. https://qwenlm.github.io/blog/qwen2.5-1m/ Qwen2.5-1m: Deploy your own qwen with context length up to 1m tokens

    2025

  21. [33]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and 1 others. 2023 b . Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  22. [34]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2023. https://arxiv.org/abs/1706.03762 Attention is all you need . Preprint, arXiv:1706.03762

    2023 arXiv

  23. [35]

    Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, and Maosong Sun. 2024 a . Infllm: Training-free long-context extrapolation for llms with an efficient context memory. In The Thirty-eighth Annual Conference on Neural Information Proce...

    2024

  24. [37]

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024 b . https://arxiv.org/abs/2309.17453 Efficient streaming language models with attention sinks . Preprint, arXiv:2309.17453

    2024 arXiv

  25. [38]

    Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, a...

    2023 arXiv

  26. [39]

    Chejian Xu, Wei Ping, Peng Xu, Zihan Liu, Boxin Wang, Mohammad Shoeybi, and Bryan Catanzaro. 2025. From 128k to 4m: Efficient training of ultra-long context large language models. arXiv preprint

    2025

  27. [40]

    Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. 2023. Retrieval meets long context large language models. arXiv preprint arXiv:2310.03025

    2023 arXiv

  28. [41]

    Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. https://arxiv.org/abs/2010.11934 mt5: A massively multilingual pre-trained text-to-text transformer . Preprint, arXiv:2010.11934

    2021 arXiv

  29. [42]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025 a . Qwen3 technical report. arXiv preprint arXiv:2505.09388

    2025 arXiv

  30. [43]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, and Jialong Tang... 2024. https://arxiv.org/abs/2407.10671 Qwen2 technical report . Preprint, arXiv:2407.10671

    2024 arXiv

  31. [44]

    An Yang, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoyan Huang, Jiandong Jiang, Jianhong Tu, Jianwei Zhang, Jingren Zhou, Junyang Lin, Kai Dang, Kexin Yang, Le Yu, Mei Li, Minmin Sun, Qin Zhu, Rui Men, Tao He, and 9 others. 2025 b . Qwen2.5-1m technical report. arXiv p...

    2025 arXiv

  32. [45]

    Jiajin Zhang, Hanqing Chao, Amit Dhurandhar, Pin-Yu Chen, Ali Tajer, Yangyang Xu, and Pingkun Yan. 2022 a . https://arxiv.org/abs/2212.00850 When neural networks fail to generalize? a model sensitivity perspective . Preprint, arXiv:2212.00850

    2022 arXiv

  33. [46]

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2...

    2022 arXiv

  34. [47]

    Dawei Zhu, Nan Yang, Liang Wang, Yifan Song, Wenhao Wu, Furu Wei, and Sujian Li. 2024. https://arxiv.org/abs/2309.10400 Pose: Efficient context window extension of llms via positional skip-wise training . Preprint, arXiv:2309.10400

    2024 arXiv

  35. [48]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  36. [49]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.