Pith. sign in

REVIEW 4 major objections 6 minor 73 references

Breaking the Stage Barrier: A Novel Single-Stage Approach to Long Context Extension for Large Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A single training stage reaches 128k context by giving each attention head its own RoPE base.

desk verdict A plausible and budget-matched single-stage long-context recipe whose headline margin is partly fitted to the test benchmark; worth a serious referee but needs a stronger baseline, error bars, and a reproducible base search. read the letter →

arxiv 2412.07171 v1 pith:UG5YCOF7 submitted 2024-12-10 cs.CL

Haoran Lian , Junmin Chen , Wei Huang , Yizhe Xiong , Wenping Hu , Guiguang Ding , Hui Chen , Jianwei Niu
show 3 more authors
Zijia Lin Fuzheng Zhang Di Zhang
This is my paper · ORCID
classification cs.CL
keywords long-contextextensionRoPEbasefrequencyhead-adaptivepositionalencodingsingle-stagecontinualpretrainingmulti-stageneedle-in-a-haystackRULERbenchmarkLLMcontextlength
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LLMs trained on short contexts lose accuracy on long inputs, and the standard fix is to grow the context window through several expensive, hand-tuned training stages. This paper claims that the staged schedule can be collapsed into one stage: assign each attention head a different RoPE base frequency, run one continual-pretraining pass at the target 128k context, and the model learns long-context behavior that matches or beats staged training. If true, the result matters because it removes the need to design and tune intermediate context lengths, data mixes, and checkpoints. The reported evidence covers four benchmarks, with the largest gain on the upgraded needle-in-a-haystack test (86.82 average for HARPE versus 81.36 for the staged adjusted-base-frequency baseline).

What carries the argument

The central mechanism is head-adaptive rotary position encoding (HARPE). In standard RoPE one base frequency controls the rotation angles for all attention heads; HARPE instead assigns each head its own base, effectively letting different heads specialize to different effective context lengths during a single training run. The main experiments use a peak–valley search (Algorithm 1) that selects bases whose attention-waveform valleys overlap other bases' peaks, producing the per-head bases in Table 2 (roughly 1M to 4.54M with a 30k search stride).

What would settle it

Reproduce Algorithm 1 with explicit peak and valley functions to see whether it yields the exact Table 2 base set; then, with the same 6B-token budget and data, compare HARPE against a multi-stage adjusted-base-frequency schedule that uses the paper's own carefully selected data ordering from Table 1. If that baseline reaches or exceeds the HARPE average of 86.82 on the upgraded needle-in-a-haystack test, the claim that a single stage breaks the stage barrier would fail.

Watch

Extended reading notes

Core claim

The paper's central claim is that long-context capability does not require gradual context growth during continual pretraining. By giving each attention head its own rotary position encoding base value, chosen so that the peaks of one head's attention waveform fill the valleys of another's, a single 6B-token training run at 128k context equips Llama2-7B with long-context modeling that the authors report as outperforming the multi-stage adjusted-base-frequency baseline by 5.46% on the upgraded needle-in-a-haystack test, while preserving short-context performance on MMLU, HellaSwag, ARC-C, PIQA, and TriviaQA.

Load-bearing premise

The claimed advantage over multi-stage training rests on the paper's three-stage adjusted-base-frequency baseline being a representative multi-stage schedule, and on the reported per-head base values being reproducible from the search algorithm.

Editorial extensions

If this is right

  • The training pipeline reduces to one stage: choose a per-head base set, run 6B tokens at 128k context, and evaluate.
  • Long-context gains do not have to come at the cost of short-context accuracy; the reported averages on five short-context tasks are comparable to or slightly better than the baselines.
  • The approach changes only the RoPE base per head during training, adding no new modules and no inference-time machinery.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the result holds at larger scales, per-head RoPE bases could turn long-context extension from a scheduling problem into a base-set search problem, which is easier to automate.
  • Editorial inference: the peak–valley complementarity idea predicts that small-base heads handle short-range dependencies and large-base heads handle long-range ones; ablating either end of the base range should degrade the corresponding length regime.
  • Editorial inference: the 5.46% margin over the staged baseline may partly reflect the baseline's data schedule, so a fair cross-check would compare HARPE against production-style multi-stage schedules with matched token budgets.
  • Editorial inference: the paper leaves supervised fine-tuning and other later training stages unexplored; whether per-head bases survive those stages or need to be reset to a single base is an open question.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Head-Adaptive Rotary Position Encoding (HARPE), a single-stage continual pretraining method that assigns different RoPE base frequencies to different attention heads and directly trains Llama2-7B at the target 128k context length with a fixed 6B-token budget. The base set is chosen either by uniform distribution or by a peak-valley complementarity search (Algorithm 1), with the stride-30k configuration selected after ablations. The authors evaluate on sliding-window perplexity (Proof-pile, GovReport), an upgraded eight-task Needle-in-a-Haystack test, five short-context benchmarks, and the RULER benchmark, reporting that HARPE matches or outperforms the tested multi-stage ABF baseline and other long-context methods, including a 5.46% NiaH margin over multi-stage ABF. Section 7 notes that the method has only been studied for continual pretraining, not for supervised fine-tuning.

Significance. If substantiated, HARPE would provide a meaningful simplification of long-context adaptation: it removes the multi-stage scheduling and manual tuning that current pipelines require, while preserving short-context performance. The paper has clear strengths: the main comparisons are budget-matched at 6B tokens per method; the upgraded NiaH evaluation covers eight tasks and six lengths; the NiaH advantage is consistent across all seven base-configuration variants in Table 6; and Table 7 shows a competitive RULER score. These strengths make the empirical result worth taking seriously. However, the central claim that HARPE 'consistently matches or surpasses' multi-stage methods is not yet fully established, because the multi-stage baseline is a single hand-picked schedule, the peak-valley search is not reproducible as written, and the headline configuration was selected on the same benchmark used to report the main result.

major comments (4)
  1. [Sec. 3.2, Algorithm 1] Algorithm 1 is not self-contained: the functions fp(b) and fv(b), which map a base to 'peak positions' and 'valley positions in attention waveforms', are defined only by name in lines 1-2. The paper gives no formula, no specification of which attention scores are used (initial model, theoretical RoPE decay, or trained attention weights), and no exact pointer to the definition in Chen et al. (2024b). Since Table 2, the stride-30k base set, and therefore every HARPE result depend on this search, a reader cannot reproduce the central configuration. Please provide explicit definitions of fp and fv, or quote the exact equations from the cited paper and state any adaptations.
  2. [Sec. 4.1, Table 1 and Table 4] The claim that HARPE 'consistently matches or surpasses' multi-stage methods rests on a single ABF Multi-Stage schedule (1m/32k, 2m/64k, 5m/128k, 6B tokens). Table 1 shows that the same three-stage recipe yields 67.83 or 81.36 on the upgraded NiaH test depending only on how tokens are scheduled, so one schedule does not establish the upper envelope of multi-stage training. A tuned multi-stage pipeline with different stage lengths, token allocations, or stage counts could shrink or erase the reported 5.46% margin, and the production models cited in Section 1 use exactly such tuned schedules. Please compare HARPE against additional multi-stage schedules or restrict the claim to the tested baseline.
  3. [Sec. 5.2, Table 6; Sec. 5.1, Table 4] The stride-30k peak-valley configuration was selected because it achieved the highest average score on the upgraded NiaH test (86.82 in Table 6), and the same benchmark and same configuration then produce the headline result in Table 4 (86.82 vs. 81.36). This selection-on-evaluation makes the reported margin optimistically biased; adjacent configurations in Table 6 differ by up to 3.14 points. Please validate the chosen configuration on a benchmark that was not used for hyperparameter selection, or explicitly report the selection procedure and quantify its bias, for example with a validation split or nested evaluation.
  4. [Sec. 4.4 and Tables 3-5] No run-to-run variance is reported for any experiment. For the main 'matches or surpasses' claim, the PPL differences are tiny (Table 3: 3.02 vs. 3.03 on Proof-pile; 3.54 vs. 3.57 on GovReport) and the short-context average differences are also small (Table 5: 62.75 vs. 62.69). With a single run, these differences may be within normal noise for 7B continual pretraining. Please provide at least three seeds, or error bars, or a statistical analysis for the central comparisons.
minor comments (6)
  1. [Abstract and Section 1] There are typos in the abstract and body: 'pertaining' should be 'pretraining' in the abstract, 'revist' appears in Section 3.1, and 'Adjused' appears in Section 1; please copyedit throughout.
  2. [Section 2] Section 2 contains an unresolved citation '?' after (Cai et al., 2024); please replace it with a proper reference or delete it.
  3. [Section 3.2 and Table 2] Since Llama2-7B has 32 attention heads per layer and many layers, state explicitly whether the 32 bases in Table 2 are reused in every layer or assigned per head across all layers.
  4. [Table 1] Define what 'Uniform 2B Tokens' and 'Carefully Selected' mean (per-stage token allocation, data mixture, or both) and state whether the 'Carefully Selected' pipeline is the configuration used for the ABF Multi-Stage results in Tables 3-5.
  5. [Section 5.2 and Table 6] The rows labeled 'same stride ascending order' and 'descending order' are described in the text as uniform-distribution experiments, but the table label is unclear; specify which stride (30k?) and which ordering refers to.
  6. [Section 4.3] Specify how the reported PPL values are aggregated over the 2k-128k lengths and document the sliding-window settings (window size, stride, and whether overlapping windows are used) precisely.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: HARPE's reported gains are empirical benchmark results from a transparent ablation, not quantities that reduce by construction to the paper's inputs.

full rationale

The paper does not claim to derive HARPE's performance from first principles; it proposes a per-head RoPE base assignment and reports benchmark measurements. The only configuration choices that could look like fitted inputs are the base stride (s=30k in Sec. 4.2) and the peak-valley base set, but Sec. 5.2 explicitly presents a five-stride ablation on the upgraded NiaH benchmark and states that stride 30k performed best, so the final comparison in Tab. 4 is a disclosed model-selection result rather than a hidden fit. Choosing a hyperparameter on a benchmark and then reporting that benchmark is a commonplace selection-bias concern, not a circular reduction: the reported 86.82 average is an empirical value for that configuration, not algebraically entailed by the selection rule. The multi-stage ABF baseline is also a hand-picked schedule rather than a tuned production system, and Tab. 1 shows that schedule and data choice can swing NiaH by 13.5 points, but this bears on baseline representativeness and the strength of the 'matches or surpasses multi-stage' conclusion, not on circularity. The paper's self-citations in the introduction and related work are ordinary references to prior author work and do not supply any load-bearing premise. One genuine support gap is that Algorithm 1's peak and valley functions fp and fv are never defined, which makes the reported base set unreproducible; that is an omitted-support and reproducibility problem, not circularity, because the central claim is evaluated against external benchmarks whose outcomes are not defined in terms of the algorithm's internals. Overall, no equation or claimed prediction reduces to its own input, so the circularity score is 0.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim rests on empirical choices rather than derivations: a benchmark-selected stride, a hand-chosen base range, an under-specified greedy search, and the asserted stage-simulation mechanism. The only mathematical content is the standard RoPE formulation. No invented entities are introduced.

free parameters (4)
  • Peak-valley search stride s = 30k (winner of 10k, 20k, 30k, 40k, 50k sweep)
    Selected as the best performer on the same upgraded NiaH benchmark that produces the headline claim (Tab 6 selection, Tab 4 headline). This is the main benchmark-fitted choice.
  • Base range bmin, bmax = 1m and 5m
    Hand-chosen to mirror the ABF base values used for a 128k target context (Sec 4.2); not derived from any stated principle.
  • Per-head base set Bs = 32 values listed in Tab 2 (1.00m to 4.54m)
    Output of the greedy search in Algorithm 1; it depends on the peak/valley functions fp and fv, which the paper never defines, so the set is effectively a partially unspecified modeling choice.
  • Multi-stage baseline token split = Unspecified; paper states 6B tokens total for all models
    The number of tokens per stage in the three-stage ABF baseline is not given, and this split directly controls baseline strength (Sec 4.1, Sec 4.4).
assumptions (5)
  • standard math RoPE formulation, Eqs. (1)-(5) (Su et al. 2024a)
    Background definition of rotary position encoding, standard in the field.
  • domain assumption Larger RoPE base b mitigates attention decay for distant tokens and enables longer effective context (Xiong et al. 2023)
    Invoked in Sec 3.1 to justify base scaling; a widely used empirical claim from the cited ABF work.
  • domain assumption Different attention heads can acquire distinct knowledge during training (Li et al. 2023)
    Used in Sec 1 and Sec 3.2 as the motivation for distributing 'stages' across heads; borrowed from a citation, not re-established here.
  • ad hoc to paper Heads with different RoPE bases trained at full 128k length simulate a multi-stage curriculum, so single-stage training suffices
    The central mechanism of HARPE (Sec 3.2 and Fig. 1). Asserted without head-level analysis; if false, the method might still work empirically, but the claimed principle would not.
  • ad hoc to paper Maximizing distance between peaks of one base's attention waveform and valleys of another's is a good coverage objective (Algorithm 1, from Chen et al. 2024b)
    Adopted from the cited inference-time work and applied to training-time base selection; the functions fp and fv are undefined, so the objective is not fully specified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Breaking the Stage Barrier: A Novel Single-Stage Approach to Long Context Extension for Large Language Models." pith.science (2026). https://pith.science/paper/UG5YCOF7

@misc{pith2026241207171,
  author       = {Pith},
  title        = {Pith review of: Breaking the Stage Barrier: A Novel Single-Stage Approach to Long Context Extension for Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UG5YCOF7}},
  note         = {Machine review of arXiv:2412.07171}
}
read the original abstract

Recently, Large language models (LLMs) have revolutionized Natural Language Processing (NLP). Pretrained LLMs, due to limited training context size, struggle with handling long token sequences, limiting their performance on various downstream tasks. Current solutions toward long context modeling often employ multi-stage continual pertaining, which progressively increases the effective context length through several continual pretraining stages. However, those approaches require extensive manual tuning and human expertise. In this paper, we introduce a novel single-stage continual pretraining method, Head-Adaptive Rotary Position Encoding (HARPE), to equip LLMs with long context modeling capabilities while simplifying the training process. Our HARPE leverages different Rotary Position Encoding (RoPE) base frequency values across different attention heads and directly trains LLMs on the target context length. Extensive experiments on 4 language modeling benchmarks, including the latest RULER benchmark, demonstrate that HARPE excels in understanding and integrating long-context tasks with single-stage training, matching and even outperforming existing multi-stage methods. Our results highlight that HARPE successfully breaks the stage barrier for training LLMs with long context modeling capabilities.

Figures

Figures reproduced from arXiv: 2412.07171 by the authors.

Figure 1
Figure 1. Illustration of the multi-stage and our proposed single-stage (HARPE) continual pretraining pipeline. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Traditional Single-Key Needle-in-a-Haystack: the x-axis represents the number of tokens in the test sample, ranging up to 128k tokens with finer granularity. The y-axis shows the depth of the needle’s position within the current test sample. Method 4k 8k 16k 32k 64k 128k Avg. Llama2-7B-Base 90.90 - - - - - - PI 77.56 26.59 16.50 0.00 0.00 0.00 20.11 ABF Single-Stage 92.44 88.78 84.16 78.03 70.81 62.72 79.49(3rd) ABF… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

73 extracted references · 18 canonical work pages

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  2. [2]

    AI21. 2024. Introducing jamba: Ai21’s groundbreaking ssm-transformer model. https://www.ai21.com/blog/announcing-jamba. Accessed: 2024-08-31

    work page 2024

  3. [3]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609

    arXiv 2023

  4. [4]

    Yushi Bai, Xin Lv, Jiajie Zhang, Yuze He, Ji Qi, Lei Hou, Jie Tang, Yuxiao Dong, and Juanzi Li. 2024. Longalign: A recipe for long context alignment of large language models. arXiv preprint arXiv:2401.18058

    arXiv 2024

  5. [5]

    Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150

    arXiv 2020

  6. [6]

    Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2019. https://arxiv.org/abs/arXiv:1911.11641 Piqa: Reasoning about physical commonsense in natural language

    arXiv 2019

  7. [7]

    by parts

    bloc97. 2023. https://github.com/jquesnelle/yarn/pull/1 Add ntk-aware interpolation "by parts" correction . Accessed: 2023

    work page 2023

  8. [8]

    Tom B Brown. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165

    arXiv 2020

Show all 73 references
  1. [9]

    Aydar Bulatov, Yury Kuratov, and Mikhail Burtsev. 2022. Recurrent memory transformer. Advances in Neural Information Processing Systems, 35:11079--11091

    2022

  2. [10]

    Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. 2024. Internlm2 technical report. arXiv preprint arXiv:2403.17297

    2024 arXiv

  3. [11]

    ChatGLM. 2024. https://medium.com/@ChatGLM/glm-long-scaling-pre-trained-model-contexts-to-millions-caa3c48dea85 Glm: Long scaling pre-trained model contexts to millions . Accessed: 2024-09-14

    2024

  4. [12]

    Longze Chen, Ziqiang Liu, Wanwei He, Yunshui Li, Run Luo, and Min Yang. 2024 a . Long context is not long at all: A prospector of long-dependency data for large language models. arXiv preprint arXiv:2405.17915

    2024 arXiv

  5. [13]

    Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. 2023. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595

    2023 arXiv

  6. [14]

    Yuhan Chen, Ang Lv, Ting-En Lin, Changyu Chen, Yuchuan Wu, Fei Huang, Yongbin Li, and Rui Yan. 2024 b . Fortify the shortest stave in attention: Enhancing context awareness of large language models for effective tool use. In Proceedings of the 62nd Annual Meeting of the Associ...

    2024

  7. [15]

    Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. 2024 c . Longlora: Efficient fine-tuning of long-context large language models. In Proceedings of the International Conference on Learning Representations (ICLR)

    2024

  8. [16]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. https://arxiv.org/abs/arXiv:1803.05457 Think you have solved question answering? try arc, the ai2 reasoning challenge

    2018 arXiv

  9. [17]

    Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. 2019. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860

    2019 arXiv

  10. [18]

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher R \'e . 2022. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:16344--16359

    2022

  11. [19]

    Jacob Devlin. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

    2018 arXiv

  12. [20]

    Ming Ding, Chang Zhou, Hongxia Yang, and Jie Tang. 2020. Cogltx: Applying bert to long texts. Advances in Neural Information Processing Systems, 33:12792--12804

    2020

  13. [21]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  14. [22]

    Jiayi Fu, Lei Lin, Xiaoyang Gao, Pengli Liu, Zhengzong Chen, Zhirui Yang, Shengnan Zhang, Xue Zheng, Yan Li, Yuliang Liu, et al. 2023. Kwaiyiimath: Technical report. arXiv preprint arXiv:2310.07488

    2023 arXiv

  15. [24]

    Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannaneh Hajishirzi, Yoon Kim, and Hao Peng. 2024 b . https://arxiv.org/abs/arXiv:2402.10171 Data engineering for scaling language models to 128k context

    2024 arXiv

  16. [25]

    Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, et al. 2024. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793

    2024 arXiv

  17. [26]

    Junqing He, Kunhao Pan, Xiaoqun Dong, Zhuoyang Song, LiuYiBo LiuYiBo, Qianguosun Qianguosun, Yuxin Liang, Hao Wang, Enming Zhang, and Jiaxing Zhang. 2024. Never lost in the middle: Mastering long-context question answering with position-agnostic decompositional training. In Pr...

    2024

  18. [27]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. https://arxiv.org/abs/arXiv:2009.03300 Measuring massive multitask language understanding

    2020 arXiv

  19. [28]

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. 2024. Ruler: What's the real context size of your long-context language models? arXiv preprint arXiv:2404.06654

    2024 arXiv

  20. [29]

    Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, Xinrong Zhang, Zheng Leng Thai, Kaihuo Zhang, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, Dahai L...

    2024 arXiv

  21. [30]

    Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. 2021. Efficient attentions for long document summarization. arXiv preprint arXiv:2104.02112

    2021 arXiv

  22. [31]

    Gautier Izacard and Edouard Grave. 2020. Leveraging passage retrieval with generative models for open domain question answering. arXiv preprint arXiv:2007.01282

    2020 arXiv

  23. [32]

    Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zirui Liu, Chia-Yuan Chang, Huiyuan Chen, and Xia Hu. 2024. Llm maybe longlm: Self-extend llm context window without tuning. arXiv preprint arXiv:2401.01325

    2024 arXiv

  24. [33]

    Weld, and Luke Zettlemoyer

    Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. https://arxiv.org/abs/arXiv:1705.03551 Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension

    2017 arXiv

  25. [34]

    Marzena Karpinska, Katherine Thai, Kyle Lo, Tanya Goyal, and Mohit Iyyer. 2024. One thousand and one pairs: A" novel" challenge for long-context language models. arXiv preprint arXiv:2406.16264

    2024 arXiv

  26. [35]

    Shanda Li, Chong You, Guru Guruganesh, Joshua Ainslie, Santiago Ontanon, Manzil Zaheer, Sumit Sanghai, Yiming Yang, Sanjiv Kumar, and Srinadh Bhojanapalli. 2023. Functional interpolation for relative positions improves long context transformers. arXiv preprint arXiv:2310.04418

    2023 arXiv

  27. [36]

    Haoran Lian, Yizhe Xiong, Zijia Lin, Jianwei Niu, Shasha Mo, Hui Chen, Peng Liu, and Guiguang Ding. 2024 a . Lbpe: Long-token-first tokenization to improve large language models. arXiv preprint arXiv:2411.05504

    2024 arXiv

  28. [37]

    Haoran Lian, Yizhe Xiong, Jianwei Niu, Shasha Mo, Zhenpeng Su, Zijia Lin, Peng Liu, Hui Chen, and Guiguang Ding. 2024 b . Scaffold-bpe: Enhancing byte pair encoding with simple and effective scaffold token removal. arXiv preprint arXiv:2404.17808

    2024 arXiv

  29. [38]

    Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. 2024 a . Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434

    2024 arXiv

  30. [39]

    Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. 2024 b . https://github.com/LargeWorldModel/LWM Lwm . https://github.com/zhangir-azerbayev/proof-pile. Accessed: 2024-09-13

    2024

  31. [40]

    Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. 2024 c . World model on million-length video and language with blockwise ringattention. arXiv preprint arXiv:2402.08268

    2024 arXiv

  32. [41]

    Xiaoran Liu, Hang Yan, Shuo Zhang, Chenxin An, Xipeng Qiu, and Dahua Lin. 2023. Scaling laws of rope-based extrapolation. arXiv preprint arXiv:2310.05209

    2023 arXiv

  33. [42]

    Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. arXiv preprint arXiv:1908.08345

    2019 arXiv

  34. [43]

    Xuezhe Ma, Xiang Kong, Sinong Wang, Chunting Zhou, Jonathan May, Hao Ma, and Luke Zettlemoyer. 2021. Luna: Linear unified nested attention. Advances in Neural Information Processing Systems, 34:2441--2453

    2021

  35. [44]

    Andr \'e Martins, Ant \'o nio Farinhas, Marcos Treviso, Vlad Niculae, Pedro Aguiar, and Mario Figueiredo. 2020. Sparse and continuous attention mechanisms. Advances in Neural Information Processing Systems, 33:20989--21001

    2020

  36. [45]

    Xin Men, Mingyu Xu, Bingning Wang, Qingyu Zhang, Hongyu Lin, Xianpei Han, and Weipeng Chen. 2024. Base of rope bounds context length. arXiv preprint arXiv:2405.14591

    2024 arXiv

  37. [46]

    Mistral.AI. 2023. La plateforme. https://mistral.ai/news/la-plateforme/. Accessed: 2024-08-31

    2023

  38. [47]

    Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2023. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.00071

    2023 arXiv

  39. [48]

    Ofir Press, Noah A Smith, and Mike Lewis. 2021. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108.12409

    2021 arXiv

  40. [49]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9

    2019

  41. [50]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1--67

    2020

  42. [51]

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024 a . Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063

    2024

  43. [52]

    Zhenpeng Su, Zijia Lin, Xue Bai, Xing Wu, Yizhe Xiong, Haoran Lian, Guangyuan Ma, Hui Chen, Guiguang Ding, Wei Zhou, et al. 2024 b . Maskmoe: Boosting token-level learning via routing mask in mixture-of-experts. arXiv preprint arXiv:2407.09816

    2024 arXiv

  44. [53]

    Together.AI. 2023. Preparing for the era of 32k context: Early learnings and explorations. https://www.together.ai/blog/llama-2-7b-32k. Accessed: 2024-08-31

    2023

  45. [54]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023 a . Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

    2023 arXiv

  46. [55]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023 b . Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  47. [56]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. https://arxiv.org/abs/arXiv:1706.03762 Attention is all you need

    2017 arXiv

  48. [57]

    Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, and Furu Wei. 2024. Augmenting language models with long-term memory. Advances in Neural Information Processing Systems, 36

    2024

  49. [58]

    Kaiwen Wei, Xian Sun, Zequn Zhang, Li Jin, Jingyuan Zhang, Jianwei Lv, and Zhi Guo. 2023 a . https://doi.org/10.1109/TKDE.2022.3218830 Implicit event argument extraction with argument-argument relational knowledge . IEEE Trans. Knowl. Data Eng. , 35(9):8865--8879

    2023

  50. [59]

    Kaiwen Wei, Xian Sun, Zequn Zhang, Jingyuan Zhang, Zhi Guo, and Li Jin. 2021. https://doi.org/10.18653/V1/2021.ACL-LONG.360 Trigger is not sufficient: Exploiting frame-aware knowledge for implicit event argument extraction . In Proceedings of the 59th Annual Meeting of the Ass...

    2021 doi

  51. [60]

    Kaiwen Wei, Yiran Yang, Li Jin, Xian Sun, Zequn Zhang, Jingyuan Zhang, Xiao Li, Linhao Zhang, Jintao Liu, and Zhi Guo. 2023 b . https://doi.org/10.18653/V1/2023.ACL-LONG.272 Guide the many-to-one assignment: Open information extraction via iou-aware optimal transport . In Proc...

    2023 doi

  52. [61]

    Chuhan Wu, Fangzhao Wu, Tao Qi, Yongfeng Huang, and Xing Xie. 2021. Fastformer: Additive attention can be all you need. arXiv preprint arXiv:2108.09084

    2021 arXiv

  53. [62]

    Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, et al. 2023. Effective long-context scaling of foundation models. arXiv preprint arXiv:2309.16039

    2023 arXiv

  54. [63]

    Yizhe Xiong, Xiansheng Chen, Xin Ye, Hui Chen, Zijia Lin, Haoran Lian, Jianwei Niu, and Guiguang Ding. 2024. Temporal scaling law for large language models. arXiv preprint arXiv:2404.17785

    2024

  55. [64]

    Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. 2023. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305

    2023 arXiv

  56. [65]

    Yaofu. 2023 a . Llama-2 7b 80k model. https://huggingface.co/yaofu/llama-2-7b-80k/tree/main. Accessed: 2024-08-31

    2023

  57. [66]

    Yaofu. 2023 b . Slimpajama-per-source-length-upsample. https://huggingface.co/datasets/yaofu/slimpajama-per-source-length-upsample. Accessed: 2024-09-03

    2023

  58. [67]

    Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 33:17283--17297

    2020

  59. [68]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830

    2019 arXiv

  60. [69]

    Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Hao, Xu Han, Zhen Thai, Shuo Wang, Zhiyuan Liu, et al. 2024 a . bench: Extending long context evaluation beyond 100k tokens. In Proceedings of the 62nd Annual Meeting of the Association for Computational Lin...

    2024

  61. [70]

    Yiyuan Zhang, Handong Li, Jing Liu, and Xiangyu Yue. 2024 b . Explore the limits of omni-modal pretraining at scale. arXiv preprint arXiv:2406.09412

    2024 arXiv

  62. [71]

    Zhenyu Zhang, Runjin Chen, Shiwei Liu, Zhewei Yao, Olatunji Ruwase, Beidi Chen, Xiaoxia Wu, and Zhangyang Wang. 2024 c . Found in the middle: How language models use long contexts better via plug-and-play positional encoding. arXiv preprint arXiv:2403.04797

    2024 arXiv

  63. [72]

    Bartosz Piotrowski Zhangir Azerbayev, Edward Ayers. 2022. proof-pile. https://github.com/zhangir-azerbayev/proof-pile. Accessed: 2022

    2022

  64. [73]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  65. [74]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.