pith. sign in

arxiv: 2606.23687 · v1 · pith:UVD2M6V3new · submitted 2026-06-22 · 💻 cs.CL

Randomized YaRN Improves Length Generalization for Long-Context Reasoning

Manas Mehta , Fangcong Yin , Greg Durrett This is my paper

Pith reviewed 2026-06-26 08:12 UTC · model grok-4.3

classification 💻 cs.CL
keywords length generalizationpositional encodingYaRNlong-context reasoningLLMscontext extrapolationBABILongMRCR
0
0 comments X

The pith

Randomized YaRN training on short contexts improves reasoning on sequences up to 128K tokens.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper shows that large language models trained only on sequences shorter than 8K tokens can be made to reason better on much longer inputs by randomly sampling YaRN positional encodings from an extended range during that short-context training. This randomized exposure, paired with a length curriculum, is evaluated on two reasoning benchmarks and produces gains that grow with context length, beating ordinary fine-tuning especially at the longest out-of-distribution sizes. A sympathetic reader would care because the method offers a way to stretch effective context length without retraining on the full target lengths or adding new attention mechanisms.

Core claim

When tokens in short-context training data are assigned YaRN positional encodings drawn randomly from a larger position range, the resulting model generalizes better to actual long contexts than standard fine-tuning does, with the biggest advantages appearing at lengths far beyond the training distribution on BABILong and MRCR.

What carries the argument

Randomized YaRN: the practice of sampling YaRN positional encodings from an extended range while training on short sequences, combined with a length curriculum.

If this is right

  • Models trained on under 8K context can reach usable reasoning performance at 16K to 128K lengths.
  • The performance gap versus standard fine-tuning widens as context length increases.
  • The approach requires no separate fixes for attention or memory at test time.
  • Progressive exposure to out-of-distribution positional values during training supports length generalization.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same random-sampling idea might transfer to other positional schemes if the core mechanism is distribution exposure rather than the specific YaRN formula.
  • Combining Randomized YaRN with continued pretraining on modestly longer data could produce further gains at extreme lengths.
  • The method may reduce the need for synthetic long-context data generation in some training pipelines.

Load-bearing premise

Randomly sampling positional encodings from a wider range on short data will produce genuine generalization to real long contexts without harming other model capabilities.

What would settle it

A controlled comparison in which a model trained with Randomized YaRN shows no improvement or degrades relative to standard fine-tuning on 128K-token versions of BABILong or MRCR would falsify the central claim.

Figures

Figures reproduced from arXiv: 2606.23687 by Fangcong Yin, Greg Durrett, Manas Mehta.

Figure 1
Figure 1. Figure 1: Randomized YaRN trains LLMs for length generalization by: (1) Random position sampling: ran￾domly sampling positions from a longer length distribu￾tion, assigning the corresponding YaRN encodings to each token in the short-context training data to expose models to OOD positional encodings; and (2) Length￾generalization curriculum: gradually increasing the sam￾pling length distribution as a training curricu… view at source ↗
Figure 2
Figure 2. Figure 2: Learning-rate sweep results on BABILong for Olmo3-7B-Instruct. OOD average is reported in the figure. C Method Implementation YaRN As discussed in Section 2.2, the scale fac￾tor is an important hyperparameter for YaRN dur￾ing both training and inference. Let s denote the scale factor used during training and s ′ the factor used during inference. When s ′ = s, inference extrapolation matches training-time e… view at source ↗
read the original abstract

Large language models (LLMs) are typically pretrained on short sequences and then extended to work on longer sequences with additional training. However, such LLMs still struggle to further generalize to very long sequences. We propose Randomized YaRN, a training method that improves length generalization by combining YaRN-based positional extrapolation with randomized positional encoding and a length curriculum. During training on short context data, tokens are assigned YaRN positional encodings sampled from a larger position range, exposing the model to out-of-distribution positional representations even on short-context inputs. We evaluate Randomized YaRN on two challenging long-context reasoning benchmarks, BABILong and Multi-Round Coreference Resolution (MRCR). When training on data with <8K context, Randomized YaRN consistently improves reasoning performance on context lengths from 16K to 128K and outperforms standard fine-tuning, with the largest gains appearing at far out-of-distribution lengths. Our results suggest that progressively exposing models to OOD positional distributions provides an effective recipe for generalizable long-context reasoning.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript proposes Randomized YaRN, which augments YaRN positional extrapolation by randomly sampling scaling factors from a larger position range during training on short-context data (<8K tokens) together with a length curriculum. The central empirical claim is that this yields consistent gains on long-context reasoning benchmarks (BABILong and MRCR) at 16K–128K tokens relative to standard fine-tuning, with the largest improvements at far out-of-distribution lengths.

Significance. If the randomization step is shown to be the active ingredient, the approach would supply a lightweight recipe for length generalization that avoids full-scale long-context pretraining. The choice of reasoning-focused benchmarks (BABILong, MRCR) is appropriate and strengthens the evaluation.

major comments (2)
  1. [Experiments / Results] The headline result and title attribute OOD gains specifically to randomized sampling of YaRN encodings, yet no ablation is presented that keeps YaRN scaling and the length curriculum fixed while disabling randomization. Without this isolation, the reported improvements at 16K–128K cannot be attributed to randomization rather than curriculum exposure to longer sequences.
  2. [Results] Benchmark results are reported without error bars, multiple random seeds, or statistical tests, so the consistency and reliability of the claimed gains across context lengths cannot be assessed from the provided data.
minor comments (2)
  1. [Method] The precise sampling distribution (e.g., uniform over which range of scaling factors) used for randomization is not stated explicitly, hindering exact reproduction.
  2. [Tables] Tables comparing Randomized YaRN against baselines would benefit from clearer column headers indicating training context length and exact YaRN parameters.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback. We address each major comment below and will revise the manuscript to incorporate the suggested improvements.

read point-by-point responses

  1. Referee: [Experiments / Results] The headline result and title attribute OOD gains specifically to randomized sampling of YaRN encodings, yet no ablation is presented that keeps YaRN scaling and the length curriculum fixed while disabling randomization. Without this isolation, the reported improvements at 16K–128K cannot be attributed to randomization rather than curriculum exposure to longer sequences.

    Authors: We agree that an ablation isolating the randomization component (while retaining YaRN and the length curriculum) is necessary to strengthen attribution of the gains. The existing comparisons are to standard fine-tuning, which lacks both YaRN and the curriculum. We will add this ablation experiment in the revised manuscript, comparing fixed-scaling YaRN + curriculum against the randomized version, and update the results and discussion accordingly. revision: yes

  2. Referee: [Results] Benchmark results are reported without error bars, multiple random seeds, or statistical tests, so the consistency and reliability of the claimed gains across context lengths cannot be assessed from the provided data.

    Authors: We acknowledge that reporting variability is important for assessing reliability. In the revised version we will rerun the primary experiments across multiple random seeds, report means with standard deviations or error bars on BABILong and MRCR, and include statistical tests where appropriate. revision: yes

Circularity Check

0 steps flagged

No circularity: purely empirical method and benchmark evaluation

full rationale

The paper proposes Randomized YaRN as a training recipe (YaRN extrapolation + random sampling of scaling factors + length curriculum) and reports performance on external benchmarks BABILong and MRCR. No equations, first-principles derivations, fitted parameters renamed as predictions, or uniqueness theorems appear in the manuscript. All claims rest on measured accuracy differences rather than any reduction of outputs to inputs by construction. Self-citations, if present, are not load-bearing for any derivation chain.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

No explicit free parameters, axioms, or invented entities are stated in the abstract; the approach reuses the existing YaRN method with added randomization.

pith-pipeline@v0.9.1-grok · 5703 in / 1056 out tokens · 24467 ms · 2026-06-26T08:12:54.844977+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

13 extracted references · 9 canonical work pages · 6 internal anchors

  1. [1]

    InThe 2023 Conference on Empirical Methods in Natural Language Processing

    GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. InThe 2023 Conference on Empirical Methods in Natural Language Processing. Cem Anil, Yuhuai Wu, Anders Johan Andreassen, Aitor Lewkowycz, Vedant Misra, Vinay Venkatesh Ra- masesh, Ambrose Slone, Guy Gur-Ari, Ethan Dyer, and Behnam Neyshabur

    2023

  2. [2]

    Extending Context Window of Large Language Models via Positional Interpolation

    Exploring Length Generalization in Large Language Models. InAd- vances in Neural Information Processing Systems. Yushi Bai, Xin Lv, Jiajie Zhang, Yuze He, Ji Qi, Lei Hou, Jie Tang, Yuxiao Dong, and Juanzi Li. 2024a. LongAlign: A Recipe for Long Context Alignment of Large Language Models. InFindings of the Asso- ciation for Computational Linguistics: EMNLP...

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  3. [3]

    The Llama 3 Herd of Models

    The Llama 3 Herd of Models.Preprint, arXiv:2407.21783. Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    work page internal anchor Pith review Pith/arXiv arXiv

  4. [4]

    arXiv preprint arXiv:2406.10149

    BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack. arXiv preprint arXiv:2406.10149. 5 Yi Lu, Jing Nathan Yan, Songlin Yang, Justin T Chiu, Siyu Ren, Fei Yuan, Wenting Zhao, Zhiyong Wu, and Alexander M Rush

    work page arXiv

  5. [5]

    Olmo 3

    Olmo 3.Preprint, arXiv:2512.13961. Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole

    work page internal anchor Pith review Pith/arXiv arXiv

  6. [6]

    InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Vol- ume 2: Short Papers), pages 1889–1903

    Randomized Positional Encodings Boost Length Generalization of Trans- formers. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Vol- ume 2: Short Papers), pages 1889–1903. Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu

    1903

  7. [7]

    RoFormer: Enhanced Transformer with Rotary Position Embedding

    RoFormer: En- hanced Transformer with Rotary Position Embedding. Preprint, arXiv:2104.09864. Qwen Team, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, and 24 oth- ers

    work page internal anchor Pith review Pith/arXiv arXiv

  8. [8]

    Qwen2.5 Technical Report

    Qwen2.5 Technical Report.Preprint, arXiv:2412.15115. Kiran V odrahalli, Santiago Ontanon, Nilesh Tripuraneni, Kelvin Xu, Sanil Jain, Rakesh Shivanna, Jeffrey Hui, Nishanth Dikkala, Mehran Kazemi, Bahare Fatemi, and 1 others

    work page internal anchor Pith review Pith/arXiv arXiv

  9. [9]

    Jason Weston, Antoine Bordes, Sumit Chopra, Alexan- der M

    Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queries.arXiv preprint arXiv:2409.12640. Jason Weston, Antoine Bordes, Sumit Chopra, Alexan- der M. Rush, Bart van Merriënboer, Armand Joulin, and Tomas Mikolov

    work page arXiv

  10. [10]

    Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks

    Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks. Preprint, arXiv:1502.05698. Tong Wu, Yanpeng Zhao, and Zilong Zheng

    work page internal anchor Pith review Pith/arXiv arXiv

  11. [11]

    Xinyu Zhao, Fangcong Yin, and Greg Durrett

    LongSky- work: A Training Recipe for Efficiently Extending Context Length in Large Language Models.Preprint, arXiv:2406.00605. Xinyu Zhao, Fangcong Yin, and Greg Durrett

    work page arXiv

  12. [12]

    InICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models

    Trans- formers Can Achieve Length Generalization But Not Robustly. InICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models. Dawei Zhu, Nan Yang, Liang Wang, Yifan Song, Wen- hao Wu, Furu Wei, and Sujian Li

    2024

  13. [13]

    where was the football before the garden?

    and consists of multiple reasoning subtasks. We focus on a challenging subtask (QA3) that requires multi-hop reasoning with 3 support- ing facts and has not shown saturated performance even among frontier LLMs. In this subtask, each example is constructed by embedding three bAbI facts into long passages of irrelevant text drawn from PG19, a corpus of Engl...

    2023