RePo: Language Models with Context Re-Positioning

Deng Cai; Huayang Li; Richard Sproat; Tianyu Zhao

REVIEW 3 major objections 5 minor 1 cited by

RePo: Language Models with Context Re-Positioning

T0 review · 3 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read RePo replaces fixed linear token order with learned continuous positions and achieves consistent gains on noisy, structured, and long-context tasks.

desk verdict RePo is a clean, lightweight learned-position module with consistent empirical gains, but the missing same-capacity control leaves the re-positioning claim underdetermined. read the letter →

arxiv 2512.14391 v3 pith:RNQCDEQA submitted 2025-12-16 cs.LG cs.AIcs.CL

Huayang Li , Tianyu Zhao , Deng Cai , Richard Sproat This is my paper

classification cs.LGcs.AIcs.CL

keywords contextre-positioninglearnedpositionalencodingRoPEin-contextlearninglong-contextgeneralizationneedle-in-a-haystackattentionallocationcognitiveloadtheory

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the fixed linear order imposed on tokens is itself a burden: it claims that letting a language model learn where each token belongs—a real-valued position generated by a small module f_phi for each attention head—improves tasks that demand restructuring of context, such as noisy long contexts, tables, and extended sequences. It proposes RePo, which plugs the learned positions into a differentiable rotary position encoder so the whole model trains end-to-end. On 1B- and 7B-scale models, continued training with RePo beats the fixed-position baseline by 5.4 points on RULER needle-in-a-haystack, 2.27 exact-match points on HybridQA, and 6.93 average points on LongBench at 1B scale, with parallel gains at 7B, while general short-context benchmarks stay essentially flat. A sympathetic reader would care because it suggests a lightweight architectural change can reallocate model attention toward distant relevant tokens without sacrificing general ability.

What carries the argument

The central object is the re-positioning module f_phi, a lightweight two-part network: a SwiGLU sub-layer that projects a token's hidden state into a low-dimensional position representation, and a per-head linear map that outputs a single real value z_i. This replaces the integer index i in relative attention, with the rotary position encoding rotating queries and keys by g_theta(z_j - z_i). Because the rotary encoding is differentiable in the position difference, f_phi can be trained end-to-end, and because positions are continuous, the model can express constant (NoPE-like), monotonic (RoPE-like), or hybrid positional patterns per attention head. RePo is applied from roughly the one-third

What would settle it

Train a model identical to the RePo model except that the learned module is frozen or regularized to output standard linear positions z_i = i for every token, with the same parameter count, training data, and continued-training recipe. If this control matches RePo's gains on RULER needle-in-a-haystack, HybridQA, and LongBench, the re-positioning explanation is not supported; if it does not, the re-positioning mechanism is the likely cause. A complementary check is to measure attention mass on needle tokens in that control: if it stays at baseline levels, the attention-redistribution story depe

Watch

Extended reading notes

Core claim

The central claim is that token positions in an LLM should be learned, not prescribed. RePo replaces each token's integer index with a real value z_i produced by a small differentiable module f_phi that reads the token's hidden state. Attention scores become q_i^T g_theta(z_j - z_i) k_j, so positions enter through the relative difference in a rotary position encoding and can be optimized with ordinary backpropagation. After continued training of 1B- and 7B-scale open models on 50B tokens, RePo outperforms the fixed-position rotary baseline on noisy-context tasks (RULER needle-in-a-haystack: +5.4 average at 1B), on structured data (HybridQA: +2.27 exact match at 1B, +4.09 at 7B), and on long-

Load-bearing premise

The causal claim rests on the assumption that the observed gains come from re-positioning itself rather than from the extra trainable parameters and per-head flexibility the method adds; the paper includes no control that adds the same capacity while keeping the original linear order.

Editorial extensions

If this is right

On needle-in-a-haystack tasks within the 4K training length, RePo raises average accuracy by 5.4 points at 1B scale, so the benefit is not only about length extrapolation.
Used with YaRN context extension, RePo-trained models hold or widen their margin over fixed-position baselines at 8K and 16K tokens, lengths unseen during training.
On HybridQA, RePo improves exact match by 2.27 points at 1B and 4.09 points at 7B, suggesting that linearized tables retain more usable structure when positions are learned.
On LongBench, RePo improves the average score by 6.93 points at 1B and 6.38 at 7B across multidoc QA, summarization, and few-shot tasks.
General short-context benchmarks remain close to the fixed-position baseline, so the learned positions do not come at the cost of common-case performance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the mechanism, not just extra parameters, drives the gains, learned re-positioning could also benefit retrieval-augmented generation and agentic tools, where context is assembled from heterogeneous sources; the paper does not test this directly.
The synthetic text-reversal experiment shows that RePo can learn mirror-like position assignments; a natural next test is code or math benchmarks, where order is semantically load-bearing, to see whether learned positions preserve or disrupt structure.
A control that adds the same number of parameters while forcing the original linear positions would separate re-positioning from raw capacity; the current experiments do not include it, so part of the improvement could come from the added module rather than from re-arranging positions.
Because RePo works through any differentiable position encoder, combining it with bias-based or other continuous encodings is a testable extension that would show whether the benefit is tied to rotary geometry or generalizes across encoding families.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Hi—

Quick read on RePo: it's a genuinely different position-encoding idea—predict unconstrained continuous per-head positions with a small SwiGLU module on hidden states, then feed the differences into RoPE. That's distinct from COPE, and the RoPE/FlashAttention compatibility is a real practical point. The empirical work is honest and the gains are consistent across two scales: +5.4 on RULER NIAH (1B), +2.27 EM on HybridQA, +6.93 LongBench, with similar direction at 7B. They keep general-task performance flat, so the story is coherent.

What they do well: continued pretraining on OLMo-2 with identical recipe, transparent setup, code and weights promised, and the analyses of attention mass and position patterns are appropriate as descriptive evidence, not overclaimed. The synthetic reversal experiment is a nice sanity check that the module can learn non-trivial re-positioning.

The main soft spot is the one the reader flagged, and I think it's load-bearing: there is no control that isolates the added capacity of f_phi. The module is only 0.9% extra parameters, but a per-head affine map z_i = alpha_h * i + beta_h sits inside the same function class, and the paper never tests a fixed-linear or fixed-random variant with the same parameter count. Without that, the abstract's claim that re-positioning per se drives the gains is underdetermined. Long-context numbers also depend on YaRN applied to all RoPE layers, which is fine but should be remembered when interpreting the 4K-to-16K story. Single-run numbers with no error bars is a minor but real weakness, especially in tables where some differences are ~1 point.

Also, the starting-layer ablation (Fig. 4) doesn't hold the module fixed in capacity, so it doesn't answer the control question either. And the CLT framing is decorative—not harmful, but it doesn't do work beyond motivation.

Overall: solid, honest, useful paper with a novel mechanism and a clear empirical case that something helps. The missing control means the central mechanism claim is not yet proven, but the paper deserves a serious referee and the authors should be pushed to run the affine control. If that control matches RePo, the contribution shrinks to per-head scale/offset; if it doesn't, this is a nice result.

I'd take this to reading group and would cite it. Send it to review without hesitation.

Referee Report

3 major / 5 minor

Summary. RePo replaces the fixed integer/constant position assignments of causal LLMs with continuous positions z_i produced by a lightweight per-head module f_phi from hidden states (Eqs. 4–7). The module is trained during continual pretraining of OLMo-2 1B and 7B and is applied from the 1/3 layer onward; RoPE encodes the relative difference z_j - z_i. The empirical core is Tables 1–3: RePo outperforms RoPE by 5.4 NIAH average (1B), 2.27 EM on HybridQA (1B), and 6.93 LongBench average (1B), with analogous 7B gains; general-task accuracy is roughly unchanged. Analyses examine attention mass on distant needles, the range/pattern of assigned positions, and a synthetic text-reversal proof of concept.

Significance. The paper reports a substantial empirical study: continual pretraining of two open OLMo-2 scales (1B/7B) on 50B tokens, evaluation across noisy-context, structured-data, long-context, and general benchmarks, with consistent positive deltas on the target dimensions at both scales. If the causal attribution is correct, the contribution is a lightweight (0.9% parameter) module compatible with RoPE and other differentiable encodings, with open code/weights expected. The attention-mass and position-pattern analyses are useful descriptive evidence, and the synthetic reversal task supports the mechanism in a simplified setting. However, the missing equivalent-capacity control means the paper currently establishes that a trainable position-prediction module helps under this training protocol, not that non-linear content-dependent re-positioning per se is the cause. The long-context conclusion additionally depends on YaRN, whose interaction with RePo's positions is not analyzed. These are fixable but essential.

major comments (3)

[§3.2, Eq. (5); Tables 1–3] The causal claim that content-dependent re-positioning drives the gains is not isolated from the added capacity of f_phi. The comparisons use RoPE/NoPE/R2N1/N2R1, none of which adds a trainable position module. RePo adds ~0.9% parameters (Sec. 5.4) and a per-head linear assignment W_z in Eq. (5); f_phi can in principle learn to track the token index and produce a per-head affine rescaling, i.e., learned per-head RoPE frequency/offset. Fig. 4 varies only the starting layer and does not control for this. Please add an equivalent-capacity control whose position output is constrained to a learnable per-head affine function of the original index (or of a frozen linear position). If that control matches RePo, the contribution reduces to per-head scale/offset; if not, it strengthens the causal claim.
[§4.1, Fig. 1, Table 3] Long-context conclusions are confounded by YaRN. In Sec. 4.1 and Fig. 1/Table 3, YaRN's factor is chosen relative to the 4K training context and is designed for RoPE's integer index differences. RePo instead feeds z_j - z_i, whose range is much smaller than the token index (Fig. 2). Applying the same YaRN scaling to RePo and to RoPE is therefore not the same operation; the reported 8K/16K and LongBench gains could partly stem from an interaction between RePo's compressed position ranges and YaRN. Please evaluate RePo with YaRN parameters matched to its own position scale, and/or without YaRN, to confirm the long-context claim.
[Tables 1–3, 5] All reported numbers are single-run, with no variance estimates. Some differences are small (e.g., +0.6 NIAH average at 7B, -0.62 general-task average at 7B), so it is difficult to calibrate the 'consistently enhances' claim. Please provide variance over at least evaluation seeds for a representative subset (NIAH, HybridQA, LongBench), or state which results are known to be stable across runs.

minor comments (5)

[Throughout] The method name is rendered inconsistently as RePo and REPO across the title, abstract, and body; please standardize.
[Abstract, §7] Abstract: 'replying on pre-defined order' should be 'relying'; Conclusion: 'continues space' should be 'continuous space'.
[§4.1] 'We train those models on 4 H100 GPUs for 50B tokens' is surprising for the 7B model; please clarify whether this is per model or a typo.
[Appendix C] The synthetic reversal experiment shares f_phi across heads and uses a 4-layer GPT-2, so it is not directly comparable to the main per-head setup; consider labeling it as a proof of concept only.
[Table 5, 7B row] The 7B general-task average for RePo is -0.62 relative to RoPE; 'comparable' is defensible, but the text should acknowledge this small negative delta explicitly.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: RePo is an empirical architecture study evaluated on external benchmarks; the missing capacity-matched control is a confound, not a circular reduction.

full rationale

The paper does not claim to derive RePo's gains from first principles. The module f_phi (Eqs. 4-7) is trained with ordinary language-model loss on OLMo-2 stage-2 data, and all reported gains are measured on external benchmarks (RULER, HybridQA, LongBench, MMLU-Pro, etc.); no benchmark is used to fit or select the module's parameters, so the empirical claims are not fitted-input predictions. The reformulation of NoPE as RoPE with constant position (Eq. 8) is a mathematical identity, not a circular derivation. The main caveat is that RePo adds a trainable 0.9%-parameter module absent from the baselines, so the improvement could in principle come from extra capacity rather than from re-positioning per se. However, this is a causal-attribution limitation or missing control, not a case where the prediction is equivalent to its input by construction. Self-citations (e.g., Li et al. 2025, SeqPE) are used only as examples of existing position-encoding families and are not load-bearing for the central claim. Attention-mass and position-pattern analyses are descriptive post-hoc measurements. Verdict: no significant circularity.

Assumptions & free parameters 2 free parameters · 3 assumptions · 1 invented entities

The paper does not derive a law or constant; the central claim is empirical. The main free choices are architectural hyperparameters (starting layer, representation dimension), while the module parameters are trained. The key assumption is that the hidden states carry enough signal for the module, and that the observed gains are due to re-positioning rather than the added parameters.

free parameters (2)

Starting layer for RePo application = l=5 for 1B; l=10 for 7B (1/3 of layers)
Chosen by hand to balance efficiency and effectiveness; ablation (Fig. 4) shows robustness but a trade-off between NIAH and MMLUPro.
Position representation dimension dp = d/8 (d = model hidden size)
Chosen by hand based on the assumption that positional information is lower-dimensional than hidden states; no systematic sweep is reported.

assumptions (3)

domain assumption Hidden states h_i contain enough information for a lightweight SwiGLU+linear module to extract useful re-positioning signals.
Required for Eqs. 4-6 to produce effective positions; not theoretically guaranteed beyond the empirical results.
standard math RoPE's rotation g_theta remains a valid, differentiable function when applied to arbitrary real-valued position differences z_j - z_i.
RoPE is defined over continuous angles; the paper assumes this extension is valid and trainable.
domain assumption Continuing pre-training on 50B tokens after stage-1 does not require re-learning general knowledge; lower layers are kept at standard RoPE because they handle surface/local features.
Motivates the 1/3-layer design choice (Tenney et al., 2019); partially tested by ablation in Fig. 4.

invented entities (1)

Context re-positioning module f_phi (with SwiGLU position representation r_i and per-head linear position assignment z_i) independent evidence
purpose: Assigns continuous, per-head token positions used inside RoPE attention (Eq. 7), replacing fixed linear positions.
The module is a trainable function whose outputs are observable; the released code/weights and the reported attention/position analyses provide a falsifiable handle.

how reviews work

0 comments

Cite this review

Pith. "Pith review of RePo: Language Models with Context Re-Positioning." pith.science (2026). https://pith.science/paper/RNQCDEQA

@misc{pith2026251214391,
  author       = {Pith},
  title        = {Pith review of: RePo: Language Models with Context Re-Positioning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RNQCDEQA}},
  note         = {Machine review of arXiv:2512.14391}
}

abstract

In-context learning is fundamental to modern Large Language Models (LLMs); however, prevailing architectures impose a rigid and fixed contextual structure by assigning linear or constant positional indices. The rigid position information poses the full burden of organizing the input structure to attention layers, thus reducing the amount of attention that could be allocated for more critical information. To address this, we propose RePo, a novel mechanism that alleviates the burden for attention layers via context re-positioning. Unlike conventional approaches, RePo utilizes a differentiable module, $f_\phi$, to assign token positions that capture contextual dependencies, rather than replying on pre-defined order. By continually pre-training on the OLMo-2 1B \& 7B models, we demonstrate that RePo consistently enhances performance on tasks involving noisy contexts, structured data, and longer context length, while maintaining competitive performance on general short-context tasks. Analysis reveals that RePo successfully allocates more attention mass to distant but relevant information, assigns positions in a dense and non-linear space, and captures the intrinsic structure of the input context. Our code is at https://github.com/SakanaAI/repo.

Figures

Figures reproduced from arXiv: 2512.14391 by the authors.

**Figure 1.** Long-context evaluation on OLMo-2 1B model. We use the data in RULER (Hsieh et al.) benchmark and apply YaRN (Peng et al., 2024) for all RoPE layers to extend the context. Similar results on OLMo-7B are shown in Appendix B.4. • Noisy Context evaluates the model’s robustness to contexts containing large amounts of irrelevant information. Such “noise” increases extraneous cognitive load (Paas et al., 2003), which can… view at source ↗

**Figure 2.** Statistics for the distances between maximum and minimum positions in each attention head of the LLM. The averaged and maximum number of tokens in the MMLUPro-Math benchmark are 1971 and 2512, while those for RULER-QA are 2995 and 3555, respectively. length may not be necessary from the model’s perspective. Furthermore, the distribution of distances is non-uniform, unlike the linear positional assignment in RoPE. … view at source ↗

**Figure 5.** Long-context evaluation on OLMo-2 7B model. We use the data in RULER (Hsieh et al.) benchmark and apply YaRN (Peng et al., 2024) for all RoPE layers to extend the context. B.4. Additional Results on Long Context [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: Sensitivity to the starting layer of REPO (i.e. l = 3, 5, 7). We validate on the NIAH subtask of RULER benchmark and MMLUPro of general benchmarks. Results for the context extrapolation on OLMo-2 7B model are shown in [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]

**Figure 6.** Figure 6: Visualization of predicted positions from a 4-layer GPT-2 model in the reversal task. The area with blue background color indicates input context, while the orange region is the generated sequence. We use A-K to replace the real tokens to save space for illustration. T…

**Figure 7.** Figure 7: Performance on the text reversal task. We report the accuracy on all lengths of input sequences. As shown in [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]

**Figure 8.** Figure 8: Visualization of positions assigned by REPO. The REPO is continuously trained on general data. The visualization data is from MMLUPro with few-shot examples. Symbols in orange belong to the prompt, while symbols in blue and red represent questions and answers in few-sh…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Frontier Language Models Struggle to Copy: Text Can Be Better Viewed in 2D
cs.CL 2026-07 conditional novelty 6.0 of 10

2D-RoPE, which arranges text by line breaks into rows and columns, lets Transformers copy strings hundreds of times longer than training lengths, while standard 1D positional encodings fail on the same task.

Reference graph

Works this paper leans on

4 extracted references · 1 linked inside Pith · cited by 1 Pith paper

[1]

rope type

8K Tokens:{"rope type": "yarn", "factor": 2.0, "original max position embeddings": 4096}
[2]

rope type

16K Tokens:{"rope type": "yarn", "factor": 4.0, "original max position embeddings": 4096} We use the setting for “16K Tokens” for all the experiments on LongBench (Table 3). B.3. General Tasks We use the following task ids in olmes for the evaluation in Table 5: arc challenge:rc::large, arc easy:rc::olmes, boolq:rc::large,coqa::large,drop::large,hellaswag...

2024
[1252]

Gelberg, Y ., Eguchi, K., Akiba, T., and Cetin, E

PMLR, 2017. Gelberg, Y ., Eguchi, K., Akiba, T., and Cetin, E. Extending the context of pretrained llms by dropping their positional embeddings.arXiv preprint arXiv:2512.12167, 2025. Golovneva, O., Wang, T., Weston, J., and Sukhbaatar, S. Contextual position encoding: Learning to count what’s important.arXiv preprint arXiv:2405.18719, 2024. Hsieh, C.-P., ...

arXiv 2017
[2023]

Kazemnejad, A., Padhi, I., Natesan Ramamurthy, K., Das, P., and Reddy, S

URL https://github.com/gkamradt/LLMTest NeedleInAHaystack/tree/main. Kazemnejad, A., Padhi, I., Natesan Ramamurthy, K., Das, P., and Reddy, S. The impact of positional encoding on length generalization in transformers.Advances in Neural Information Processing Systems, 36:24892–24928, 2023. LeCun, Y . and Bengio, Y . Convolutional networks for im- ages, sp...

arXiv 2023

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.

[1] [1]

rope type

8K Tokens:{"rope type": "yarn", "factor": 2.0, "original max position embeddings": 4096}

[2] [2]

rope type

16K Tokens:{"rope type": "yarn", "factor": 4.0, "original max position embeddings": 4096} We use the setting for “16K Tokens” for all the experiments on LongBench (Table 3). B.3. General Tasks We use the following task ids in olmes for the evaluation in Table 5: arc challenge:rc::large, arc easy:rc::olmes, boolq:rc::large,coqa::large,drop::large,hellaswag...

2024

[3] [1252]

Gelberg, Y ., Eguchi, K., Akiba, T., and Cetin, E

PMLR, 2017. Gelberg, Y ., Eguchi, K., Akiba, T., and Cetin, E. Extending the context of pretrained llms by dropping their positional embeddings.arXiv preprint arXiv:2512.12167, 2025. Golovneva, O., Wang, T., Weston, J., and Sukhbaatar, S. Contextual position encoding: Learning to count what’s important.arXiv preprint arXiv:2405.18719, 2024. Hsieh, C.-P., ...

arXiv 2017

[4] [2023]

Kazemnejad, A., Padhi, I., Natesan Ramamurthy, K., Das, P., and Reddy, S

URL https://github.com/gkamradt/LLMTest NeedleInAHaystack/tree/main. Kazemnejad, A., Padhi, I., Natesan Ramamurthy, K., Das, P., and Reddy, S. The impact of positional encoding on length generalization in transformers.Advances in Neural Information Processing Systems, 36:24892–24928, 2023. LeCun, Y . and Bengio, Y . Convolutional networks for im- ages, sp...

arXiv 2023