The role of positional encodings in the ARC benchmark

Arlindo L. Oliveira; Guilherme H. Bandeira Costa; Miguel Freire

REVIEW 5 major objections 5 minor 13 references

The role of positional encodings in the ARC benchmark

T0 review · 5 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that the positional encoding inside a transformer—not just its size—decides how well it reasons on ARC-style grids, and that 2D sinusoidal encoding is the data-efficient default.

desk verdict A plausible data-regime finding about 2D vs RoPE positional encodings on grid tasks, but the ARC conclusion overreaches given the non-few-shot training protocol, missing error bars, and unspecified CodeT5+ modification; worth a workshop referee, not a citable benchmark claim. read the letter →

arxiv 2502.00174 v1 pith:LKNAGSDR submitted 2025-01-31 cs.AI cs.LG

Guilherme H. Bandeira Costa , Miguel Freire , Arlindo L. Oliveira This is my paper · ORCID

classification cs.AIcs.LG

keywords positionalencodingARCbenchmarktransformer2DsinusoidalrotarypositionembeddingabstractreasoninggridCodeT5+

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the positional encoding inside a transformer—not just the model's size—is decisive for abstract grid reasoning, and that a 2D sinusoidal encoding is the best default when training data is scarce. The authors first show that CodeT5+, which uses a relative positional encoding, performs much worse on vertical examples of a line-connecting ARC task than on identical horizontal examples, because the tokens that need to relate are farther apart in the flattened sequence. They then train custom transformers with 1D, 2D, Rotary, and learned positional encodings on generated examples of ten ARC tasks. Across small, medium, large, and decoder-only models, 2D encoding consistently beats 1D; with 10,000 training examples it also beats RoPE and learned embeddings, while RoPE takes a slight lead with 100,000 examples. The practical consequence is that choosing the right position signal can matter more than adding parameters for ARC-like problems.

What carries the argument

The object that does the work is the 2D sinusoidal positional encoding, which assigns each token a fixed vector built from sine and cosine functions of its row and column coordinates, instead of its flat position in the token list. In the paper's comparisons it is the one component changed while architecture, data, tokenization, and hyperparameters are held constant; the paper attributes the consistent accuracy gap between 1D and 2D to whether grid structure is explicitly present in the position signal. RoPE and learned embeddings serve as the modern baselines that 2D must beat in the data-efficiency comparison.

What would settle it

Train the same small, medium, and large transformers with 1D, 2D, and RoPE encodings on the original ARC protocol, using only the two to four demonstration pairs per task, and score against the official test grids under exact-match evaluation; if 2D encoding does not lead the field in that setting, the paper's central claim about ARC is wrong. A complementary check is to rotate every held-out task by 90 degrees and see whether 2D-encoded models preserve accuracy; if an orientation gap remains, the encoding has not actually eliminated the spatial bias the paper identifies in CodeT5+.

Watch

Extended reading notes

Core claim

The central claim is that a transformer's positional encoding controls whether it can solve ARC-style grid transformations, and that the 2D extension of the original sinusoidal encoding is the strongest choice in data-limited settings. The authors report that 2D positional encoding outperforms the standard 1D sinusoidal encoding across model sizes and in a decoder-only configuration, that it outperforms RoPE and learned embeddings when only 10,000 training examples are available, and that RoPE shows a slight edge once 100,000 examples are supplied. They also argue that CodeT5+'s relative positional encoding creates a horizontal-versus-vertical performance gap on a line-connecting task, and that modifying the encoding to bring relevant tokens closer together closes that gap. The claim, put simply, is that spatial reasoning in transformers is gated by how position is encoded.

Load-bearing premise

The load-bearing premise is that training a transformer on tens of thousands of generated examples of one ARC task tells us how the model would perform on ARC itself; but ARC's defining challenge is inferring a rule from only a handful of examples, so these experiments never test the few-shot setting the benchmark is built around.

Editorial extensions

If this is right

Defaulting to 2D sinusoidal positional encoding should improve transformer accuracy on ARC-like grid tasks without any change to the architecture or training procedure.
The benefit holds across small, medium, and large transformers and in decoder-only models, so smaller models can recover much of the performance gap with a better position signal.
When a reliable dataset generator is available and 100,000 examples can be produced, RoPE becomes competitive and slightly better, so the choice of encoding should depend on data availability.
Pretrained models that rely on relative positional encoding, such as CodeT5+, are likely to underperform systematically on spatial tasks where the relevant tokens are far apart in the flattened sequence.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper's experiments train on 10,000–100,000 generated examples per task; a direct test it does not run is to apply the same encodings to ARC's actual few-shot protocol, where the 2D advantage may be larger, smaller, or absent.
The CodeT5+ orientation result suggests a cheap diagnostic for any pretrained transformer on spatial inputs: rotate the input by 90 degrees and compare accuracy, since large drops would signal a positional-encoding bias rather than task difficulty.
If the data-constrained advantage transfers, 2D sinusoidal encoding should also help other grid-structured reasoning problems that are normally flattened into sequences, such as maze solving, matrix manipulation, or board-game state prediction.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take on arXiv:2502.00174. The paper compares positional encodings for grid reasoning on ARC-derived tasks and finds that 2D sinusoidal encoding beats 1D, learned embeddings, and RoPE under 10k training examples, while RoPE takes a slight edge at 100k. That data-regime flip is the genuinely new bit, and it is a plausible, practically useful observation. The CodeT5+ case study is also a nice illustration that relative positional encoding creates a horizontal/vertical bias in grid tasks because of how tokens are linearized.

The experiments are honest in design: the authors vary model size, use a decoder-only variant, and compare two tokenization formats, reporting aggregate accuracies across ten tasks. That is more than a toy study. But the central ARC claim does not hold up. Training on 100k (or even 10k) generated examples per task is not the few-shot paradigm ARC is built around, and the paper itself admits in the conclusion that RoPE might be better with a robust generator, which undercuts the abstract's unconditional preference for 2D in data-constrained scenarios. There are also no error bars or multiple seeds, only ten tasks, and the altered CodeT5+ positional encoding in Section 3 is never specified. Without code or data, I cannot tell whether the task generator is representative or whether the directional disparity is just a sequence-distance artifact.

The reader's REJECT verdict is about right for the current version, though this is a lean reject rather than a bad paper. I would rather see this sent to a workshop or a small venue where a referee can push for a proper few-shot evaluation, per-task variance, and full reproducibility. It deserves a serious referee, but it should not be cited as establishing a benchmark-level conclusion yet.

Referee Report

5 major / 5 minor

Summary. This paper studies how positional encodings (PE) affect transformer performance on ARC-style grid reasoning. It first presents a CodeT5+ case study in which models trained on horizontal vs. 90-degree-rotated examples of one ARC task perform differently, and a modified PE is said to improve vertical performance. It then trains custom transformer models of several sizes on 100,000 synthetic examples per ARC task, comparing 1D sinusoidal, 2D sinusoidal, RoPE, and learned embeddings. The paper reports that 2D PE is best in a 10,000-example data-constrained setting, while RoPE is slightly better with 100,000 examples, and concludes that 2D PE is particularly effective for ARC.

Significance. If the claims were supported, the paper would provide a practical recommendation for PE choice in grid-reasoning transformers. However, the experimental protocol does not match ARC's few-shot setting, no variance estimates or per-task results are reported, and the key CodeT5+ modification is unspecified. The paper is clearly written and the direction is interesting, but the current evidence does not substantiate the benchmark-level claims.

major comments (5)

[§4.1, §5] The experimental protocol trains each model on 100,000 (or 10,000) generated examples per single ARC task, whereas an ARC task provides only a handful of demonstration examples. This does not measure the few-shot generalization that defines the benchmark; moreover, the generated examples come from Hodel's generator, whose distributional similarity to real ARC tasks is not demonstrated. The paper's own limitation statement in §5 says ARC examples are treated independently. The abstract's claim that 2D encoding is effective 'for ARC tasks' is therefore not supported by this protocol, which is more naturally a study of PE on synthetic grid tasks with abundant data.
[§4.2–§4.3, Figs. 6–7] All accuracy comparisons are reported as aggregated averages with no error bars, no per-task breakdown, and a single seed (Table 2). The central conclusion that 2D PE 'consistently achieves the best performance' in the 10,000-example regime rests on one run per configuration, so the observed differences could be within noise. Multiple seeds and per-task results are needed before the ranking of PE methods can be trusted.
[§3] The altered CodeT5+ PE that 'prioritizes relationships specifically between the goal tokens' is never specified: no equations, no description of how relative position biases were changed, and no training hyperparameters for the CodeT5+ runs. Figure 5 consequently cannot be reproduced or interpreted. In addition, the horizontal/vertical disparity is explained by the linear token distance under relative PE, so the conclusion that the bottleneck is PE rather than tokenization is not established.
[§4.3, §5] The conclusion states that 2D PE 'may not necessarily be the optimal choice if a robust dataset generator is available,' and Figure 7a shows RoPE ahead at 100,000 examples. The abstract's unconditional statement that 2D encoding 'excels in data-constrained scenarios, emphasizing its effectiveness for ARC tasks' overstates the conditional evidence.
[§A.3, §4.1] The text says experiments use 10 distinct ARC tasks, but Appendix A.3 lists nine task IDs (22eb0ac0, 36d67576, 3aa6fb7a, 08ed6ac7, e8593010, e21d9049, 39e1d7f9, 913fb3ed, 68b16354). The discrepancy should be fixed.

minor comments (5)

[Figs. 3, 5] The figures report 'average correct matches across 3 distinct batch sizes' but do not define the metric or show the variance across batch sizes; please clarify.
[General] No code or data availability statement is provided, which limits reproducibility of the experiments.
[Table 2] The entry 'V ocabulary Size' has a stray space, and fixing the seed at 42 means all results are single-seed.
[References] The Shaw et al. reference contains a line break in 'V olume'; please clean the bibliography formatting.
[§4.1, §A.1] The tokenization details are only in the appendix; consider summarizing bracketed vs. raw input in the main text, since Section 4.1 depends on this distinction.

Circularity Check

0 steps flagged · score 0.0 of 10

No material circularity: the study is an empirical comparison with no fitted-input-as-prediction or load-bearing self-citation chain; its main limitation is external validity, not circularity.

full rationale

This paper is an empirical study, not a derivation, so the circularity patterns do not apply. The central claim that 2D sinusoidal positional encoding outperforms alternatives under data constraints is established by training transformer models on generated ARC-like tasks with different fixed positional encoding schemes and comparing measured accuracies (Sections 4.1–4.3). No parameter is fitted to the target claim: 1D, 2D, RoPE, and Learned Embeddings are external, fixed encoding definitions (Vaswani et al. 2017; Wang & Liu 2021; Su et al. 2024), and the accuracies are reported as observed outcomes rather than as quantities constructed from the encodings. The Section 3 CodeT5+ demonstration likewise compares default versus modified positional encoding on the same task, and while the modification is described only qualitatively, that is an under-specification rather than a circular reduction. The limitation acknowledged in Section 5—that ARC examples are treated as independent and the study omits few-shot relational reasoning—undermines the benchmark-level generalization of the conclusions, but it is a validity threat, not a circularity. There are no load-bearing self-citations: all cited prior results (e.g., 2D encoding from Wang & Liu 2021, RoPE from Su et al. 2024, Abacus Embeddings from McLeish et al. 2024) are external and do not presuppose the paper's conclusions. Consequently, no step reduces by definition to its inputs, and the appropriate circularity score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters were fitted to the target results; the hyperparameters and data-regime choices are standard or explicit conditions. The key unstated assumptions are about the validity of the training protocol and generator as stand-ins for ARC.

assumptions (3)

domain assumption The Hodel ARC generator produces examples that faithfully represent ARC tasks.
Used in Sections 3 and 4.1 to create training data; if the generator's distribution differs from real ARC, results may not transfer.
domain assumption The selected 9-10 ARC tasks are representative of the benchmark's diversity.
Only a small subset of 400 tasks is used; the paper gives no analysis of task difficulty or diversity.
domain assumption Per-task supervised training on 100,000 examples measures a model's ability to reason on ARC.
This protocol does not reproduce ARC's few-shot structure, so the link to ARC reasoning is an unstated assumption.

how reviews work

0 comments

Cite this review

Pith. "Pith review of The role of positional encodings in the ARC benchmark." pith.science (2026). https://pith.science/paper/LKNAGSDR

@misc{pith2026250200174,
  author       = {Pith},
  title        = {Pith review of: The role of positional encodings in the ARC benchmark},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LKNAGSDR}},
  note         = {Machine review of arXiv:2502.00174}
}

read the original abstract

The Abstraction and Reasoning Corpus challenges AI systems to perform abstract reasoning with minimal training data, a task intuitive for humans but demanding for machine learning models. Using CodeT5+ as a case study, we demonstrate how limitations in positional encoding hinder reasoning and impact performance. This work further examines the role of positional encoding across transformer architectures, highlighting its critical influence on models of varying sizes and configurations. Comparing several strategies, we find that while 2D positional encoding and Rotary Position Embedding offer competitive performance, 2D encoding excels in data-constrained scenarios, emphasizing its effectiveness for ARC tasks

Figures

Figures reproduced from arXiv: 2502.00174 by the authors.

**Figure 2.** The selected task is derived from the original ARC dataset where the objective is to connect [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Average correct matches across 3 distinct batch sizes when trained and tested for horizontal [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: Visual illustration of token creation for vertical (top) and horizontal (bottom) examples. [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]

**Figure 5.** Figure 5: Average correct matches across 3 distinct batch sizes when trained and tested for vertical [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]

**Figure 6.** Figure 6: Impact of positional encoding: (a) shows the performance of 1D and 2D positional en [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]

**Figure 7.** Figure 7: Impact of different positional encodings on a Medium model trained with (a) 100,000 [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]

**Figure 8.** Figure 8: An illustration of the tokenization process used in the experiments. The original grid is [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 3 canonical work pages

[2]

These configurations were chosen to ensure a fair comparison while maintaining computational efficiency. Table 1: Difference between transformer architectures Model Feedforward Layer Number of Layers Number of Heads Smaller Transformer 1024 1 4 Medium Transformer 1024 4 8 Larger Transformer 2024 6 8 8 Table 2: Consistent hyperparameters across all model a...

work page 2024
[4]

Michael Hodel

Accessed: 2025-01-17. Michael Hodel. Addressing the abstraction and reasoning corpus via procedural example generation. CoRR, abs/2404.07353,

arXiv 2025
[5]

URL https://doi.org/ 10.48550/arXiv.2404.07353

doi: 10.48550/ARXIV .2404.07353. URL https://doi.org/ 10.48550/arXiv.2404.07353. Wenhao Li, Yudong Xu, Scott Sanner, and Elias Boutros Khalil. Tackling the abstraction and reasoning corpus with vision transformers: the importance of 2d representation, positions, and objects. CoRR, abs/2410.06405,

arXiv doi:10.48550/arxiv.2404.07353
[6]

URL https: //doi.org/10.48550/arXiv.2410.06405

doi: 10.48550/ARXIV .2410.06405. URL https: //doi.org/10.48550/arXiv.2410.06405. Sean McLeish, Arpit Bansal, Alex Stein, Neel Jain, John Kirchenbauer, Brian R. Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, Jonas Geiping, Avi Schwarzschild, and Tom Goldstein. Transform- ers can do arithmetic with the right embeddings. CoRR, abs/2405.17399,

arXiv doi:10.48550/arxiv.2410.06405
[7]

URLhttps://doi.org/10.48550/arXiv.2405.17399

doi: 10.48550/ ARXIV .2405.17399. URLhttps://doi.org/10.48550/arXiv.2405.17399. Melanie Mitchell, Alessandro B. Palmarini, and Arseny Moskvichev. Comparing humans, GPT-4, and GPT-4V on abstraction and reasoning tasks. CoRR, abs/2311.09247,

arXiv doi:10.48550/arxiv.2405.17399
[10]

URL https://doi.org/10.1016/j.neucom

doi: 10.1016/J.NEUCOM.2023.127063. URL https://doi.org/10.1016/j.neucom. 2023.127063. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V . N. Vishwanathan, and Roman Ga...

arXiv 2023
[12]

URL https://doi.org/10.18653/v1/2023.emnlp-main.68

doi: 10.18653/V1/2023.EMNLP-MAIN.68. URL https://doi.org/10.18653/v1/2023.emnlp-main.68. 7 Zelun Wang and Jyh-Charn Liu. Translating math formula images to latex sequences using deep neural networks with sequence-level training. Int. J. Document Anal. Recognit. , 24(1): 63–75,

doi:10.18653/v1/2023.emnlp-main.68 2023
[2017]

Yue Wang, Hung Le, Akhilesh Gotmare, Nghi D

URL https://proceedings.neurips.cc/paper/2017/hash/ 3f5ee243547dee91fbd053c1c4a845aa-Abstract.html. Yue Wang, Hung Le, Akhilesh Gotmare, Nghi D. Q. Bui, Junnan Li, and Steven C. H. Hoi. Codet5+: Open code large language models for code understanding and generation. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on ...

work page 2017

Show all 13 references

[2018]

URL https://doi.org/10.18653/v1/n18-2074

doi: 10.18653/V1/N18-2074. URL https://doi.org/10.18653/v1/n18-2074. Jianlin Su, Murtadha H. M. Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063,

doi
[2019]

Franc ¸ois Chollet, Mike Knoop, Gregory Kamradt, and Bryan Landers

URL http: //arxiv.org/abs/1911.01547. Franc ¸ois Chollet, Mike Knoop, Gregory Kamradt, and Bryan Landers. ARC prize 2024: Technical report. CoRR, abs/2412.04604,

1911 arXiv
[2021]

URL https://doi.org/10.1007/ s10032-020-00360-2

doi: 10.1007/S10032-020-00360-2. URL https://doi.org/10.1007/ s10032-020-00360-2 . A A PPENDIX A.1 T OKENIZATION For these experiments, we used both raw and bracketed tokenization formats. Figure 8 illustrates how this mechanism works. Figure 8: An illustration of the tokeniza...

doi
[2023]

URLhttps://doi.org/10.48550/arXiv.2311.09247

doi: 10.48550/ ARXIV .2311.09247. URLhttps://doi.org/10.48550/arXiv.2311.09247. Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position represen- tations. In Marilyn A. Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference ...

doi arXiv
[2024]

URL https:// doi.org/10.48550/arXiv.2412.04604

doi: 10.48550/ARXIV .2412.04604. URL https:// doi.org/10.48550/arXiv.2412.04604. Franc ¸ois Chollet. Openai’s o3 model: A breakthrough on the arc challenge. https:// arcprize.org/blog/oai-o3-pub-breakthrough ,

doi arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.

[1] [2]

These configurations were chosen to ensure a fair comparison while maintaining computational efficiency. Table 1: Difference between transformer architectures Model Feedforward Layer Number of Layers Number of Heads Smaller Transformer 1024 1 4 Medium Transformer 1024 4 8 Larger Transformer 2024 6 8 8 Table 2: Consistent hyperparameters across all model a...

work page 2024

[2] [4]

Michael Hodel

Accessed: 2025-01-17. Michael Hodel. Addressing the abstraction and reasoning corpus via procedural example generation. CoRR, abs/2404.07353,

arXiv 2025

[3] [5]

URL https://doi.org/ 10.48550/arXiv.2404.07353

doi: 10.48550/ARXIV .2404.07353. URL https://doi.org/ 10.48550/arXiv.2404.07353. Wenhao Li, Yudong Xu, Scott Sanner, and Elias Boutros Khalil. Tackling the abstraction and reasoning corpus with vision transformers: the importance of 2d representation, positions, and objects. CoRR, abs/2410.06405,

arXiv doi:10.48550/arxiv.2404.07353

[4] [6]

URL https: //doi.org/10.48550/arXiv.2410.06405

doi: 10.48550/ARXIV .2410.06405. URL https: //doi.org/10.48550/arXiv.2410.06405. Sean McLeish, Arpit Bansal, Alex Stein, Neel Jain, John Kirchenbauer, Brian R. Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, Jonas Geiping, Avi Schwarzschild, and Tom Goldstein. Transform- ers can do arithmetic with the right embeddings. CoRR, abs/2405.17399,

arXiv doi:10.48550/arxiv.2410.06405

[5] [7]

URLhttps://doi.org/10.48550/arXiv.2405.17399

doi: 10.48550/ ARXIV .2405.17399. URLhttps://doi.org/10.48550/arXiv.2405.17399. Melanie Mitchell, Alessandro B. Palmarini, and Arseny Moskvichev. Comparing humans, GPT-4, and GPT-4V on abstraction and reasoning tasks. CoRR, abs/2311.09247,

arXiv doi:10.48550/arxiv.2405.17399

[6] [10]

URL https://doi.org/10.1016/j.neucom

doi: 10.1016/J.NEUCOM.2023.127063. URL https://doi.org/10.1016/j.neucom. 2023.127063. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V . N. Vishwanathan, and Roman Ga...

arXiv 2023

[7] [12]

URL https://doi.org/10.18653/v1/2023.emnlp-main.68

doi: 10.18653/V1/2023.EMNLP-MAIN.68. URL https://doi.org/10.18653/v1/2023.emnlp-main.68. 7 Zelun Wang and Jyh-Charn Liu. Translating math formula images to latex sequences using deep neural networks with sequence-level training. Int. J. Document Anal. Recognit. , 24(1): 63–75,

doi:10.18653/v1/2023.emnlp-main.68 2023

[8] [2017]

Yue Wang, Hung Le, Akhilesh Gotmare, Nghi D

URL https://proceedings.neurips.cc/paper/2017/hash/ 3f5ee243547dee91fbd053c1c4a845aa-Abstract.html. Yue Wang, Hung Le, Akhilesh Gotmare, Nghi D. Q. Bui, Junnan Li, and Steven C. H. Hoi. Codet5+: Open code large language models for code understanding and generation. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on ...

work page 2017