Pith. sign in

REVIEW 5 major objections 4 minor 21 references

Theoretical Analysis of Positional Encodings in Transformer Models: Impact on Expressiveness and Generalization

T0 review · 5 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that wavelet-based positional encodings let transformers extrapolate to longer sequences better than sinusoidal encodings, with a unified theory of expressiveness, generalization, and extrapolation.

desk verdict Interesting wavelet and Legendre PE proposals, but the main theoretical bound for wavelet extrapolation is internally inconsistent and the toy experiments are too weak to carry the claim. read the letter →

arxiv 2506.06398 v1 pith:IEQ7CHLI submitted 2025-06-05 cs.LG cs.AI

Yin Li This is my paper
classification cs.LGcs.AI MSC 68T0742C4041A10
keywords positionalencodingstransformersextrapolationwavelettransformsRademachercomplexityALiBiLegendrepolynomialsexpressiveness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to put positional encodings in transformers on a common theoretical footing, treating expressiveness as function approximation, generalization as Rademacher complexity, and extrapolation as behavior past the training length. Its central claim is that encodings built from orthogonal transforms, especially wavelets, can beat the standard sinusoidal encoding for both generalization and extrapolation. If true, this matters because transformer users could choose position schemes by predicted length behavior instead of by tradition. The supporting experiments are deliberately small, but they align with the theory on a running-sum task.

What carries the argument

The central object is an encoding vector formed by evaluating an orthonormal wavelet basis, specifically Daubechies-4 scaling and wavelet functions at integer positions, at scales up to $\lfloor \log_2(N_{\max}) \rfloor$. The mechanism doing the work is multi-resolution locality: fine-scale wavelets have small supports and disappear for positions beyond $N_{\max}$, while coarse scales continue to carry position information, so the normalized encoding changes little past the training length. The argument also relies on Rademacher complexity, a standard capacity measure, to bound generalization, and on a Lipschitz-continuity model of the attention function to bound ALiBi's extrapolation error.

What would settle it

Compute the exact normalized Daubechies-4 wavelet encodings used in the paper for positions $N_{\max}+1$ through $N_{\max}+100$ and measure $\|PE_{\text{wavelet}}(pos) - PE_{\text{wavelet}}(N_{\max})\|_2$. If the difference is exactly zero, or normalization becomes undefined, once $pos$ leaves the support of every basis function, then the claimed exponential-decay extrapolation bound has no object to apply to; if a correctly normalized encoding still satisfies the bound, the premise is confirmed.

Watch

Extended reading notes

Core claim

The paper claims that positional encodings are not neutral: they determine whether a transformer can keep working when the input is longer than anything seen in training. It unifies sinusoidal, learned, relative, ALiBi, wavelet, and Legendre encodings under one framework, with expressiveness defined by universal approximation, generalization bounded by Rademacher complexity, and extrapolation measured by how the encoding vector changes for positions beyond $N_{\max}$. The specific new claim is that wavelet encodings preserve coarse positional information after fine-scale components vanish, so their vector remains close to the encoding at $N_{\max}$; on the synthetic running-sum task, the wavelet encoding achieves lower MSE than sinusoidal at two and four times the training length, and slightly lower than ALiBi at both. The paper also derives that ALiBi's linear bias gives graceful but linearly growing extrapolation error, and that tanh-saturated Legendre encodings collapse toward a constant vector for very long positions.

Load-bearing premise

The load-bearing assumption is the exponential-decay bound in Section 6.1.4 for wavelet values beyond the training length; the Daubechies-4 wavelets used in the experiments are exactly zero outside their support, so the bound cannot be true for them.

Editorial extensions

If this is right

  • If the theory holds, practitioners who need to test on sequences longer than training should prefer wavelet or ALiBi encodings over sinusoidal or learned absolute encodings.
  • Normalized wavelet encodings should not hurt interpolation accuracy, because the Rademacher bound shows the same input-norm capacity as sinusoidal encodings.
  • Legendre encodings with the tanh map are safe only for modest extrapolation; beyond roughly $N_{\max} + O(N_{\max}/d_{\text{model}}^{2.5})$ they collapse toward a constant vector.
  • ALiBi's extrapolation error grows linearly with the distance beyond $N_{\max}$, so the slope should be chosen with the expected test length in mind.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Not proven by the paper, but suggested: the wavelet advantage would need a proof not based on exponential decay, since Daubechies-4 wavelets have compact support; one route is to treat the exact zero tail as a truncation and show a continuation rule preserves coarse position.
  • A testable extension the paper does not run: combine wavelet position vectors with an ALiBi-style bias and compare on long-context language or vision tasks rather than the synthetic running-sum task.
  • If the multi-scale mechanism is what drives extrapolation, other bounded high-frequency feature maps, such as truncated Fourier or random features, should show similar length robustness, a prediction the paper leaves implicit.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The manuscript proposes a theoretical framework for analyzing positional encodings in transformers, with claims about expressiveness, Rademacher-complexity generalization bounds, and extrapolation to longer sequences. It introduces two new encoding schemes based on wavelets and Legendre polynomials, and reports a lightweight NumPy experiment on a synthetic running-sum task which the authors interpret as showing that wavelet encodings outperform sinusoidal and ALiBi encodings when extrapolating to sequences of length 100 and 200. The central theoretical claim is that orthogonal-transform encodings, especially wavelet-based ones, provide better extrapolation than sinusoidal encodings.

Significance. The topic is timely and important, and the paper is clearly organized. The idea of using wavelet and Legendre bases as positional encodings is interesting and the synthetic running-sum task is a sensible testbed for extrapolation. If the theoretical results were correct, the paper would offer useful guidance for choosing positional encodings in long-context transformers. However, the main theoretical justification for the wavelet extrapolation advantage is invalid: it relies on an exponential-decay assumption that contradicts the compact support of the Daubechies wavelets used in the experiments. In addition, the generalization bounds in Section 4 are mostly asserted rather than derived, and the ALiBi extrapolation bound in Section 5 is essentially a restatement of a Lipschitz assumption. The experimental evidence is a single run with no error bars, seeds, or code, which is too weak to support the headline empirical claim.

major comments (5)
  1. [Section 6.1.4, displayed bound |ψ_{j,k}(pos)| ≤ C 2^{-j/2} exp(-β(pos-Nmax)/2^{-j})] This bound is asserted for Daubechies-4 wavelets, which are compactly supported. For any compactly supported wavelet, ψ_{j,k}(pos) is exactly zero outside a finite interval, not a smooth exponential tail. The inequality is trivially true where the wavelet is zero, but it does not imply that ‖PE_wavelet(pos) − PE_wavelet(Nmax)‖₂ decays as O(exp(-β(pos−Nmax))); the actual difference is step-like, becoming constant after an abrupt drop and ultimately zero once all basis functions have exited their support. For the normalized encoding ĥPE_wavelet(pos) = PE_wavelet(pos)/‖PE_wavelet(pos)‖₂, positions beyond all supports yield a zero vector and the normalized encoding is undefined. Since this extrapolation bound is the only theoretical argument for the wavelet advantage claimed in Section 7, the paper's central theoretical claim is unsupported.
  2. [Section 4.2, Rademacher complexity bounds] The bounds R_m(F) = O(L_trans B/√m) are introduced through a 'standard covering-number argument', but no covering-number calculation, no chain rule for Rademacher complexity, and no dependence on depth, width, or number of parameters is given. The section does not state a theorem; it simply writes order-of-magnitude expressions. As a result, the comparison in Table 2 reduces to comparing input norm bounds, and the claimed contribution on how PEs affect generalization is not established beyond a generic Lipschitz argument.
  3. [Section 5.3, Eq. (9)] The bound |A(µ, −αd) − A(µ, −αNmax)| ≤ L_A α(d − Nmax) is an immediate consequence of assuming that A is Lipschitz in its second argument; it does not explain why the learned attention function of a transformer should have this Lipschitz property, nor does it connect this local statement about attention weights to an end-to-end guarantee on sequence-level extrapolation error. The 'unified extrapolation framework' is therefore a restatement of the assumption rather than a theoretical derivation, and it does not distinguish ALiBi from other Lipschitz biases.
  4. [Section 3.5 and Table 1] The paper claims that ALiBi yields universal approximation for arbitrary sequence length, marked 'Yes' in the 'Universal (Any N)' column of Table 1, but no construction or proof is provided. This claim is in tension with Section 3.4, which acknowledges that purely relative encodings may fail when absolute position information is required. Because ALiBi's bias depends only on |i−j|, it is not clear how it can represent absolute positions; without a proof, the claim is unsupported and appears inconsistent with the paper's own discussion of relative encodings.
  5. [Section 7, Table 5] The experimental support for the headline claim consists of a single run on a toy running-sum task with no error bars, no seeds, no code, and no ablations. The encoder is a minimal 2-layer transformer without layer normalization or dropout, and the implementation details of the wavelet basis selection (e.g., which 64 of the available coefficients are kept) are not fully specified. Consequently, the conclusions in Section 7.3 that wavelet encodings 'retain best performance' and that results 'align with theoretical extrapolation bounds' are not empirically justified.
minor comments (4)
  1. [Section 6.1.1] The text defines δ_pos as 'the Dirac delta at position pos' and then writes ⟨δ_pos, ϕ⟩ = ϕ(pos); for integer positions this should be phrased as a Kronecker delta or as an evaluation functional on a continuous domain, since the Dirac delta is a distribution.
  2. [Section 6.2.4] In the Taylor expansion, the approximation e^{-2γΔpos/Nmax} ≈ 1 − 2γΔpos/Nmax requires Δpos ≪ Nmax/(2γ), but this condition is not stated before the bound is used to claim an extrapolation range up to O(Nmax/d^{2.5}_{model}).
  3. [Section 5.4, first paragraph] The statement that 'wavelet basis functions extend beyond any finite interval (though they decay)' is imprecise for the Daubechies wavelets used later, which are compactly supported; this wording contributes to the confusion about the extrapolation analysis.
  4. [Section 2.4 and elsewhere] The paper uses 'ALiBi' as a positional encoding in the abstract and Section 2, but ALiBi is a bias applied to attention logits rather than an additive position embedding; the distinction should be made explicit earlier, especially because the generalization-bound comparison in Table 2 treats it as an input-norm effect.

Circularity Check

1 steps flagged · score 2.0 of 10

Minor self-definitional step in ALiBi extrapolation bound; wavelet bound is unsupported but not circular.

  1. self definitional [Section 5.3, Eq. (9)]
    "If A is Lipschitz with constant LA in its second argument, then for d > Nmax, |A(µ, −α d) − A(µ, −α Nmax)| ≤ LA α (d − Nmax). (9) Thus, if α is small enough and d does not exceed Nmax by a huge margin, the difference in attention weight is small. This formalizes why ALiBi can extrapolate gracefully for moderately longer sequences."

    The bound in Eq. (9) is exactly the Lipschitz continuity assumption restated: applying the definition of a Lipschitz function with constant LA to the second argument (the bias) yields |A(µ, −αd) − A(µ, −αNmax)| ≤ LA |−αd + αNmax| = LA α (d − Nmax) immediately. No transformer-specific or ALiBi-specific structure is used. The paper presents this as an 'Upper Bound on Extrapolation Error' and as the formal reason ALiBi extrapolates, so the theoretical support for ALiBi's extrapolation reduces to the assumption itself. The Limitations section later acknowledges the assumption, but the derivation is definitional rather than substantive.

full rationale

The paper's main contributions—expressiveness characterization, Rademacher complexity bounds, and the proposed wavelet/Legendre encodings—are not circular: they are conditional mathematical statements or standard applications of known results, and the experimental results are not statistically forced by any fitted parameter. The only self-citation is a blog post reference used to mention informal prior proposals, which is not load-bearing. The ALiBi extrapolation bound in Eq. (9) is a direct restatement of the Lipschitz assumption and is therefore a mild self-definitional step, but it is not used to fit any quantity and retains conditional validity. The wavelet extrapolation bound in Section 6.1.4 relies on an unproven and factually questionable exponential-decay property for compactly supported Daubechies-4 wavelets; this is a correctness risk and missing support, but it is not circular because the claimed bound does not feed back into any fitted parameter or self-citation chain. Overall, the derivation chain is largely self-contained, and the central claims do not reduce to their inputs by construction.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

No new physical or mathematical entities are introduced; the proposed wavelet and Legendre encodings are just new functions of position. The paper's central claims rest on the listed axioms, several of which are either unproven or false for the specific wavelet family used.

free parameters (3)
  • ALiBi slope alpha = 0.1/Ntrain
    Chosen by hand for the experiment in Section 7.1; not derived by the theory. In Section 5.3 the same alpha is a user-set hyperparameter that controls extrapolation behavior.
  • Legendre scaling gamma = 1
    Hyperparameter in Eq. (11), set to 1 in experiments without sensitivity analysis; it controls the saturation point and the extrapolation range O(gamma e^{-2gamma} ...).
  • Wavelet coefficients per scale = 10
    In Section 7.1, 10 coefficients per scale for j=0..5, then 'selecting top 64 basis functions by support coverage'; this selection is ad hoc and not justified by theory.
assumptions (5)
  • domain assumption The attention function A is Lipschitz continuous in its bias argument.
    Invoked in Section 5.3 to bound extrapolation error (Eq. 9); the paper's Limitations admit it 'may not hold exactly for ReLU-based networks or large biases'.
  • ad hoc to paper Wavelet basis functions decay exponentially outside their main support, with |ψ_{j,k}(pos)| <= C 2^{-j/2} exp(-β (pos - Nmax) / 2^{-j}).
    Used in Section 6.1.4 to claim exponential decay of encoding differences; not satisfied by compactly supported Daubechies wavelets, which are exactly zero outside their support, and this does not imply exponential decay of the full encoding vector.
  • standard math A standard covering-number argument for Lipschitz transformer classes bounds Rademacher complexity as O(L_trans B0 / sqrt(m)).
    Invoked in Section 4.2 but not derived; no explicit constants or theorem statement; depends on unspecified norm constraints.
  • domain assumption Transformers with sinusoidal PE are universal approximators for fixed-length sequences (Yun et al., 2019).
    Cited in Section 3.2 and used to conclude universality for each PE scheme; the paper does not prove this itself.
  • standard math Legendre polynomials satisfy |P_l(x)| <= 1 for x in [-1,1].
    Used in Section 6.2.3 to bound the encoding norm; true, but the training-range mapping only covers [0, tanh(γ)], where orthogonality is lost.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Theoretical Analysis of Positional Encodings in Transformer Models: Impact on Expressiveness and Generalization." pith.science (2026). https://pith.science/paper/IEQ7CHLI

@misc{pith2026250606398,
  author       = {Pith},
  title        = {Pith review of: Theoretical Analysis of Positional Encodings in Transformer Models: Impact on Expressiveness and Generalization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IEQ7CHLI}},
  note         = {Machine review of arXiv:2506.06398}
}
read the original abstract

Positional encodings are a core part of transformer-based models, enabling processing of sequential data without recurrence. This paper presents a theoretical framework to analyze how various positional encoding methods, including sinusoidal, learned, relative, and bias-based methods like Attention with Linear Biases (ALiBi), impact a transformer's expressiveness, generalization ability, and extrapolation to longer sequences. Expressiveness is defined via function approximation, generalization bounds are established using Rademacher complexity, and new encoding methods based on orthogonal functions, such as wavelets and Legendre polynomials, are proposed. The extrapolation capacity of existing and proposed encodings is analyzed, extending ALiBi's biasing approach to a unified theoretical context. Experimental evaluation on synthetic sequence-to-sequence tasks shows that orthogonal transform-based encodings outperform traditional sinusoidal encodings in generalization and extrapolation. This work addresses a critical gap in transformer theory, providing insights for design choices in natural language processing, computer vision, and other transformer applications.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

21 extracted references · 8 canonical work pages

  1. [1]

    Unconstrained representation of orthogonal matrices with application to common principle components

    Allen-Zhu, Z. and Li, Y. (2020). Towards understanding the role of over-parametrization in gen- eralization of neural networks. International Conference on Learning Representations (ICLR) . https://arxiv.org/abs/1906.00587

    work page Pith review arXiv 2020

  2. [2]

    and Mendelson, S

    Bartlett, P.L. and Mendelson, S. (2002). Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research , 3:463–482

    work page 2002

  3. [3]

    and Sellke, M

    Bubeck, S. and Sellke, M. (2021). A universal law of robustness via isoperimetry. Advances in Neural Information Processing Systems (NeurIPS) , 34:28811–28822. https://arxiv.org/abs/ 2105.12806

    arXiv 2021

  4. [4]

    Chen, M., Peng, H., Fu, J., and Ling, H. (2021). AutoFormer: Searching transformers for visual recognition. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 12270–12280. https://arxiv.org/abs/2107.00651

    work page Pith review arXiv 2021

  5. [5]

    Du, S., Lee, J., Li, H., Cubuk, E.D., and Zhai, X. (2021). How does self-attention learn positional information? arXiv preprint arXiv:2105.00641 . https://arxiv.org/abs/2105.00641

    work page Pith review arXiv 2021

  6. [7]

    Kazemnejad, A., Kuchaiev, O., and Salakhutdinov, R. (2021). A mathematical framework for trans- former circuits. Transformer Circuits. https://transformer-circuits.pub/2021/framework/ index.html

    work page 2021

  7. [8]

    Ke, G., He, D., and Liu, T.Y. (2020). Rethinking positional encoding in language pre-training. International Conference on Learning Representations (ICLR) . https://arxiv.org/abs/2006. 15595

    work page 2020

  8. [9]

    Li, X. (2021). Understanding positional encoding in transformers. Blog post. Accessed: 2025-01-01

    work page 2021

Show all 21 references
  1. [10]

    Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. (2020). ConvBERT: Improving BERT with span-based dynamic convolution.arXiv preprint arXiv:2008.02496. https: //arxiv.org/abs/2008.02496

    2020 arXiv

  2. [11]

    Neyshabur, B., Li, Z., Bhojanapalli, S., LeCun, Y., and Srebro, N. (2018). Towards under- standing the role of over-parametrization in generalization of neural networks.arXiv preprint arXiv:1805.12076. https://arxiv.org/abs/1805.12076 19

    2018 arXiv

  3. [12]

    Press, O., Smith, N.A., and Lewis, M. (2021). Train short, test long: Attention with linear biases en- ables input length extrapolation.International Conference on Learning Representations (ICLR) . https://arxiv.org/abs/2108.12409

    2021 arXiv

  4. [13]

    Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P.J. (2020). Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of Machine Learning Research (JMLR) , 21(140):1–67. https://arxiv.org/abs/1910.10683

    2020 arXiv

  5. [14]

    Shaw, P., Uszkoreit, J., and Vaswani, A. (2018). Self-attention with relative position represen- tations. Proceedings of the 2018 Conference of the North American Chapter of the Associa- tion for Computational Linguistics: Human Language Technologies (NAACL-HLT) , 2:464–468. h...

    2018 arXiv

  6. [15]

    Su, J., Lu, Y., Pan, S., Wen, B., and Liu, Y. (2021). RoFormer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2104.09864 . https://arxiv.org/abs/2104.09864

    2021 arXiv

  7. [16]

    Tancik, M., Srinivasan, P.P., Mildenhall, B., Fridovich-Keil, S., Raghavan, N., Singhal, U., Ra- mamoorthi, R., Barron, J.T., and Ng, R. (2020). Fourier features let networks learn high fre- quency functions in low dimensional domains.Advances in Neural Information Processing ...

    2020 arXiv

  8. [17]

    Tay, Y., Dehghani, M., Bahri, D., and Metzler, D. (2020). Efficient transformers: A survey.ACM Computing Surveys, 55(6):1–28. https://arxiv.org/abs/2009.06732

    2020 arXiv

  9. [18]

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polo- sukhin, I. (2017). Attention is all you need.Advances in Neural Information Processing Systems (NeurIPS), 30:5998–6008. https://arxiv.org/abs/1706.03762

    2017 arXiv

  10. [19]

    Xiong, R., Yang, Y., He, D., Zheng, K., Zheng, S., Xing, C., Zhang, H., Lan, Y., Wang, L., and Liu, T.Y. (2020). On layer normalization in the transformer architecture.International Conference on Machine Learning (ICML), 37:10524–10533. https://arxiv.org/abs/2002.04745

    2020 arXiv

  11. [20]

    Yao, Z., Cao, Z., Luo, W., Huang, C., Li, K., and Zhong, M. (2018). Efficient attention: Attention with linear complexities.arXiv preprint arXiv:1812.01243. https://arxiv.org/abs/1812.01243

    2018 arXiv

  12. [21]

    Yun, C., Bhojanapalli, S., Rawat, A.S., Reddi, S.J., and Kumar, S. (2019). Are transformers universal approximators of sequence-to-sequence functions? arXiv preprint arXiv:1912.10077 . https://arxiv.org/abs/1912.10077

    2019 arXiv

  13. [22]

    Yun, C., Bhojanapalli, S., Rawat, A.S., Reddi, S.J., and Kumar, S. (2019). Transformers without tears: Improving the normalization of self-attention. arXiv preprint arXiv:1910.05895 . https: //arxiv.org/abs/1910.05895 20

    2019 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.