{"id":"a3354418-3346-4dc5-ab52-a6d018c17aff","arxiv_id":"2412.11459","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A two-layer transformer with relative positional encoding keeps its induction head active across the whole sequence, while absolute positional encoding loses it in the second half.","lead":"This paper studies how a two-layer transformer with relative positional encoding can recognize repeated patterns anywhere in a long prompt, and how it weighs trained statistical knowledge against in-context patterns when predicting the next token. The authors derive the learned attention weights and output logits for a bigram language model with triggered transitions, and confirm the trends in controlled experiments.","discovery_kind":"first_principles","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 2's proof is internally inconsistent: Eq. 13 defines W1K as Σ χ(k) wE(k) r_{-1}^T, but the claimed nonzero score r_{-1}^T W1K wE(v) requires the transposed orientation Σ χ(k) r_{-1} wE(k)^T.","rationale":"The paper's central claim is that RPE makes the previous-token association independent of token position, unlike APE. For that claim to hold, the learned W1K must map any token embedding wE(v) to a vector aligned with the relative position vector r_{-1}, i.e., W1K must contain terms r_{-1} wE(k)^T. The formal statement in Eq. 13 instead writes W1K = Σ χ(k) wE(k) r_{-1}^T, which is the opposite outer-product order. Under the paper's own near-orthogonality assumption, this orientation gives r_{-1}^T W1K wE(v) ≈ 0, so the theorem's displayed score is not what the proof derives. The inconsistency is not merely cosmetic: it determines whether the first-layer attention actually attends to the previous token. The proof's own final formula and Proposition 4 use the transposed orientation, suggesting a typo rather than a fundamentally wrong idea, but the paper as written does not contain a correct proof of its headline theorem. I do not reject the work because the qualitative RPE advantage is empirically demonstrated in Section 5 and the orientation error appears fixable; however, the conditional verdict should stand until the orientation is corrected and the theorem re-derived. The sequential-training simplification identified by the reader is a real limitation, but the transposition inconsistency is more load-bearing because it affects internal validity under the paper's own assumptions.","tokens_in":44398,"tokens_out":10782,"duration_ms":90170,"concrete_test":"Implement the Appendix C.2 setup numerically: d=1024, V=64, Gaussian unit-norm wE(v) and r_{-i}; compute the one-step gradient for W1K exactly, preserving all matrix orders, and check whether the result is Σ χ(k) wE(k) r_{-1}^T or Σ χ(k) r_{-1} wE(k)^T. Then evaluate the first-layer attention score (W1K(wE(z_s)+r_{s-t}))^T wE(z_t) for s=t-1 versus s≠t-1 under both orientations. If the Eq. 13 orientation yields uniform attention, the theorem as written fails; if the transposed orientation is required, the paper must correct Eq. 13, Definition 2, and Theorem 3 accordingly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central theorem is internally inconsistent on matrix orientation. Formal Theorem 3 (Eq. 13) states W1K = Σ_{k∈Q} χ(k) wE(k) r_{-1}^T + (W1K)'. Under Assumption 1, r_{-1} is near-orthogonal to every wE(v). Hence W1K wE(v) = Σ_k χ(k) wE(k)(r_{-1}^T wE(v)) ≈ 0 for every v, so the claimed score r_{-1}^T W1K wE(v) vanishes for all v. The proof of Theorem 2/3, however, computes a nonzero value for this score, which is only possible if the learned matrix has the transposed form W1K = Σ_k χ(k) r_{-1} wE(k)^T. The same transposed orientation is used in Proposition 4's proof to make the first-layer attention peak at the previous token. With the orientation written in Eq. 13, attention scores (W1K (wE(z_s)+r_{s-t}))^T wE(z_t) are ≈0 for all s, so no previous-token head exists. Thus, as stated, the proof does not establish the paper's main theoretical claim—even granting the simplified sequential one-step training setup.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper analyzes a two-layer attention-only transformer with relative positional encoding (RPE), trained on bigram sequences with triggered transitions. The main theoretical claim is that, unlike absolute positional encoding (APE), where the previous-token association decays as 1/t, RPE learns a position-independent previous-token head; a second set of results describes how the final logits combine pretrained bigram log-probabilities with in-context pattern frequencies. The analysis is carried out under an explicit simplified protocol (one gradient step per matrix, sequential training from zero initialization, loss restricted to the final token), and the experiments measure memory recall and prompt-frequency-dependent predictions.","tokens_in":44677,"tokens_out":20962,"duration_ms":175627,"significance":"The paper addresses a real question—why RPE transformers retain in-context information at long sequence positions and how global and in-context knowledge are combined—and it offers explicit associative-memory formulas together with controlled experiments that match the qualitative predictions. The authors are also transparent about the simplified training setup and the two-layer scope. However, the main theorem as stated has a matrix-orientation inconsistency: the formal expression for W1K in Eq. (13) makes the claimed score vanish under the paper's own near-orthogonality assumption, and the proof only obtains a nonzero value for the transposed matrix. This is fixable by choosing one consistent convention, but until then the central claim is not established. With the orientation corrected, the paper would be a useful theoretical complement to Bietti et al. and to empirical studies of positional encoding and length generalization.","major_comments":[{"comment":"Eq. (13) states W1K = Σ_{k∈Q} χ(k) wE(k) r_{-1}^T. Under Assumption 1, r_{-1}^T wE(v) ≈ 0 for every v, so W1K wE(v) ≈ 0 and the score r_{-1}^T W1K wE(v) claimed in Theorem 2 is identically small. The proof of Theorem 3 computes a nonzero value only for the transposed orientation (Σ χ(k) r_{-1} wE(k)^T), and the same transposed orientation is used in Proposition 4 and in the memory-recall metric of §5.1. The same issue affects Eq. (14): with W2K = Σ ψ(k) wE(k)(Φ1wE(k))^T, the quantity (Φ1wE(zs))^T W2K wE(zT) used in Step 3 is ≈0, whereas the proof evaluates the transposed matrix. Because Theorem 2 is the paper's central claim, the statement and proof must be made consistent; for example, keep Eq. (13) and state the result as r_{-1}^T W1K^T wE(v) (equivalently wE(v)^T W1K r_{-1}), or replace Eq. (13) by its transpose throughout. This is not a cosmetic issue: if implemented as written, the §5.1 memory-recall metric would be zero for all k.","section":"Appendix C.3 (Eqs. 13–14) and Theorem 2"},{"comment":"The derivation of W1K and W2K assumes a very specific training protocol: W2O, W2K, and W1K are trained sequentially, each with one gradient descent step from zero initialization, while the loss is computed only on sequences of length T ending in the second occurrence of the trigger token and only on the final output token (Appendix C.2). The experiments in §5.1 instead train a full model end-to-end with SGD and momentum, with all attention matrices updated jointly over 1000 iterations. As a result, the experiments verify the end-state qualitative prediction but do not test the sequential one-step derivation that produces the closed form of W1K. The paper should either include an experiment that follows the Appendix C.2 protocol, or explicitly state that the training-dynamics theorem is an idealized derivation whose connection to end-to-end training is not proven.","section":"Appendix C.2 and §5.1"}],"minor_comments":[{"comment":"The first term of the displayed formula in Theorem 2 is written with 't · O(1)' inside a sum over t; please make explicit which error terms are uniform in t and V, since the claimed independence of t and v depends on those terms being uniformly small.","section":"Theorem 2"},{"comment":"The 'score' columns in Table 2 are not defined in the table or its caption; if they are the memory-recall values from §5.1, this should be stated explicitly.","section":"Table 2"},{"comment":"There are several typographical errors, including 'V on Oswald' (should be 'Von Oswald') in the references and 'encoder-ony' (should be 'encoder-only') in Appendix A.","section":"References and typos"},{"comment":"In the proof of Proposition 4, the softmax weights are written using Euler's number e in expressions such as e/(t+e-1); please define this convention at the point of use, since the same symbol is also used for token indices and it is easy to confuse the base of the softmax with an index.","section":"Proposition 4 proof"}],"recommendation":"major_revision","confidential_remarks":"The orientation flaw in Eqs. (13)–(14) is most likely a transposition convention that the authors can fix in revision; I do not see evidence of a deeper conceptual error in the intended construction. The larger gap between the sequential one-step training protocol and the end-to-end experiments should be addressed honestly, either by an additional experiment or by softening the training-dynamics claim. The paper is within the scope of the journal if the central theorem is made internally consistent."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a serious look, but with a specific fix required. The paper's real contribution is an RPE-specific analysis of the previous-token head in the associative-memory framework, plus a closed-form logit decomposition into pretrained bigram knowledge and in-context pattern counts. The experiments qualitatively confirm both stories.\n\nI agree with the stress-test: the central theorem as written is internally inconsistent. Theorem 3 (Eq. 13) states W1K = Σ χ(k) wE(k) r_{-1}^T. Under the paper's own near-orthogonality assumption, W1K wE(v) is approximately zero for every v, so the score r_{-1}^T W1K wE(v) that Theorem 2 asserts is nonzero cannot be nonzero from that expression. The proof only produces a nonzero score with the transposed orientation, W1K = Σ χ(k) r_{-1} wE(k)^T, and the proof of Proposition 4 actually substitutes that transposed form. This is very likely a typo rather than a wrong idea, but it is load-bearing: as stated, the proof does not establish the main theoretical claim. Easy to fix, but must be fixed.\n\nThe training protocol is also heavily simplified: sequential one-step gradient descent on W2O, W2K, W1K, with frozen random other matrices, and sequences with exactly two trigger tokens ending at T. The authors acknowledge this, and it matches how Bietti et al. proceed, but it means the theorem is about that protocol, not about end-to-end training. Propositions 2 and 3 are partly circular because Definition 2 hard-codes log π_b into the feed-forward weights; the trained-model experiments give some external grounding, but the distinction between constructed and emergent is not clearly drawn.\n\nNo code is shipped, and the main quantitative claims rest on a handful of plots; I'd ask for error bars and the exact prompt-generation code before relying on the empirical side.\n\nWho is it for: readers in mechanistic interpretability and length generalization who want a tractable model of how RPE avoids position-dependent decay. It deserves a serious referee, and I would send it to review with the expectation of a revision focused on the orientation fix and on separating construction from emergence.","headline":"Worth a serious look, but the central RPE theorem has a matrix-orientation error in its stated form; the proof uses the transposed matrix, so the main claim needs a fix before it is citable.","tokens_in":45205,"tokens_out":5208,"would_cite":false,"duration_ms":44379,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Relative positional encoding makes a two-layer transformer's induction head position-independent, and its logits come out as a weighted blend of pretrained bigram knowledge and in-context pattern counts.","keywords":["induction head","in-context learning","associative memory","relative positional encoding","length generalization","bigram model","two-layer transformer","gradient-descent dynamics"],"falsifier":"Train a two-layer transformer with relative positional encoding end-to-end, updating all weights jointly, on the bigram-with-trigger task, and measure the memory-recall score for the association $r_{-1}^\\top W_1^K w_E(v)$ separately for early and late positions; if recall for positions $t > 128$ falls below recall for early positions, the claimed position independence is false.","tokens_in":44105,"feed_emoji":"🧠","tokens_out":13030,"duration_ms":99408,"temperature":0.7,"pith_summary":"This paper sets out to explain two behaviours of the induction head, the attention circuit behind in-context learning, in a two-layer transformer: why it fails to use patterns that appear late in a long prompt, and how it combines pretrained bigram knowledge with patterns offered in context when predicting the next token. Framing attention and feed-forward weights as associative memories, the paper claims that relative positional encoding removes the first failure: the learned previous-token association $r_{-1}^\\top W_1^K w_E(v)$ is independent of the vocabulary item $v$ and of the position $t$, whereas with absolute positional encoding the same association decays as $1/t$ and late-sequence patterns are skipped. For the second question, it claims the next-token logits are a weighted sum of the pretrained bigram log-probability and the in-context frequency of trigger--output patterns, with pattern positions dropping out of the formula. These two claims turn vague intuitions about context hijacking and long-context attention into concrete equations that can be checked directly on trained weights and outputs.","feed_headline":"Relative encoding keeps induction heads attentive at any length","feed_subtitle":"Previous-token attention stays flat across the sequence, so long-prompt patterns are not skipped.","key_machinery":"The carrier of the argument is the associative-memory representation of weight matrices, $W = \\sum_{i,j} \\alpha_{i,j} u_i v_j^\\top$, read off through inner products $u_i^\\top W v_j$ under a near-orthogonality assumption on random embeddings. The central identity is the score $r_{-1}^\\top W_1^K w_E(v)$, which pairs the relative vector of the immediately preceding position with any token embedding; Theorem 2 shows this quantity is independent of both $v$ and the position $t$, so the first layer forms a position-independent previous-token head of the form $W_1^K = \\sum_{k \\in Q} w_E(k) r_{-1}^\\top$. A second piece is the key--value construction of the feed-forward block, whose keys detect token embeddings and whose values store the global distribution $\\log \\pi_b(u|v)$; scaling the three memories by coefficients $\\tau_1, \\tau_2, \\tau_3$ yields the logit formula of Proposition 3. Both results rest on a sequential one-step gradient-descent analysis that trains $W_2^O$, $W_2^K$, and $W_1^K$ in that order, with all other weights frozen.","core_discovery":"The paper's central claim is that the way positions enter the first attention layer changes what gradient descent writes into the key matrix. Following a prior associative-memory analysis, the paper studies a sequential training procedure in which $W_2^O$, then $W_2^K$, then $W_1^K$ each receive one gradient step from zero initialization, on sequences generated by a bigram model with trigger tokens. For relative positional encoding, the learned first-layer matrix becomes an associative memory pairing each token embedding $w_E(v)$ with the fixed previous-position vector $r_{-1}$, and the paper proves (Theorem 2 in the text, Theorem 3 in the appendix) that the score $r_{-1}^\\top W_1^K w_E(v)$ carries no dependence on $v$ or on $t$; the previous-token head therefore attends with constant strength at every sequence position, unlike the absolute-position case where the same score decays as $1/t$. The paper then constructs an explicit associative-memory transformer and shows its logits equal a weighted blend of the global bigram log-probability and the in-context counts of trigger--output pairs (Propositions 2 and 3), so where a pattern occurred in the prompt does not influence the final prediction. Experiments on a character-level bigram task and on an analogy task built from capital--country pairs are reported as evidence that trained networks behave this way.","pith_inferences":["A testable extension of the same mechanism: other relative-position schemes, such as rotary or linear-bias encodings, should also show flat memory recall at long positions if the position-independence is driven by the relative key structure rather than by the specific fixed-vector encoding used here.","The additive logit decomposition offers a quantitative diagnostic for context hijacking in larger models: if a misleading pattern's count shifts the output according to the same weighted-sum law, global and in-context knowledge compete additively, and the ratio $\\tau_1/\\tau_2$ measures how strongly pretraining resists the prompt.","Read backwards, the result locates absolute-position length-generalization failures inside the positional attention score itself, suggesting that making the score translation-invariant should be sufficient to restore late-sequence pattern use."],"forward_implications":["A transformer with relative positional encoding should keep its previous-token head working at every position of a sequence, including positions beyond the training length, while an absolute-position transformer's head weakens; this is exactly the length-generalization behaviour the memory-recall experiments measure.","At the final trigger token of a prompt, the logit for a vocabulary item is the sum of a global term proportional to $\\log \\pi_b(v|q)$ and an in-context term proportional to the frequency of the pattern $q v$, so the prediction shifts as pattern counts shift.","Positional information about where in the prompt a pattern occurred drops out of the final logit; only the frequencies of trigger--output pairs matter, by Proposition 3.","When the prompt contains out-of-distribution tokens, the learned induction head does not activate and the relative-encoding advantage disappears, as the paper notes in Remark 1."],"supporting_citations":[{"why":"Supplies the associative-memory framework, the bigram-with-triggers data model, the sequential one-step gradient-descent procedure, and the absolute-positioning baseline (the paper's Theorem 1) that the relative-positioning result is contrasted with.","marker":"[5]"},{"why":"Identifies the induction head as a circuit formed in two-layer attention-only transformers, the phenomenon this paper analyses.","marker":"[17]"},{"why":"Provides the evidence that induction heads underlie in-context learning, which motivates treating the head's behaviour as consequential.","marker":"[42]"},{"why":"Establishes the key--value view of feed-forward layers that the paper's construction of $W_1$ and $W_2$ as detectors and global-knowledge encoders relies on.","marker":"[19]"},{"why":"Its construction of absolute positions by a transformer block (cited as Theorem 4) is what the paper uses to build a three-layer induction head without positional encoding.","marker":"[29]"},{"why":"Supplies the near-orthogonality assumption on random embeddings that makes the associative-memory inner products $u_i^\\top W v_j$ well defined throughout.","marker":"[24]"},{"why":"Provides the memory-recall metric used to measure whether $W_1^K$ stores the intended previous-token associations in the experiments.","marker":"[13]"}],"fun_headline_variants":["Relative positions give induction heads constant attention","Previous-token attention stays flat with relative encoding","Associative memory blends bigram and in-context counts","Position-independent attention emerges from relative encoding","Induction heads ignore prompt position with relative encoding"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The proof that the previous-token association is position-independent trains the three weight matrices one at a time, from zero initialization, with a single gradient step each, and only on sequences where one trigger token appears exactly twice and ends the sequence; if ordinary end-to-end training does not follow this sequential path, the derived form of $W_1^K$ and the position-independence claim need not hold.","fun_headline_variants_meta":{"raw":{"variants":["Relative positions give induction heads constant attention","Previous-token attention stays flat with relative encoding","Associative memory blends bigram and in-context counts","Position-independent attention emerges from relative encoding","Induction heads ignore prompt position with relative encoding"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000837,"raw_usage":{"total_tokens":3659,"prompt_tokens":960,"completion_tokens":2699,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":576,"completion_tokens_details":{"reasoning_tokens":2630}},"tokens_in":576,"tokens_out":2699,"duration_ms":18494,"temperature":1.0,"reasoning_tokens":2630,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:56:38.723467+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a two-layer transformer with relative positional encoding end-to-end, updating all weights jointly, on the bigram-with-trigger task, and measure the memory-recall score for the association $r_{-1}^\\top W_1^K w_E(v)$ separately for early and late positions; if recall for positions $t > 128$ falls below recall for early positions, the claimed position independence is false.","supporting_citations":[{"cited_title":"Birth of a transformer: A memory viewpoint.Advances in Neural Information Processing Systems, 36, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the associative-memory framework, the bigram-with-triggers data model, the sequential one-step gradient-descent procedure, and the absolute-positioning baseline (the paper's Theorem 1) that the relative-positioning result is contrasted with."},{"cited_title":"The impact of positional encoding on length generalization in transform- ers.Advances in Neural Information Processing Systems, 36, 2024","cited_arxiv_id":null,"evidence_quote":"Its construction of absolute positions by a transformer block (cited as Theorem 4) is what the paper uses to build a three-layer induction head without positional encoding."}],"review_version":1}