{"id":"f32e6e15-73df-4d64-ab61-d2a5cde820fc","arxiv_id":"2608.10251","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Intermediate transformer states sit far from the output axis on purpose: that position insulates attention's cross-token mixing, and the frame can be prescribed in advance without loss.","lead":"A 12-layer transformer holds its working concepts in a subspace nearly orthogonal to the vocabulary read-out, and the paper shows this off-axis position protects cross-token mixing from blurring the prediction. The off-axis frame behaves like a free gauge that can be prescribed in training, without changing quality.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Off-axis angles and insulation ratios are measured with the logit lens, which the paper itself disowns as a mid-depth read-out; no calibration against the tuned lens is reported.","rationale":"The reader's weakest assumption identifies the same load-bearing concern, and I agree with it. The paper contains the evidence for its own vulnerability: Section 2.1 fixes the logit lens as the measuring instrument, while Section 10 identifies the tuned lens as the reading instrument and says the logit lens resolves little before the last layers. Because the headline numbers (angles, insulation ratios, forced-on-axis floor) are produced through the repudiated instrument, the functional-insulation claim is not yet independently verified. This is not a disagreement with consensus; it is an internal inconsistency in the measurement pipeline. The Procrustes additive-commit result and the gauge-prescription experiments are supported by independent controls and are less exposed. One calibration experiment would settle the matter, so the CONDITIONAL verdict is appropriate rather than a rejection. I therefore keep the reader's verdict unchanged.","tokens_in":29150,"tokens_out":9756,"duration_ms":109841,"concrete_test":"On one or more converged baselines, train a tuned lens (per-layer affine translator from residual states to final logits, following Belrose et al. 2023) on held-out data. Replace the frozen final-LN-plus-unembedding in the Section 2.1 write-angle definition and in the Section 3.1 rotation target with the tuned lens's layer-l projection for layers 0 through 7, then recompute Figure 2a and the 10 degree and 20 degree damage ratios in Figure 3. The insulation claim stands only if the tuned-lens angles remain in the reported 75–96 degree band and the damage ratios remain of order 64–84x. A shift toward 0 degrees or a collapse of the ratio toward the random control would show the off-axis geometry is a decoder artifact. Also report the norm of logit-lens-projected early writes versus tuned-lens projections as a direct blindness check.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that the off-axis workspace is functional insulation—rests on write-angle measurements defined in Section 2.1 by sending each sublayer write through the final LayerNorm and the tied unembedding (the logit lens), and on the Section 3.1 insulation experiment that rotates attention values toward the read-out using that same unembedding direction. But Section 10 and Table 1 name the tuned lens as the reading instrument and state that the logit lens resolves little before the last layers. No comparison of the two instruments is reported for the angles or for the perturbation target. If early- and mid-depth writes are written in per-layer bases that the frozen final unembedding projects into its null space, then the 75–96 degree angles, the 13 degree floor in Section 4, and the 64–84x damage ratio could be artifacts of a decoder blind before depth 8 rather than properties of the model. Section 9.4 itself concedes that the functional account is an inference; the measurement it is inferred from is exactly the uncalibrated logit-lens one. The additive-commit and gauge-prescription results are less exposed because they rely on orthogonal Procrustes and inserted rotations, but the headline insulation claim is not independently verified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper trains 12-layer GPT-2-small-class models on OpenWebText and studies the geometry of the residual stream relative to the tied unembedding. It reports that attention writes 75–96 degrees off the read-out at every depth, that rotating attention values toward the read-out is 64–84 times more damaging than a matched random rotation only when cross-token mixing is active, and that the final prediction is committed late by additive writes rather than by rotation of accumulated off-axis content. It then shows that forcing every layer on-axis reaches benchmark parity while reducing the concept-phase participation ratio, that prescribing the off-axis/on-axis phase structure through a loss is a lottery (2/8 seeds), and that inserting a fixed rotation at the phase boundary makes the prescription reliable (7/8 dense, 9/9 sparse) without changing quality. Finally it presents evidence that the choice of 90-degree frame is a free gauge: different device rotations produce the same quality, two baselines' concept frames are nearly orthogonal, and a randomly prescribed basis is adopted during training.","tokens_in":29422,"tokens_out":12528,"duration_ms":121890,"significance":"If the central claims hold, the paper reframes off-axis intermediate representations from an obstacle into a functional insulation mechanism and demonstrates that the off-axis frame is a prescribable gauge. The experimental design is stronger than is typical: matched random-rotation controls, an inserted-rotation control that recovers R^2=1.00, foreign-basis controls, explicit convergence-rate reporting, and candid limitation statements in Section 9.4. The paper also identifies that effective-dimensionality changes are invisible to standard benchmarks, which is a useful methodological point. The main risks are instrument validity and reproducibility: the headline angle measurements use the logit lens despite the paper's own statement that it fails before the last layers, a key intervention's target direction is underspecified, and the L_frame coefficient is missing. No code or data release is mentioned.","major_comments":[{"comment":"The write-angle statistic is defined by sending each sublayer write through the final LayerNorm and tied unembedding (the logit lens), and the headline claim that attention writes 75–96 degrees off the read-out at every depth (Figure 2a) comes from this statistic. However, Section 10 states that the logit lens 'fails on early and middle layers' and names the tuned lens as 'our reading instrument,' and Table 1 lists the tuned lens as the instrument for reading the concept phase. No calibration of the two instruments is reported for the write-angle measurement. If early- and mid-depth writes live in per-layer bases that the frozen final unembedding projects into or near its null space, the reported angles could be an artifact of the decoder rather than a property of the writes. Please either justify why the logit lens is the correct instrument for the angle-to-read-out claim (as opposed to reading latent predictions), or report the same angles under the tuned lens and show they agree.","section":"Section 2.1 and Section 10"},{"comment":"The insulation experiment is the load-bearing evidence for the paper's central functional claim, but the target direction used to 'rotate each attention head's value vectors toward the read-out' is never specified. It is not stated whether the target is the unembedding row of the next token, the direction of the final logits at that position, a projection of the value vector through the logit lens, or a global read-out subspace. The interpretation of the 64–84 times damage ratio as showing that read-out-aligned values are dangerous only when mixed depends on this construction, and the matched random-rotation control cannot compensate for an underspecified target. Please give the exact perturbation rule, including how the angle and axis are computed at each position and head.","section":"Section 3.1, Figure 3"},{"comment":"The frame-prescription experiment does not report the coefficient with which L_frame is added to the cross-entropy objective, despite Section 2's statement that all loss terms are added with a fixed coefficient. The claim that a randomly drawn basis is adopted with quality unchanged cannot be reproduced, and its sensitivity to the coefficient cannot be assessed, without this value. Please report the coefficient and, ideally, a small sweep or at least the convergence/failure count for the chosen value.","section":"Section 8.3, Eq. (8)"},{"comment":"The object whose motion is measured as the 'concept constellation' is never defined. The rigidity rotation is fitted to token residual states, but the text reports R^2=0.93 for carrying 'the whole concept constellation' from layer to layer and a 29-degree turn per layer. Without an explicit definition of the concept directions or the concept constellation, these numbers are statements about token residual clouds, and the inference that 'the frame the concept phase computes in' rotates is not supported as stated. Please define the concept directions used in the Procrustes/rigidity analysis, or rephrase the claims in terms of the token residual states that were actually measured.","section":"Sections 3.2–3.3"}],"minor_comments":[{"comment":"The sentence 'because it is orthogonal it preserves the residual norm and the participation ratio exactly' is true only at the instant of application; after training, the dense-device models show a concept-phase PR near 54 versus 24 for the baseline. Please reword to avoid implying the trained model's PR is preserved.","section":"Section 6.1"},{"comment":"The table caption and figure text describe the concept-phase participation ratio as an average over layers 0–6, but Equation 4 defines the concept phase as layers 0–5 with the boundary at layer 6. Please resolve this inconsistency and state which layer set is used in the reported PR means.","section":"Table 4 and Figure 15"},{"comment":"The statement 'Across all twenty-five, LAMBADA spans...' followed by 'Two further dense runs are set aside' is ambiguous as to whether the twenty-five includes the set-aside runs; please state the total number of runs, the number excluded, and the reason for exclusion in one place.","section":"Section 8.1"},{"comment":"The coefficient used for the angle constraint (Equation 3) is not stated; 'the same coefficient' refers back to a sweep reported for the distributional penalty, but the reader cannot tell which value was used for the fifteen-seed experiment. Please state it explicitly.","section":"Section 4.3"},{"comment":"No code or data release is mentioned; given the number of bespoke interventions (rotations, losses, schedules, screening criteria), a public release of the training and analysis code would substantially strengthen the paper.","section":"Reproducibility"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within the journal's scope and the experiments are expensive; I would encourage the editor to seek the instrument-calibration analyses in the major comments rather than new experiments at scale. The paper cites prior and concurrent work generously; the main concern is that the central measurements would be strengthened by a direct comparison with the tuned lens, which the authors already possess."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Mark's paper is worth a read if you work on residual-stream interpretability. The headline result is that the off-axis workspace is functional insulation, and the causal experiment behind it is genuinely well-built: rotating attention values toward the read-out is 64–84x more damaging than a matched random rotation, and the damage vanishes when cross-token averaging is removed. That is a real result, not a story. The additive-commit claim also holds up; the Procrustes decomposition with controls is the right way to ask whether the model turns its content onto the read-out, and the inserted-rotation control grounds the measurement.\n\nThe frame-prescription half is more speculative but internally consistent: demanding a phase structure through the loss is a lottery, while inserting one fixed rotation at the boundary makes it routine, and the frame is a free gauge the model will adopt if prescribed. The seed-attrition reporting is refreshingly honest.\n\nNow the soft spots. The logit-lens inconsistency is real: Section 2.1 defines write angles through the final LayerNorm and tied unembedding, and Section 10 says the tuned lens is the reading instrument and the logit lens resolves little before the last layers. The descriptive angle measurements should at least report projected norms or compare against the tuned lens. That said, I don't think it sinks the central insulation claim, because the perturbation experiment targets the read-out direction directly in residual space; the angle numbers are illustrative, not load-bearing. Still, the inconsistency needs fixing.\n\nThe other issues are practical: no code/data release, and at least one loss coefficient (for L_frame) is unspecified. Both are addressable. The paper also leans on a single architecture family and scale, but it says so clearly in Section 9.4.\n\nWho is it for: interpretability researchers, and people designing early-exit or representation-alignment training. It deserves a serious referee — conditional accept, not desk reject. I'd ask for calibration of the logit-lens angles, specification of all coefficients, and code.","headline":"Off-axis insulation is a real, well-measured result; fix the logit-lens inconsistency and release code before betting too hard on the frame-prescription story.","tokens_in":29923,"tokens_out":2111,"would_cite":true,"duration_ms":22011,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A transformer's hidden states sit nearly perpendicular to its answer on purpose, and that off-axis frame can be prescribed before training.","keywords":["off-axis representation","residual stream","logit lens","attention mixing","participation ratio","gauge freedom","early exit pressure","two-phase computation"],"falsifier":"Recompute the layer-by-layer write angles and the commit-span Procrustes split using per-layer learned translators (the tuned lens) instead of the frozen final decoder on layers 1–7; if early writes then point substantially toward the answer (angles far below $75$ degrees) or the commit $R^2$ rises well above $0.38$, the off-axis insulation and additive-commit claims would be artifacts of the read-out rather than properties of the model.","tokens_in":28939,"feed_emoji":"🎯","tokens_out":12236,"duration_ms":92483,"temperature":0.7,"pith_summary":"This paper argues that the off-axis position of a transformer's intermediate states—usually treated as an obstacle to interpretation—is functional. Across the first eight layers of a 12-layer GPT-2-class model, every sublayer writes into a subspace held near-orthogonal to the unembedding read-out, and rotating attention's values onto the read-out is $64$–$84\\times$ more damaging than a matched random rotation, with the damage entirely in cross-token mixing. The answer is then written on-axis, late, by addition rather than by turning accumulated content. The paper further shows that the off-axis frame is a free gauge that can be prescribed: inserting one fixed rotation at the phase boundary turns a loss-only prescription that converges two seeds in eight into one that converges seven in eight (dense) or nine in nine (sparse), at baseline quality. If true, the geometry is not a curiosity; it is a mechanism that protects composition from vocabulary blur, and a control surface for training.","feed_headline":"Transformer hidden states sit 90° off the answer on purpose","feed_subtitle":"Off-axis states protect the prediction from attention's blur; one fixed rotation makes the geometry easy to impose.","key_machinery":"The argument runs on four linked instruments. (1) The read-out axis: the direction(s) the tied unembedding reads, the fixed reference every angle is measured against. (2) The write-angle measurement: each sublayer's residual write is sent through the final layer-norm and tied unembedding (the logit lens) and its angle to the final logits computed; a value near $90^\\circ$ means the write lives in a subspace the unembedding discards. (3) The insulation experiment: a small rotation of attention's values toward the read-out, compared against a matched random rotation and against the same rotation with cross-token averaging removed, isolating the cost of mixing on-axis values. (4) The frame device: a fixed orthogonal rotation (dense Haar quarter-turn or sparse signed permutation) inserted once at the concept/token boundary, which hands the model an off-axis frame instead of requiring it to grow one under loss pressure. Two supporting measurements carry the load: the participation ratio counts the effective dimensionality of the workspace, and the orthogonal Procrustes decomposition separates rotation from freshly written content across a span of layers.","core_discovery":"On a 12-layer GPT-2-class transformer trained on OpenWebText, the paper claims, computation splits into two phases with a functional geometry. Through the concept phase (roughly layers 0–8), every sublayer writes into a subspace held near-orthogonal to the unembedding read-out: attention's writes sit $75$–$96^\\circ$ off it at every depth, and feed-forward writes about $84^\\circ$ until the last four layers. The off-axis position is not incidental: rotating attention's values onto the read-out is $64$–$84\\times$ more damaging to next-token loss than a matched random rotation, and a matched control that removes cross-token averaging collapses the asymmetry to about $1\\times$. The paper reads this as insulation: attention can average values across token positions without blurring vocabulary predictions, because the averaging happens in a subspace the unembedding discards. The answer is assembled additively and late—over the commit span a best-fit rotation explains only $R^2 = 0.38$ of the change, $64\\%$ of the committed state is newly written content, the residual norm grows by $1.74\\times$, and the final prediction first becomes the top decoded token at layer $9.2$ on average. The paper further claims the frame is a free gauge: $25$ runs across $13$ distinct $90^\\circ$ rotations reach the same quality, two baselines from different seeds hold concepts in frames about $90^\\circ$ apart while agreeing on their read-outs, and a basis drawn at random and prescribed before training is adopted with quality unchanged. Supplying a fixed rotation at the phase boundary—dense Haar or sparse signed permutation—makes the prescribed geometry reliably trainable: convergence under the loss-only step schedule rises from $2/8$ seeds to $7/8$ (dense) or $9/9$ (sparse) at baseline quality, while forcing every layer on-axis, as early-exit training does, matches perplexity, LAMBADA, and BLiMP but collapses the concept-phase workspace from about $25$ to $14$ effective dimensions.","pith_inferences":["The insulation account predicts that increasing attention's mixing pressure (more heads, higher temperature, or longer contexts) should push the converged geometry closer to $90^\\circ$; a seed sweep varying mixing strength would test this without new machinery.","The gauge-freedom result suggests a cheap comparability trick the paper does not fully exploit: training a family of models into one prescribed basis would let differences in concept geometry be read directly as differences in computation, rather than being aligned away post hoc.","The sparse-device convergence edge ($9/9$ vs $5/9$) points beyond interpretability: fixing an absorbable signed-permutation rotation at mid-depth may serve as a general training-stabilization intervention for transformers, testable at larger scales where from-scratch divergence rates are higher.","The invisible workspace collapse under early-exit pressure raises a testable concern: models that match quality on next-token benchmarks while losing half their effective dimensionality may degrade on tasks that use the removed directions, such as long-range dependency or compositional generalization."],"forward_implications":["A read-out-alignment penalty can force a transformer on-axis, but the resulting model writes the answer by near-cancellation: feed-forward contributes $+9103$ to the answer logit while attention writes $-8113$ against it, so on-axis quality is real but the model is a caution, not a solution.","Training for early exit (pressing every layer onto the read-out) can match perplexity, LAMBADA, and BLiMP while halving the concept-phase effective dimensionality from about $25$ to $14$; benchmarks alone would not detect the change.","Prescribing an off-axis/on-axis two-phase schedule through the loss alone is a lottery ($2/8$ seeds converge), while inserting one fixed rotation at the boundary makes the same schedule land reliably ($7/8$ dense, $9/9$ sparse) at baseline quality.","The $90^\\circ$ frame is a free gauge: thirteen distinct rotations across $25$ runs yield the same model, two same-recipe baselines land in near-orthogonal concept frames, and a randomly drawn prescribed basis is adopted without quality loss.","If the geometry is what the paper claims, the off-axis position is a control surface: models can be trained into a chosen frame, making intermediate representations directly comparable across runs and across models."],"supporting_citations":[{"why":"Introduces the logit lens, the direct frozen-decoder read-out whose blind spots motivate the paper's geometric measurements.","marker":"[34]"},{"why":"Introduces the tuned lens, the per-layer translator the paper names as its reading instrument and contrasts with the frozen logit lens.","marker":"[4]"},{"why":"Documents the off-axis condition as a structural consequence of residual connections and motivates the paper's complementary functional reading.","marker":"[31]"},{"why":"Provides the closed-form orthogonal Procrustes solution used to split commit-span changes into rotation versus freshly written content.","marker":"[43]"},{"why":"Applies transition-wise Procrustes across depth in six instruction-tuned models, the layer-relative prior the paper extends by referencing the unembedding.","marker":"[5]"},{"why":"Establishes that the residual stream has no privileged basis, which makes the fixed rotation device a usable intervention rather than a trivial one.","marker":"[10]"},{"why":"Shows signed-permutation rotations are near-exact gauge symmetries under RMSNorm, explaining why the sparse device can be absorbed by the network.","marker":"[47]"},{"why":"Measures from-scratch divergence rates at the same model and corpus scale, grounding the paper's five-of-nine baseline convergence rate.","marker":"[27]"},{"why":"Represents the early-exit line that pushes intermediate layers onto the read-out, whose representation-level cost the paper quantifies.","marker":"[13]"}],"fun_headline_variants":["Off-axis states shield predictions from attention's blur","Why transformers compute concepts 90° off the answer","A fixed rotation makes the transformer's off-axis geometry trainable","Hidden states dodge the read-out to protect composition","Rotating the concept frame: a free gauge for transformers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The angle measurements assume that reading every layer's write through the model's final output decoder tells you where that write points, even in early and middle layers where the paper itself says direct decoding shows little; if early writes are only legible through per-layer translators, the measured off-axis angles could be an artifact of the decoder's blind spot.","fun_headline_variants_meta":{"raw":{"variants":["Off-axis states shield predictions from attention's blur","Why transformers compute concepts 90° off the answer","A fixed rotation makes the transformer's off-axis geometry trainable","Hidden states dodge the read-out to protect composition","Rotating the concept frame: a free gauge for transformers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000274,"raw_usage":{"total_tokens":1837,"prompt_tokens":1341,"completion_tokens":496,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":957,"completion_tokens_details":{"reasoning_tokens":418}},"tokens_in":957,"tokens_out":496,"duration_ms":5075,"temperature":1.0,"reasoning_tokens":418,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:10:56.389952+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the layer-by-layer write angles and the commit-span Procrustes split using per-layer learned translators (the tuned lens) instead of the frozen final decoder on layers 1–7; if early writes then point substantially toward the answer (angles far below $75$ degrees) or the commit $R^2$ rises well above $0.38$, the off-axis insulation and additive-commit claims would be artifacts of the read-out rather than properties of the model.","supporting_citations":[{"cited_title":"Residual connections and the causal shift: Uncovering a structural misalignment in transformers.arXiv preprint arXiv:2602.14760, 2026","cited_arxiv_id":null,"evidence_quote":"Documents the off-axis condition as a structural consequence of residual connections and motivates the paper's complementary functional reading."},{"cited_title":"A generalized solution of the orthogonal Procrustes problem.Psychome- trika, 31(1):1–10, 1966","cited_arxiv_id":null,"evidence_quote":"Provides the closed-form orthogonal Procrustes solution used to split commit-span changes into rotation versus freshly written content."},{"cited_title":"An Analysis of Residual-Stream Geometry Across Transformer Depth","cited_arxiv_id":"2607.18348","evidence_quote":"Applies transition-wise Procrustes across depth in six instruction-tuned models, the layer-relative prior the paper extends by referencing the unembedding."},{"cited_title":"A mathematical framework for transformer circuits.Transformer Circuits Thread, 2021.https://transformer-circuits","cited_arxiv_id":null,"evidence_quote":"Establishes that the residual stream has no privileged basis, which makes the fixed rotation device a usable intervention rather than a trivial one."},{"cited_title":"Signed-Permutation Coordinate Transport for RMSNorm Transformers","cited_arxiv_id":"2606.31963","evidence_quote":"Shows signed-permutation rotations are near-exact gauge symmetries under RMSNorm, explaining why the sparse device can be absorbed by the network."}],"review_version":1}