{"id":"ea0905ba-cda5-4d03-8991-52a5c69dd7dd","arxiv_id":"2606.31963","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Introduces signed-permutation gauge B_d for RMSNorm models and sign-marginalized Hungarian matching, showing improved coordinate recovery along fine-tuning trajectories and better transfer of SAEs and steering vectors.","lead":"This paper identifies that RMSNorm transformers have a signed-permutation symmetry in residual stream coordinates, unlike the permutation symmetry in LayerNorm models. Accounting for this symmetry via sign-marginalized matching and trajectory-based transport improves feature alignment across checkpoints for interpretability tools.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"No significant objection identified","rationale":"Reader's weakest_assumption (RMSNorm gauge = B_d) is exactly the modeling step required for the symmetry-incompleteness claim and all downstream empirical results to hold. No other load-bearing step shows a comparable risk of failure from the given description.","tokens_in":1878,"tokens_out":266,"duration_ms":35369,"concrete_test":"For a minimal RMSNorm + linear layer, explicitly construct the transformed weights and gain under an arbitrary element of B_d and verify that the output distribution is identical; if any signed permutation fails to be absorbed, the gauge group is smaller than claimed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"No significant objection identified. The central claim—that B_d gauge transport along trajectories recovers coordinates and enables tool transfer where S_d fails—rests on the architecture-dependent gauge modeling (RMSNorm admits signed-permutation freedom via generic per-channel gain) plus the empirical recovery numbers. The abstract states the symmetry distinction, proves a sharp ceiling for raw signed-correlation matching under decorrelation, and reports concrete transfer gains (91.1 % vs 60.3 %, NMSE 0.004 vs 1.08). No internal inconsistency, hidden assumption, or derivation gap is apparent.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper claims that RMSNorm residual streams admit a signed-permutation gauge B_d = S_d ⋉ {±1}^d (due to generic per-channel gains) while LayerNorm admits only a permutation gauge S_d. It introduces sign-marginalized Hungarian matching, proves a sharp accuracy ceiling for raw signed-correlation matching under decorrelated coordinates, and shows that composing saved-checkpoint local B_d gauges along same-base fine-tuning trajectories recovers 91.1% of cross-run coordinates at 1500 steps (vs. 60.3% for endpoint matching). The recovered gauge improves tool transfer: TinyLlama SAE NMSE drops to 0.004 (vs. 1.08 under S_d), Qwen sentiment steering retains 95.8% effect (vs. 17.2%), refusal steering avoids sign reversal, and signed AdamW state transport preserves resumed trajectories.","tokens_in":1968,"tokens_out":604,"duration_ms":29079,"significance":"If the gauge distinction and empirical recovery numbers hold, the work supplies a concrete, architecture-aware procedure for coordinate transport that directly improves reproducibility of interpretability artifacts (SAEs, steering vectors, neuron sets) and stateful training resumption in RMSNorm models. The proof of the matching ceiling and the explicit separation of gauge transport from function-level merging are reusable contributions.","major_comments":[{"comment":"§4 (proof of sharp ceiling): the decorrelation assumption used to derive the positive-sign-fraction bound should be checked against the actual coordinate covariances of the TinyLlama and Qwen models at the layers where the 91.1% recovery is reported; if coordinates remain correlated, the ceiling may not be tight and the advantage of sign-marginalization could be overstated.","section":"§4"},{"comment":"Table 2 / §5.2 (recovery percentages): the 91.1% vs 60.3% figures at 1500 steps are load-bearing for the central claim; the manuscript should report the number of independent fine-tuning runs, the exact definition of “recovered coordinate,” and a control that isolates the contribution of intermediate checkpoints from simple base-model routing.","section":"§5.2"}],"minor_comments":[{"comment":"Notation: the semidirect product B_d = S_d ⋉ {±1}^d is introduced without an explicit action or matrix representation; adding a one-sentence definition or small example matrix in §2 would remove ambiguity.","section":"§2"},{"comment":"Figure 3 (SAE NMSE): the reported values 0.004 and 1.08 lack error bars or the number of SAE training seeds; adding these would strengthen the transfer claim.","section":"Figure 3"}],"recommendation":"minor_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the careful review and the recommendation of minor revision. The comments help strengthen the empirical grounding of the theoretical claims and improve experimental reporting. We address each major comment below.","responses":[{"response":"We agree that validating the decorrelation assumption is necessary to assess the tightness of the bound. In the revised manuscript we will compute and report the average absolute pairwise coordinate correlations at the layers and checkpoints used for the 91.1% recovery figures in both TinyLlama and Qwen. If non-negligible correlations are observed, we will explicitly discuss the implications for the sign-marginalization advantage and note that the reported gains remain an empirical lower bound under the observed covariance structure.","revision_made":"yes","referee_comment":"[§4] §4 (proof of sharp ceiling): the decorrelation assumption used to derive the positive-sign-fraction bound should be checked against the actual coordinate covariances of the TinyLlama and Qwen models at the layers where the 91.1% recovery is reported; if coordinates remain correlated, the ceiling may not be tight and the advantage of sign-marginalization could be overstated."},{"response":"We will add the requested details. The reported percentages are means over four independent fine-tuning runs initialized from the identical base checkpoint. A coordinate is counted as recovered when both its sign and its permuted index match the reference trajectory under the composed gauge. We will also include a new control that routes only through the base-model gauge (without composing intermediate checkpoints) to isolate the contribution of the trajectory composition.","revision_made":"yes","referee_comment":"[§5.2] Table 2 / §5.2 (recovery percentages): the 91.1% vs 60.3% figures at 1500 steps are load-bearing for the central claim; the manuscript should report the number of independent fine-tuning runs, the exact definition of “recovered coordinate,” and a control that isolates the contribution of intermediate checkpoints from simple base-model routing."}],"tokens_in":1591,"tokens_out":443,"duration_ms":21449,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The core point is that RMSNorm residual streams carry an extra sign freedom per coordinate on top of permutations, so plain permutation matching is incomplete for these models. The paper defines the gauge as B_d = S_d ⋉ {±1}^d when per-channel gains are generic, introduces sign-marginalized Hungarian matching to handle it, and shows that composing saved local gauges along fine-tuning trajectories recovers 91.1% of cross-run coordinates at 1500 steps versus 60.3% for endpoint matching.\n\nWhat stands out is the clean separation between the symmetry claim and the procedure. The proof that raw signed-correlation matching hits a structural ceiling at the positive-sign fraction under decorrelation is straightforward and useful. The empirical transfers are concrete: TinyLlama SAE NMSE drops from 1.08 to 0.004, Qwen sentiment steering holds 95.8% effect instead of 17.2%, and refusal steering avoids sign reversal. The AdamW state transport result is also direct evidence that the gauge choice affects training dynamics, not just post-hoc alignment.\n\nThe soft spots are mostly scope and verification. The recovery numbers rely on same-base trajectories; it is not yet clear how well the method extends to unrelated checkpoints or different optimizers. The audits on interpretability reproducibility are mentioned but not quantified in detail here. The TinyLlama and Qwen experiments are narrow, so the practical gain could be model- or scale-specific until more cases are shown.\n\nThis is aimed at people doing mechanistic interpretability, SAE training, or steering on RMSNorm models who already move features across checkpoints. The math and the failure-mode proof are solid enough that a serious referee should see it; the empirical claims are falsifiable and worth checking in review.","headline":"RMSNorm has a signed-permutation gauge that permutation alignment misses, and transporting local gauges along trajectories recovers coordinates and fixes downstream tool breakage where standard methods fail.","tokens_in":2445,"tokens_out":433,"would_cite":true,"duration_ms":11793,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"RMSNorm residual streams admit a signed-permutation gauge that permutation-only alignments miss, so composing local gauges along fine-tuning paths recovers most cross-run coordinates.","keywords":["RMSNorm","gauge symmetry","coordinate alignment","transformers","sparse autoencoders","steering vectors","model merging","AdamW state"],"falsifier":"On a suite of RMSNorm models, sign-marginalized matching fails to raise coordinate recovery above the raw signed-correlation ceiling once sign correlations are removed from the data.","tokens_in":2758,"feed_emoji":"","tokens_out":807,"duration_ms":18387,"temperature":0.7,"pith_summary":"The paper shows that LayerNorm and RMSNorm fix different residual-stream gauges: LayerNorm permits only permutations of coordinates up to global sign, while RMSNorm with per-channel gains permits signed permutations. Standard permutation matching therefore leaves an unaccounted sign degree of freedom. The authors replace raw signed-correlation matching with sign-marginalized Hungarian matching and demonstrate that chaining the local signed-permutation gauges recovered at each saved checkpoint along a fine-tuning trajectory recovers 91.1 percent of coordinates at 1500 steps. The same gauge choice determines whether downstream tools survive transport: sparse-autoencoder reconstruction error, steering-vector effect size, and even the sign of refusal behavior all degrade sharply under the wrong gauge.","feed_headline":"Signed gauges recover 91% of RMSNorm coordinates across fine-tunes","feed_subtitle":"Composing local B_d gauges along trajectories preserves SAE features and steering effects that permutation matching destroys.","key_machinery":"The signed-permutation gauge B_d together with sign-marginalized Hungarian matching for coordinate transport.","core_discovery":"The residual-stream gauge for RMSNorm with generic per-channel gain is the signed-permutation group B_d = S_d ⋉ {±1}^d. Composing saved-checkpoint local B_d gauges along same-base fine-tuning trajectories recovers 91.1% of cross-run coordinates at 1500 steps versus 60.3% for endpoint matching. The recovered gauge transfers tools that permutation-only alignment breaks: TinyLlama SAE reconstruction has NMSE 0.004 under B_d versus 1.08 under S_d; Qwen sentiment steering preserves 95.8% of its effect versus 17.2%; refusal steering reverses sign under S_d; coordinate-preserving merges behave the same way. Signed transport of AdamW state preserves the resumed trajectory while permutation-only stat","pith_inferences":["Many published neuron or feature attributions on RMSNorm models may be gauge-dependent and require re-checking once the signed gauge is fixed.","The same local-gauge composition technique could be tested on other training regimes such as continued pre-training or reinforcement learning from human feedback.","If the signed gauge is architecture-dependent, then any method that aligns checkpoints across mixed LayerNorm and RMSNorm families must first convert between the two gauges.","Gauge-sweep audits become a standard reproducibility check for any claim that names specific coordinate indices."],"forward_implications":["Sparse autoencoder reconstruction error on TinyLlama drops from 1.08 to 0.004 when the correct gauge is used.","Sentiment steering vectors retain 95.8 percent of their effect size under signed transport but only 17.2 percent under permutation transport.","Refusal steering vectors reverse sign when transported under the wrong gauge.","AdamW optimizer state transported with the signed gauge resumes the original training trajectory; permutation transport produces a different trajectory from a functionally identical checkpoint.","Index-level interpretability claims hold only relative to an explicit gauge."],"fun_headline_variants":["Signed B_d gauges recover 91% RMSNorm coordinates","Composing B_d gauges recovers 91% coordinates","B_d gauge composition beats 60% endpoint matching","Signed transport aligns RMSNorm coordinates at 91%"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"RMSNorm residual charts with generic per-channel gain really do have signed-permutation gauge freedom rather than pure permutation freedom.","fun_headline_variants_meta":{"raw":{"variants":["Signed B_d gauges recover 91% RMSNorm coordinates","Composing B_d gauges recovers 91% coordinates","B_d gauge composition beats 60% endpoint matching","Signed transport aligns RMSNorm coordinates at 91%"]},"model":"grok-4.3","cost_usd":0.005595,"raw_usage":{"total_tokens":2777,"prompt_tokens":863,"num_sources_used":0,"completion_tokens":61,"cost_in_usd_ticks":55949500,"prompt_tokens_details":{"text_tokens":863,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1853,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":863,"tokens_out":61,"duration_ms":17841,"temperature":1.0,"reasoning_tokens":1853,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-01T06:22:57.110044+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"On a suite of RMSNorm models, sign-marginalized matching fails to raise coordinate recovery above the raw signed-correlation ceiling once sign correlations are removed from the data.","supporting_citations":[],"review_version":1}