{"id":"a8f9b460-497f-4206-9b2c-1b559e905479","arxiv_id":"2605.25880","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Residual-free transformers retain near-Gaussian activations and show significantly better robustness to low-bit quantization than residual models after training with orthogonal initialization and related techniques.","lead":"This paper finds that residual connections in transformers increase non-Gaussianity in activations, worsening low-bit quantization performance. Smart generalists might read it because it suggests redesigning transformer architectures for better compressibility using residual-free designs and special training methods.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.3","headline":"Specialized training techniques applied only to residual-free models may confound attribution of Gaussianity and quantization gains to architecture alone","rationale":"Matches the reader's weakest assumption on fair comparisons exactly; the concern is internal to the experimental design rather than external consensus.","tokens_in":1737,"tokens_out":262,"duration_ms":22463,"concrete_test":"Retrain the residual baseline models using identical orthogonal initialization, spectral optimization, and depth-aware scaling; recompute activation excess kurtosis and 4-bit/8-bit quantization error on the same language tasks. If kurtosis drops and quantization robustness improves comparably to the residual-free case, the architecture-level claim is weakened.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The claim that residual connections drive excess kurtosis and quantization error rests on controlled comparisons, yet residual-free models require orthogonal initialization, spectral/second-order optimization, and depth-aware attention temperature scaling to train at all. These interventions are not applied to the residual baselines. If the same techniques also reduce kurtosis or outliers in residual models, the architectural mixing explanation (dense vs residual) cannot be isolated from optimization effects. The kurtosis analysis in the paper would then attribute to residuals an effect that is at least partly due to mismatched training regimes.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper claims that residual connections in transformers drive activations away from Gaussianity by amplifying excess kurtosis during training, resulting in higher quantization error and accuracy loss at low precision. In contrast, residual-free transformers maintain near-Gaussian activations and show substantially better robustness to low-bit quantization (with only a small full-precision accuracy drop), which the authors attribute to dense mixing contracting non-Gaussianity. They support this via controlled empirical comparisons, an excess kurtosis analysis of residual vs. dense mixing, and demonstrate that residual-free models can be trained using orthogonal initialization, spectral/second-order optimization, and depth-aware attention temperature scaling.","tokens_in":1831,"tokens_out":617,"duration_ms":25129,"significance":"If the central claims hold after addressing controls, the work would identify a previously under-appreciated architecture-level accuracy-compressibility trade-off in transformers and motivate residual-free designs for quantization-friendly models. It gives credit for attempting controlled comparisons between architectures and for providing a kurtosis-based mechanistic explanation rather than purely empirical observation.","major_comments":[{"comment":"The controlled comparisons central to the claim (abstract and §3/§4) apply orthogonal initialization, spectral or second-order optimization, and depth-aware attention temperature scaling only to residual-free models. It is not stated whether these same techniques were applied to the residual baselines; if they reduce kurtosis or quantization error when used on residual models, the attribution of non-Gaussianity and quantization degradation specifically to residual connections (vs. mismatched optimization regimes) cannot be isolated. This directly affects the load-bearing claim that residuals drive the effect.","section":"Training methodology and controlled comparisons (abstract; §3)"},{"comment":"The excess kurtosis analysis (abstract; likely §2 or §5) asserts that residual mixing amplifies non-Gaussianity while dense mixing contracts it. Without the explicit mixing equations or derivation showing how the kurtosis update depends on the residual vs. dense structure independent of the training interventions, it is unclear whether the analysis fully rules out confounding from the specialized optimizers used only on the residual-free side.","section":"Excess kurtosis analysis"}],"minor_comments":[{"comment":"The abstract refers to 'language tasks' and 'low precision' without naming the specific datasets, model sizes, or bit-widths (e.g., 4-bit vs. 8-bit) used in the quantization experiments; adding these would improve reproducibility.","section":"Abstract"},{"comment":"Figure captions and table headers should explicitly state whether error bars represent standard deviation over seeds or runs, and whether the residual baselines received any of the listed training techniques.","section":"Figures and tables"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's fit to a machine-learning theory or systems journal is reasonable, but the citation list should be checked for prior work on activation statistics and outlier mitigation in transformers to ensure novelty is accurately positioned."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive comments. We address each major comment below and indicate planned revisions to strengthen the controlled comparisons and kurtosis analysis.","responses":[{"response":"The specialized techniques were introduced specifically to stabilize training of residual-free models, which diverge under standard protocols; residual baselines follow the conventional training regime from prior literature. We acknowledge the potential for confounding and will add experiments applying orthogonal initialization and spectral optimization to residual models, reporting the resulting kurtosis and quantization metrics in the revision to isolate the architectural contribution.","revision_made":"yes","referee_comment":"The controlled comparisons central to the claim (abstract and §3/§4) apply orthogonal initialization, spectral or second-order optimization, and depth-aware attention temperature scaling only to residual-free models. It is not stated whether these same techniques were applied to the residual baselines; if they reduce kurtosis or quantization error when used on residual models, the attribution of non-Gaussianity and quantization degradation specifically to residual connections (vs. mismatched optimization regimes) cannot be isolated. This directly affects the load-bearing claim that residuals drive the effect."},{"response":"Section 5 derives the kurtosis evolution from the mixing equations for residual addition versus dense mixing. We will expand this section with the full step-by-step equations and derivation to explicitly demonstrate that the kurtosis update depends only on the mixing structure and is independent of optimizer choice.","revision_made":"yes","referee_comment":"The excess kurtosis analysis (abstract; likely §2 or §5) asserts that residual mixing amplifies non-Gaussianity while dense mixing contracts it. Without the explicit mixing equations or derivation showing how the kurtosis update depends on the residual vs. dense structure independent of the training interventions, it is unclear whether the analysis fully rules out confounding from the specialized optimizers used only on the residual-free side."}],"tokens_in":1431,"tokens_out":408,"duration_ms":36507,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main thing to know is that this paper links residual connections to excess kurtosis and worse low-bit quantization performance, while showing residual-free versions stay closer to Gaussian and quantize better. The training differences between the two setups, however, make it hard to isolate architecture as the cause.\n\nThe work is new in tying residual mixing to amplified non-Gaussianity through kurtosis analysis and then demonstrating the quantization payoff on language tasks. The empirical side shows a clear accuracy-compressibility trade-off, with residual-free models holding up under low precision despite a modest full-precision drop. The kurtosis framing gives a concrete mechanism rather than just reporting error rates.\n\nThe comparisons are a step forward for thinking about architecture choices that aid deployment. The idea that dense mixing can contract outliers is worth testing further.\n\nThe soft spot is the training regime. Residual-free models use orthogonal initialization, spectral or second-order optimization, and depth-aware attention scaling that the residual baselines do not receive. If those same interventions also reduce kurtosis when applied to residual models, the architectural explanation loses force. The paper should have matched the training conditions to separate the effects cleanly.\n\nThis is for people working on efficient transformer deployment and quantization-aware design. Readers focused on hardware constraints or alternative architectures will find the results and the kurtosis account useful. It deserves a serious referee because the topic matters and the core observation is testable, even if the current evidence needs tightening on the controls.\n\nI would send it to review with a request to rerun the comparisons under matched training.","headline":"Residual-free transformers look more quantization-friendly due to Gaussian activations, but mismatched training methods weaken the claim that residuals are the main driver.","tokens_in":2332,"tokens_out":384,"would_cite":false,"duration_ms":20489,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Residual connections drive transformer activations away from Gaussianity, raising quantization error at low precision, while residual-free models retain Gaussian activations and quantize more robustly.","keywords":["transformers","quantization","residual connections","activation distributions","low-bit precision","kurtosis","gaussianity"],"falsifier":"Train matched residual and residual-free transformers on the same language task, then compare the kurtosis of their activations or their accuracy drop when both are quantized to 4 bits or lower.","tokens_in":2639,"feed_emoji":"","tokens_out":405,"duration_ms":17783,"temperature":0.7,"pith_summary":"The paper shows that residual connections in transformers push activations toward heavy-tailed, non-Gaussian distributions during training. This architectural choice increases quantization error and accuracy loss when models are reduced to low-bit precision. Residual-free transformers, trained with orthogonal initialization, spectral optimization, and depth-aware attention scaling, keep activations closer to Gaussian. These models incur only a small full-precision accuracy cost yet degrade far less under quantization on language tasks. The work frames an accuracy-compressibility trade-off that can be addressed at the architecture level rather than only through quantizer design.","feed_headline":"Residual-free transformers keep activations Gaussian for better quantization","feed_subtitle":"Residual connections amplify non-Gaussianity and raise low-bit error; removing them improves robustness after special training.","key_machinery":"Excess kurtosis analysis of how residual versus dense mixing affects activation distributions during training.","core_discovery":"Residual mixing amplifies non-Gaussianity in transformer activations as measured by excess kurtosis, while dense mixing in residual-free transformers contracts non-Gaussianity; the latter architecture therefore exhibits substantially lower quantization error once made trainable.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["Residual-free transformers contract non-Gaussianity for quantization","Residual mixing amplifies non-Gaussianity harming low-bit quantization","Residual-free models reduce kurtosis for better transformer quantization","Removing residuals contracts non-Gaussianity in transformer activations"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The controlled comparisons between residual and residual-free transformers, along with the added training techniques, produce fairly comparable models without confounding the quantization results.","fun_headline_variants_meta":{"raw":{"variants":["Residual-free transformers contract non-Gaussianity for quantization","Residual mixing amplifies non-Gaussianity harming low-bit quantization","Residual-free models reduce kurtosis for better transformer quantization","Removing residuals contracts non-Gaussianity in transformer activations"]},"model":"grok-4.3","cost_usd":0.006572,"raw_usage":{"total_tokens":3046,"prompt_tokens":619,"num_sources_used":0,"completion_tokens":63,"cost_in_usd_ticks":65724500,"prompt_tokens_details":{"text_tokens":619,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2364,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":619,"tokens_out":63,"duration_ms":17431,"temperature":1.0,"reasoning_tokens":2364,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-29T22:49:52.496648+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Train matched residual and residual-free transformers on the same language task, then compare the kurtosis of their activations or their accuracy drop when both are quantized to 4 bits or lower.","supporting_citations":[],"review_version":1}