{"id":"144d66df-fc4b-4632-861b-7a1333baa0aa","arxiv_id":"2608.12149","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Massive activations in hybrid linear attention LLMs form pre-attention spikes before full attention layers and inter-spike plateaus that grow with full attention density.","lead":"This paper finds that in hybrid linear attention language models, the largest internal activations spike immediately before full attention layers and can persist as plateaus across the intervening linear attention layers. The pattern becomes more pronounced as full attention layers become denser, which could inform architecture design and quantization for these increasingly common models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The PAS/ISP morphology is quantified by token-level max-absolute activation, not by a sparse-outlier metric; if pre-attention peaks are diffuse vector scaling rather than massive outlier features, the central MA claim is unsupported.","rationale":"The reader's weakest-assumption identifies the single consensus sink token as the main risk, with the non-sink controls in Appendices B.2 and B.5 as mitigation. That is a real concern, but the more load-bearing issue is feature-level: even if the traced token is correct, the quantitative definitions of PAS and ISP use only the maximum absolute hidden-state value for that token, which does not establish that the observed spikes are massive (sparse, extreme) activations. The paper's mechanism section partly addresses this with a fixed-coordinate analysis, but only for one coordinate, and the appendix layerwise decompositions select a new dominant feature per layer, so they do not confirm the same outlier persists. The proposed test would settle the issue by checking whether the same layerwise morphologies appear under a sparsity-sensitive metric. If they do, the central claim is strengthened; if not, the paper's use of the term 'massive activations' is overstated and the contribution would need to be reframed as studying token-level activation envelopes. Given the breadth of cross-model and controlled-pretraining evidence for the underlying layerwise pattern, I do not think this concern alone changes the reader's CONDITIONAL verdict, but it should be addressed before acceptance.","tokens_in":41253,"tokens_out":6316,"duration_ms":63757,"concrete_test":"Recompute the layerwise PAS/ISP trajectories for the consensus sink token in the M-A-P 1.3B GDN 12:1 and 3:1 checkpoints on the running example using an outlier sparsity measure, e.g., r_l = max_j |X_{t*,j}| / median_j |X_{t*,j}| at each layer l, together with the count of features exceeding 1% of the layer maximum. Then recompute Align and ISR with r_l in place of m^(l). If pre-attention layers do not systematically maximize r_l, or if the feature count is not sparse at those layers, the PAS/ISP morphology is an artifact of token-level vector scaling rather than massive activations; if r_l spikes at the same pre-attention layers, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3 defines the traced signal as m^(l)_{x,t*} = ||X^(l)_{x,t*,:}||_infty, the maximum absolute activation across features for one sink token, and builds both Align (Eq. 6) and ISR (Eq. 7) on this scalar. This is a token-level envelope, not a test that the value is a massive activation in the sense used by Sun et al. (2024): an entry that exceeds typical activations by orders of magnitude and is sparse across features. A pre-attention peak in m could arise if the entire hidden vector of the sink token is uniformly scaled up before a full attention layer, with no single outlier feature. The non-sink token controls in Appendix B.5 address which token is traced, not whether the traced maximum is a sparse outlier. The fixed-coordinate write-sink-cancel evidence in Section 4 and Appendix E.1 is verified for one token-feature coordinate in one 1.3B GDN model, and the cross-model systematic-outlier appendix (E.2) again uses layerwise dominant-feature magnitudes rather than a sparsity or outlierness measure. Thus the central morphological claims and the claimed recovery of the full-attention MA morphology rest on a metric that cannot distinguish 'massive activation spikes' from 'token-state scaling events.' This is load-bearing because the paper's novelty is specifically about massive activations in HLA LLMs; if the spikes are not sparse outliers, the phenomenon may still be real as an activation pattern, but it is not the MA phenomenon claimed.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents the first systematic study of massive activations (MAs) in layer-interleaved hybrid linear attention (HLA) LLMs. It introduces an attention-sink-guided tracing procedure that follows a consensus sink token across depth and documents two layerwise morphologies: pre-attention spikes (PAS), in which activation magnitude peaks immediately before full attention layers, and inter-spike plateaus (ISP), in which elevated activation persists between successive PAS. Using a controlled suite of five linear attention architectures across hybridization ratios and model scales, twelve large-scale pretrained hybrid checkpoints, five input domains, and controlled pretraining of GDN-based models, the paper argues that full attention placement is the primary organizer of MA morphology. It also proposes a write–sink–cancel lifecycle account, with PAS attributed to localized cancellation and ISP to delayed cancellation, and claims that increasing full attention density recovers the stable MA morphology of full attention LLMs.","tokens_in":41592,"tokens_out":4286,"duration_ms":42143,"significance":"If the morphological claims hold, this is a useful descriptive contribution to the interpretability of hybrid linear attention architectures, which are increasingly deployed in production LLMs. The paper's strengths include a controlled model suite, evaluation across diverse large-scale checkpoints and domains, quantitative metrics with paired bootstrap confidence intervals, explicit token and layer controls, and publicly available analysis code and checkpoints. The metrics are descriptive and involve no fitted parameters, so circularity is not a concern. The main risk is that the central object being measured is a token-level activation envelope rather than the sparse-outlier phenomenon that defines MAs; this is fixable with additional sparsity-oriented measurements and does not invalidate the descriptive morphology itself.","major_comments":[{"comment":"The central empirical signal is m(l)_{x,t} = ||X(l)_{x,t,:}||_inf, the maximum absolute activation across features for one token. This is a token-level envelope, not a measure of the sparse-outlier property that defines MAs (entries exceeding typical activations by orders of magnitude and sparse across features). A pre-attention peak in m could arise if the entire hidden vector of the sink token is uniformly scaled up before a full attention layer, with no single outlier feature. The fixed-coordinate verification in Section 4.1 and Appendix E.1 traces one coordinate in one 1.3B GDN model, and the cross-model analyses in Appendix E.2 again report layerwise dominant-feature magnitudes rather than sparsity or outlierness. The paper should add a sparsity/outlierness measure (e.g., fraction of features exceeding a threshold, max-to-median ratio, top-k energy share) applied to the same suite and show that PAS and ISP are carried by a small number of outlier features. Without this, the claims are about activation magnitude spikes, not yet about massive activations as defined in the paper.","section":"Section 3.3, Eq. (6); Section 3.4, Eq. (7)"},{"comment":"All quantitative results (Align, ISR, and all trajectory plots) trace the single consensus sink token selected by the argmax in Eq. (5). The controls in Appendix B.2 and B.5 show that non-sink tokens behave differently and that the first token is a common sink, but they do not provide metrics-level evidence that the PAS–ISP organization is insensitive to the specific choice among the top-k consensus sinks. I ask for Align and ISR recomputed with an average over the top-k consensus sinks and with an 'any of the top-k sinks' variant. If the authors believe the existing controls are sufficient, the text should state explicitly why a single argmax anchor cannot distort the measured layerwise morphology.","section":"Section 3.2, Eq. (5)"},{"comment":"The claim that increasing full attention density 'recovers' the stable MA morphology of full attention LLMs is supported visually by trajectory plots and by the monotonic increase of ISR, but no quantitative similarity measure compares the layerwise HLA profiles at each hybridization ratio with the full attention baseline profile. Adding a simple quantitative comparison (e.g., Pearson correlation or normalized L2 distance between the first-token magnitude profiles, evaluated across inputs and domains) would substantiate the continuum claim that PAS, ISP, and full attention MAs form a single progression.","section":"Section 3.4; Section 4.2"}],"minor_comments":[{"comment":"Bootstrap confidence intervals are reported in Appendix B.5 but not in the main tables; adding CI columns or a compact uncertainty summary would make the main quantitative claims easier to assess.","section":"Tables 2 and 7"},{"comment":"The abstract states that code is available at a repository, while the Reproducibility Statement says the authors 'will release' the analysis code; please clarify the actual availability status.","section":"Abstract and Section 7"},{"comment":"The two panels of Figure 2 use bubble size and color in ways that are not defined in the caption; a short legend or explicit description of the bubble encoding would improve readability.","section":"Figure 2"},{"comment":"The text says 'attention distribution determines only which token is tracked; whether and how strongly that token exhibits an MA remains determined by its activation magnitude.' This is a useful clarification, but it highlights exactly the gap addressed by Major Comment 1: the measured magnitude is not tested for the sparse-outlier property.","section":"Section 3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical study with a credible descriptive finding about activation magnitude organization in HLA LLMs. The main risk is definitional: the measured quantity is a token-level max-absolute activation, not the sparse-outlier statistic used to define MAs. This is addressable with additional metrics and should not require new model training, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline: this paper finds a real, reproducible morphological regularity in hybrid linear attention LLMs—per-token max activation peaks right before full attention layers and plateaus between them as full attention density increases—but its core metrics measure token-level max, not sparsity, so the claim that these are 'massive activations' in the established sense is not yet supported.\n\nWhat's new: first systematic mapping of MA-like behavior in layer-interleaved hybrids, showing that full attention placement organizes the layerwise profile and that the full attention limit recovers the familiar stable morphology. The empirical sweep is serious: five linear attention backbones, two scales, six hybrid configurations, twelve large pretrained checkpoints, five domains, plus controlled pretraining with gating interventions. The Align and ISR metrics are clearly defined, with bootstrap CIs and token/layer controls; the qualitative figures alone are convincing.\n\nThe soft spot is the one the stress-test flags. m^(l) is the infinity norm over features of a single token, so Align and ISR register any pre-attention elevation of the whole hidden vector as a spike. That is not the same as a massive activation, which is defined as a sparse outlier exceeding typical activations by orders of magnitude. The paper never directly measures sparsity across models. The fixed-coordinate write-sink-cancel analysis in Appendix E.1 is exactly one token–feature coordinate in one 1.3B GDN model, and the cross-model appendix traces layerwise dominant-feature magnitudes, which again exists even for a uniform vector. So the central quantitative evidence does not distinguish 'MA spike' from 'token-state scaling event.' This is load-bearing because the paper's title and framing are specifically about MAs.\n\nI want to be clear about what does hold up. The layerwise organization of per-token max activation—its alignment with full attention placement and its density-dependent transition to the full attention plateau—is robust across the evaluated models and inputs. If the authors reframe the claim as being about the envelope of per-token activation, the descriptive result stands. To make the MA claim stick, they need to add a sparsity measurement, like the ratio of the max to a high quantile of the activation distribution at PAS layers, across many models. They should also verify the write-sink-cancel pattern on a few more coordinates and models; right now it is one anecdote.\n\nMinor: the reproducibility statement says 'will release' code, so no immediate way to check numbers. That is not a big deal for a preprint, but the repo should be live for the camera-ready.\n\nWho benefits: anyone working on hybrid architecture design, attention placement, or quantization of hybrid models. It deserves a serious referee; the gap is fixable without new training runs. My recommendation: send to peer review, with a clear request for sparsity-aware validation.","headline":"A solid empirical mapping of per-token activation peaks in hybrid linear attention LLMs, but the 'massive activation' label is not yet backed by sparsity evidence.","tokens_in":42117,"tokens_out":3643,"would_cite":true,"duration_ms":32875,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that in hybrid linear-attention LLMs, massive activations are organized by full-attention placement: spikes form right before full-attention layers and merge into plateaus as attention density grows, recovering the…","keywords":["massive activations","hybrid linear attention","pre-attention spikes","inter-spike plateaus","attention sinks","systematic outliers","output gating","transformer interpretability"],"falsifier":"Compute the sink-spike alignment metric in a hybrid model after masking or removing the first token so the first position is no longer a candidate attention sink. If the pre-attention spike disappears or scatters to random layers, the PAS pattern is an artifact of the chosen single-token anchor; if it stays at the same pre-full-attention layer carried by another token, the architecture-aligned claim is supported.","tokens_in":41030,"feed_emoji":"🧠","tokens_out":6241,"duration_ms":52845,"temperature":0.7,"pith_summary":"This paper tries to establish that the internal outlier structure of hybrid linear-attention LLMs is governed by where the full-attention layers sit. It reports that massive activations concentrate immediately before full-attention layers, forming pre-attention spikes, and that as full-attention layers become denser these spikes extend through the intervening linear-attention layers as inter-spike plateaus, eventually recovering the stable massive-activation morphology of pure full-attention transformers. A sympathetic reader would care because it turns an apparently chaotic numerical phenomenon into a predictable, architecture-level signature that could inform model design, quantization, and interpretability. The paper supports the claim with a controlled model suite, open-source models across scales, and training-time tracing experiments.","feed_headline":"Hybrid LLM spikes cluster before full-attention layers","feed_subtitle":"As attention gets denser the spikes merge into plateaus, so layer placement, not input, sets the outlier map.","key_machinery":"The central instrument is attention-sink-guided tracing: for each input the paper computes a consensus sink token from attention probabilities averaged over full-attention layers and heads, then follows that token's maximum absolute hidden-state entry across depth. Two derived metrics, the sink-spike alignment rate and the inter-spike retention score, convert the trajectories into quantitative claims. The explanatory machinery is a cancellation-timing lifecycle: a pre-attention layer writes a large signed outlier, the token acts as an attention sink during full attention, and a prompt opposite-signed update cancels the outlier to form a sharp spike; delayed cancellation leaves the outlier elevated through the intervening layers, forming a plateau.","core_discovery":"Massive activations in hybrid linear-attention LLMs are not scattered outliers but form a layerwise pattern set by full-attention placement. Specifically, the maximum absolute hidden-state activation of a consensus attention-sink token spikes immediately before every full-attention layer, giving pre-attention spikes; as full attention becomes denser, these spikes persist through the intervening linear-attention layers as inter-spike plateaus; and at the pure full-attention limit the two morphologies merge into the stable, plateau-like MA profile already known in full-attention transformers. The paper reports this recurrence across five linear attention backbones, six hybridization configurations, five input domains, and open-source hybrid models from 1.2B to 397B parameters. Controlled pretraining of gated-delta-rule hybrids shows that both morphologies emerge early during training, that gating the full-attention output strongly attenuates their magnitudes without deleting their layerwise organization, and that removing the linear mixer's own output gates gives only a modest amplification. Mechanistically, the paper attributes the pattern to a shared write-sink-cancel lifecycle whose outcome depends on cancellation timing: prompt cancellation localizes a spike, delayed cancellation sustains a plateau, and the full-attention limit is the endpoint of progressively deferred cancellation.","pith_inferences":["If the paper is right, one could deliberately choose the full-attention schedule to place massive-activation spikes where a downstream quantizer or pruner can handle them, or where they best support attention-sink behavior.","A testable extension would be to perturb residual-stream scaling or gating to shorten or lengthen cancellation delay and see whether spikes turn into plateaus and back without moving their layerwise position; that would confirm cancellation timing as the causal knob.","The single-sink tracing result invites a check under long-context or streaming inference, where the fixed-size recurrent state may change sink behavior; the paper does not claim its morphology persists there.","One could also test whether spike positions predict task-relevant behavior by ablating the pre-attention spike at a specific layer and measuring changes in retrieval, reasoning, or next-token accuracy."],"forward_implications":["Massive-activation positions can be read off the layer schedule: every full-attention layer's predecessor is a spike site, regardless of input domain or linear-attention backbone.","The spike-to-plateau transition is monotone in full-attention density: the inter-spike retention score rises from sparse to dense hybrids across every architecture-scale pair evaluated.","Output gating of full attention suppresses the size of spikes and plateaus but does not remove their organization, while deleting the linear mixer's gates has only a modest effect, so full attention is the primary organizer of MA dynamics.","PAS and ISP are learned structures, not checkpoint artifacts: they appear early in controlled pretraining and consolidate as training proceeds.","The full-attention MA morphology is the same write-sink-cancel lifecycle seen at the long-delay endpoint, so hybrid and full-attention massive activations form one continuum of outlier persistence."],"supporting_citations":[{"why":"Defines massive activations and documents the stable layerwise MA morphology of full-attention LLMs that the hybrid morphology is said to recover at the limit.","marker":"Sun et al., 2024"},{"why":"Establishes the systematic-outlier framework whose signed write-cancel decomposition is used to explain PAS and ISP.","marker":"An et al., 2025"},{"why":"Supplies the controlled multi-architecture hybrid suite used for the inference-time comparisons across backbones and hybridization ratios.","marker":"Wang et al., 2025"},{"why":"Provides the training implementation for the controlled pretraining runs that trace the emergence of PAS and ISP.","marker":"Yang & Zhang, 2024"},{"why":"Source of the gated-attention output-gating intervention used to test attenuation of PAS and ISP.","marker":"Qiu et al., 2026"},{"why":"Defines the gated-delta-rule linear attention architecture and the evaluation recipe used in the controlled pretraining experiments.","marker":"Yang et al., 2025b"},{"why":"Anchors the connection between massive activations and attention sinks that motivates tracing the consensus sink token across layers.","marker":"Sun et al., 2026"}],"fun_headline_variants":["Attention placement dictates massive activation spikes","Pre-attention spikes and plateaus reveal hybrid LLM structure","Layerwise spikes, not noise, mark hybrid LLM activations","Attention-layer position sets massive activation pattern"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a single consensus sink token, usually the first token, can stand in for all massive activations when tracing their layerwise organization; if that anchor is not representative, the pre-attention-spike and inter-spike-plateau pattern may be an artifact of where the analysis looks.","fun_headline_variants_meta":{"raw":{"variants":["Attention placement dictates massive activation spikes","Pre-attention spikes and plateaus reveal hybrid LLM structure","Layerwise spikes, not noise, mark hybrid LLM activations","Attention-layer position sets massive activation pattern"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000233,"raw_usage":{"total_tokens":1553,"prompt_tokens":1068,"completion_tokens":485,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":684,"completion_tokens_details":{"reasoning_tokens":424}},"tokens_in":684,"tokens_out":485,"duration_ms":4291,"temperature":1.0,"reasoning_tokens":424,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:14:30.712190+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the sink-spike alignment metric in a hybrid model after masking or removing the first token so the first position is no longer a candidate attention sink. If the pre-attention spike disappears or scatters to random layers, the PAS pattern is an artifact of the chosen single-token anchor; if it stays at the same pre-full-attention layer carried by another token, the architecture-aligned claim is supported.","supporting_citations":[{"cited_title":"Summer is warm. Winter is cold","cited_arxiv_id":null,"evidence_quote":"Provides the training implementation for the controlled pretraining runs that trace the emergence of PAS and ISP."}],"review_version":1}