{"id":"a1e5d23f-f92f-46e0-a3a0-0f9a61299887","arxiv_id":"2501.00658","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"State space models have an inherent recency bias and over-smoothing in deep stacks, and a two-channel polarization fix improves long-range associative recall.","lead":"This paper shows that state space models like Mamba strongly favor nearby tokens over distant ones, which limits their long-context memory, and that stacking many layers makes their internal representations blur together. The authors propose a simple modification that pins two memory channels to constant values, improving recall of distant information and making deeper models more useful.","discovery_kind":"first_principles","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 4.2 is unproven as written: Lemma D.6's pointwise bound is false, so the 'inevitable over-smoothing' claim lacks the stated rigorous support.","rationale":"The paper identifies two limitations of SSMs—recency bias and over-smoothing—and proposes polarization to address both. The recency theorem is an upper bound on influence and, by itself, cannot prove a strong recency bias; the paper's empirical influential-score curves provide some support, but the theorem alone is not a sufficient formal basis for 'inherently limited by strong recency bias.' The deeper problem is the over-smoothing theorem: its proof relies on a false pointwise lemma, so the theoretical foundation for the 'inevitable over-smoothing' claim is invalid. The empirical sharpness curves and associative-recall results are useful evidence, and the polarization idea may still have merit, but the manuscript as written does not rigorously establish one of its two headline contributions. The reader's verdict was CONDITIONAL based on unverified assumptions; the additional finding that the central over-smoothing proof is invalid moves the verdict to REJECT for the current version, since a main theorem is not proven and the claim of inevitability is unsupported.","tokens_in":31851,"tokens_out":17157,"duration_ms":166023,"concrete_test":"Run the scalar recurrence with T=2, α1=α2=0.5, Δ1=Δ2=0.5, z1=-1, z2=100, s0=0. Compute s1=-0.5 and s2=49.75; Lemma D.6's upper bound for s2 is 49.5, so the lemma fails under Theorem 4.2's assumptions. If confirmed, the proof of inequality (4) must be revised before the over-smoothing theorem can be cited as support for the paper's central claim.","verdict_should_be":"REJECT","load_bearing_attack":"The over-smoothing conclusion rests on Theorem 4.2, whose proof depends on Lemma D.6. Lemma D.6 claims pointwise bounds for the scalar recurrence s_t = α_t s_{t-1} + Δ_t z_t. The lemma is false even under Theorem 4.2's assumptions. Take T=2, α1=α2=0.5, Δ1=Δ2=0.5, z1=-1, z2=100, and s0=0. This satisfies (At)_nn + Δ_t = 1 and the centered condition m=-1 ≤ 0 ≤ M=100. The lemma's stated upper bound for s2 is A_min·p·m + (1 − A_min·p)·M with p=1, yielding 49.5. But the recurrence gives s1=-0.5 and s2 = 0.5·(−0.5) + 0.5·100 = 49.75, which exceeds the claimed bound. Therefore the induction step used to prove inequality (4) is invalid, and the proof of Theorem 4.2 does not go through. In addition, the paper asserts that Mamba 'can easily achieve' (At)_nn + Δ_t ≤ 1 and centered inputs, but Mamba's Δ_t = softplus(W_Δ x_t) is not bounded above by 1, so this premise is also unverified. The theorem may be repairable with additional conditions, but as stated the central claim that SSMs inevitably over-smooth with depth is not rigorously established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies two purported bottlenecks of structured state space models (SSMs): recency bias and over-smoothing. Theorem 3.1 gives an exponential upper bound on the influence score |∂y_t/∂x_s| for diagonal state transition matrices with entries in (0,1), which the authors interpret as a rigorous demonstration that SSMs are recency-biased. Supporting experiments include a needle-in-a-haystack evaluation, adversarial attacks on CIFAR-10, and influence-score measurements. Scaling experiments in Section 4.1 show that deeper Mamba models first help and then hurt with increasing depth; the paper attributes this to over-smoothing. Theorem 4.2 claims a bound on the spread of hidden states that shrinks with the context length, formalizing the over-smoothing tendency. The authors then propose \"polarization\": forcing one channel of A_t to 1 and one channel to 0, and report improved associative-recall accuracy, especially in deeper models. The paper also discusses HiPPO theory, selection mechanisms, complex parameterizations, and related work on GNN and transformer over-smoothing.","tokens_in":32142,"tokens_out":7743,"duration_ms":74177,"significance":"If both theorems were sound, the paper would give a principled account of why SSMs underperform on long-range retrieval and how depth exacerbates representation collapse, together with a simple, practical mitigation. The work is also valuable empirically: it ships code, carefully documents the training settings, and provides a rich set of observations (influence-score decay, inverted needle retrieval curves, trailing-token vulnerabilities, and associative-recall gains from polarization) that are of independent interest. The connection to HiPPO and to prior over-smoothing literature is thoughtful. However, the two theoretical pillars are not on equal footing: the recency result is a valid upper bound but is over-interpreted as a proof of actual decay, and the over-smoothing theorem's proof contains a false lemma. Because the polarization method is justified through these theorems, the theoretical underpinning of the proposed mitigation is weakened, although the empirical results in Table 2 remain suggestive.","major_comments":[{"comment":"The abstract and Section 3.1 claim that the paper \"rigorously demonstrates\" a strong recency bias, but Theorem 3.1 is only an upper bound: |∂y_t/∂x_s| = O(exp(-κ(t-s))). An upper bound does not prove that influence actually decays; it only states that distant tokens cannot have arbitrarily large influence. A model in which all tokens have uniformly small influence also satisfies the bound. The empirical evidence in Figure 1 and the needle-in-a-haystack results do show actual decay, but the theorem itself does not. The authors should either prove a matching lower bound under suitable non-degeneracy assumptions (for instance, lower-bounding |∂(Δ_s b_s(x_s))/∂x_s| away from zero along some direction) or change the wording from \"rigorously demonstrate strong recency bias\" to a claim about an at-most-exponential decay of influence.","section":"Section 3.1, Theorem 3.1"},{"comment":"Lemma D.6, which is the key step in the proof of Theorem 4.2, is false as stated. Counterexample: take T=2, α_1=α_2=0.5, Δ_1=Δ_2=0.5, z_1=-1, z_2=100, and s_0=0. These values satisfy both assumption (i), since α_t+Δ_t=1, and assumption (ii), since min z=-1≤0≤100=max z. With m=-1, M=100, and p=1 (because z_1=m), the claimed upper bound for s_2 is A_min·p·m + (1-A_min·p)·M = 0.5·1·(-1) + (1-0.5·1)·100 = 49.5. The recurrence gives s_1 = -0.5 and s_2 = 0.5·(-0.5)+0.5·100 = 49.75, which exceeds the claimed bound. Moreover, the base case for t=1 already fails: the lemma's upper bound is z_1 = -1, while s_1 = -0.5. Consequently the induction in the proof of Lemma D.6 is invalid, and inequality (4) in Theorem 4.2 is not established. The over-smoothing conclusion in Section 4.2 therefore needs a corrected lemma with different hypotheses (for example, a separate treatment of the first time step) or a different proof strategy.","section":"Appendix D.2.2, Lemma D.6 and Theorems D.5/4.2"},{"comment":"The paper asserts that conditions (i) (A_t)_n,n + Δ_t ≤ 1 and (ii) centered b_t inputs are \"easily achieved\" by prevalent SSMs, including Mamba. This is not verified for Mamba. In Mamba, Δ_t = softplus(W_Δ x_t) is unbounded above, and (A_t)_n,n = exp(Δ_t A_n,n) with A_n,n<0. For any Δ_t>1, the sum exp(Δ_t A_n,n)+Δ_t can exceed 1; for instance, with A_n,n=-1 and Δ_t=2, exp(-2)+2 > 1. The centered-input condition min_t b_t(x_t)_n ≤ 0 ≤ max_t b_t(x_t)_n is also a property of the learned matrix W_B and the input distribution, not a design guarantee. If these conditions are not checked for the actual models in Figure 5, Theorem 4.2 does not connect to the empirical Mamba models, and the claim that Mamba \"inevitably\" over-smooths with depth remains unsupported by the theory. The authors should either verify these conditions empirically on the trained models or explicitly frame Theorem 4.2 as a conditional statement whose assumptions may or may not hold in practice.","section":"Section 4.2, applicability to Mamba (Eq. (3))"}],"minor_comments":[{"comment":"The sentence \"We also study two variants: 0-polarized and 1-polarized Mamba by only pretending or appending one or zero to At\" contains a typo: \"pretending\" should presumably be \"prepending\".","section":"Section E.5.1"},{"comment":"In the definition of E(x), the normalization denominator and the index N appear to refer to the number of tokens, which is T in the notation of the paper; please use a consistent symbol to avoid confusion.","section":"Section 4.2, sharpness measure"},{"comment":"In the proof of Theorem D.2, the bound on ∂u_t/∂x_s extracts a uniform constant C_1 that depends on A^{-1}_r and ∂A_r/∂x_s, but the subsequent geometric-series step is valid only if the same bound applies uniformly over all r; this is true under the compactness assumption but could be stated more explicitly.","section":"Appendix D.1, Eq. (7)"},{"comment":"The formulation (LA) sets A_t = I, which lies on the boundary (A_t)_n,n=1 rather than in (0,1); Theorem 3.1 requires strict inequality. The paper should clarify whether the theorem extends to this boundary case or whether linear attention requires a separate argument.","section":"Appendix B, linear attention (LA)"}],"recommendation":"major_revision","confidential_remarks":"The empirical parts of this paper—the needle-in-a-haystack retrieval curves, the robustness experiments, the depth-scaling observations, and the polarization results—are interesting and likely salvageable. The main obstacle is theoretical: Lemma D.6 is demonstrably false, so Theorem 4.2 as stated does not provide the claimed rigorous support for the over-smoothing narrative, and the Mamba applicability assumption is not checked. The recency theorem is valid but is over-sold in the abstract. I would recommend asking the authors to repair or reframe Theorem 4.2, to be explicit about the scope of Theorem 3.1, and to verify the assumptions on actual Mamba models before reconsidering the manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the paper extends recency-bias analysis to selective/input-dependent SSMs (Mamba) and proposes a simple two-channel polarization trick that produces large gains on associative recall. Second, the over-smoothing theorem is not proven: its Lemma D.6 is false, and the authors' claim that Mamba easily satisfies the theorem's assumptions is unverified.\n\nThe genuinely new piece is Theorem 3.1, which gives an exponential upper bound on the influence score for SSMs with input-dependent transition matrices. That is a real extension of Wang and Xue's result for linear SSMs. The paper also includes careful empirical work: influence-score plots, needle-in-a-haystack tests on a 7B Mamba, depth-scaling pretraining experiments, and associative recall. The polarization method is simple and its empirical gains are substantial (e.g., average recall jumps from 72 to 93 on 256 key-value pairs). Code is released. Credit where due: this is a useful empirical paper with a plausible diagnosis.\n\nThe soft spots, in order. Theorem 3.1 is an upper bound on |∂y_t/∂x_s|; it shows influence cannot decay slower than some rate, but it does not prove it actually decays at that rate. The abstract's phrase \"rigorously demonstrate strong recency bias\" is stronger than the math. More seriously, Theorem 4.2's proof relies on Lemma D.6, and that lemma is false. Take α1=α2=0.5, Δ1=Δ2=0.5, z1=-1, z2=100. The lemma's upper bound for s2 is 49.5, but the recurrence gives 49.75. The base case of the induction ignores Δ1. So the \"inevitable over-smoothing\" claim lacks rigorous support. The theorem may be repairable, but as written it does not stand. Also, the assumption (At)_nn + Δ_t ≤ 1 is asserted but not checked for Mamba; Δ_t is softplus, which is unbounded, so that premise is not obviously satisfied.\n\nThe security section extrapolates from CIFAR-10 pixel attacks to LLM jailbreaks with no direct experiments; I would treat that as speculation.\n\nWho should read this? Anyone working on SSM theory or architecture design. The empirical findings and the polarization trick are valuable even if the theory needs work. For a conference referee, I would send it to review but with major revision: fix or replace Lemma D.6, verify (or drop) the Mamba assumption, and soften the claims. The core diagnosis is probably right, but the paper's strongest language currently exceeds its proof.","headline":"The recency-bias result genuinely extends to selective SSMs and the polarization trick works empirically, but the over-smoothing theorem's proof relies on a false lemma, so the strong claims outrun the current math.","tokens_in":32701,"tokens_out":5861,"would_cite":true,"duration_ms":51975,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"State space models are inherently recency-biased, and depth scaling is blocked by over-smoothing; polarizing two transition channels to 0 and 1 mitigates both.","keywords":["state space models","recency bias","over-smoothing","long-range dependencies","associative recall","Mamba","linear attention","polarization"],"falsifier":"Measure, on a trained Mamba, the empirical influential scores $\\log|\\partial y_t/\\partial x_s|$ and check whether they follow the predicted slope $\\log(A_{\\max}^{-1})$; separately, record whether any channel has $(A_t)_{n,n}+\\Delta_t > 1$. If the score does not decay exponentially with distance, or if over-smoothing fails to appear when many channels violate non-expansiveness, the central claims are wrong.","tokens_in":31634,"feed_emoji":"🧠","tokens_out":11861,"duration_ms":105502,"temperature":0.7,"pith_summary":"Structured state space models (SSMs) are often advertised as good at long sequences, but this paper argues they are intrinsically biased toward recent tokens. For any SSM whose state transition matrix is diagonal with entries in $(0,1)$, the influence of an input token on a later output decays exponentially with distance; Mamba and many linear-attention models fall in this class, so their long-range recall is limited regardless of selection mechanisms. The paper also proves that stacking SSM layers smooths token representations, with pairwise differences in memory states bounded by a factor smaller than one, which explains why depth scaling eventually stops helping. To address both limits at once, polarize two transition channels: one fixed at $1$ to preserve history and one fixed at $0$ to keep a fresh token channel, mitigating recency bias and over-smoothing simultaneously. Associative-recall experiments show the fix raises accuracy for distant key-value pairs and lets deeper models help instead of hurt.","feed_headline":"SSMs forget distant tokens exponentially; two pinned channels fix it","feed_subtitle":"The paper proves SSM recall decays exponentially; pinning two transition channels to 0 and 1 helps deep models.","key_machinery":"The workhorse is the closed-form parallel expansion of the recurrence: $h_t = \\sum_{s<t} (\\prod_{r=s+1}^{t} A_r)\\Delta_s b_s(x_s) + \\Delta_t b_t(x_t)$, which rewrites each SSM as a weighted sum over past tokens. When $A_t$ is diagonal with entries in $(0,1)$, the cumulative product decays as $A_{\\max}^{t-s}$, producing the exponential influence bound of Theorem 3.1; the same recurrence under $A_t+\\Delta_t \\le 1$ produces the over-smoothing bound of Theorem 4.2. The proposed polarization mechanism places one diagonal entry at the constant value $1$ (an infinite-memory channel) and another at $0$ (a fresh-token-only channel), forcing the spectrum of $A_t$ to span both extremes and thereby avoiding both forgetting and over-mixing. In continuous-time S4, the same analysis identifies the layer as a low-pass filter, which is why repeated application flattens token differences.","core_discovery":"On the paper's own terms, the central result is Theorem 3.1: an SSM of the form $h_t = A_t h_{t-1} + \\Delta_t b_t(x_t)$, $y_t = c_t(h_t)$, with diagonal $A_t \\in (0,1)^{N\\times N}$, has influential score $|\\partial y_t/\\partial x_s| = O(\\exp(-\\kappa(t-s)))$ with $\\kappa = \\Theta(\\log(A_{\\max}^{-1}))$, so distant inputs are exponentially under-weighted. Theorem 4.2 then shows that under a non-expansiveness condition and centered inputs, $\\max_{t,s}\\|h_t - h_s\\|_\\infty \\le (1 - A_{\\min}^{T-1}) \\max_{t,s}\\|b_t(x_t) - b_s(x_s)\\|_\\infty$, so deeper SSMs collapse token representations toward each other. The paper's proposed fix is polarization: pin one diagonal channel of $A_t$ to $1$ and one to $0$, leaving the rest learnable, which simultaneously preserves complete history and prevents over-mixing. Empirically, on associative recall, this raises accuracy at 256 key-value pairs from 33.52\\% to 81.56\\% for a four-layer Mamba, and makes deeper models strictly beneficial.","pith_inferences":["The paper does not fit $\\kappa$ from data; one could measure whether trained slopes of $\\log|\\partial y_t/\\partial x_s|$ track $\\log(A_{\\max}^{-1})$ in practice.","The polarization trick suggests a general design rule for recurrent models: keep one integrator channel and one instantaneous readout channel even when other decay factors are learned.","A natural next test is whether the same two-channel polarization helps on natural-language long-context benchmarks, not just synthetic associative recall.","The theory implies that for robustness, end-of-context tokens should see stronger adversarial effects; this could be verified directly on instruction-following language models."],"forward_implications":["If Theorem 3.1 holds, any practical SSM with sub-unit diagonal transitions, including Mamba, RetNet, RWKV, and GLA, will show exponential loss of retrieval accuracy once the sequence exceeds its effective memory, no matter how the selection mechanism is tuned.","If Theorem 4.2 holds, depth scaling alone cannot fix long-context modeling: beyond a depth threshold, added layers make token representations more alike, so validation performance plateaus and then declines, as the paper's scaling curves show.","Polarizing two channels is a cheap architectural change that does not alter gradient flow in the remaining channels, so it should transfer to any diagonal-SSM variant that wants long-range recall without over-smoothing.","Recency bias is a robustness liability: because trailing tokens dominate outputs, an attacker can achieve high success by corrupting only the last few tokens of an input sequence, while leaving leading tokens intact.","Because the theorems cover input-dependent $A_t$ and $b_t$, selection mechanisms such as Mamba's do not exempt a model from recency or over-smoothing; content-aware gating changes the constants but not the exponential form."],"supporting_citations":[{"why":"Defines the Mamba selective SSM used throughout as the empirical testbed and source of the diagonal transition structure.","marker":"Gu & Dao (2023)"},{"why":"Introduces the S4 structured state space layer whose form motivates the general recurrence in Eq. 1.","marker":"Gu et al. (2021a)"},{"why":"Provides the associative recall benchmark and training recipe that the polarization experiments adopt.","marker":"Arora et al. (2023)"},{"why":"Shows linear attention can be written as a recurrence, the basis for extending the claims to linear-attention models.","marker":"Katharopoulos et al. (2020)"},{"why":"Introduces the H3 block used inside Mamba; the paper argues its gating and local convolution do not remove recency bias.","marker":"Fu et al. (2022)"},{"why":"Formulates RWKV, an SSM instance satisfying the conservative update used to relax assumptions of Theorem 4.2.","marker":"Peng et al. (2023)"},{"why":"Supplies the Mistral-7B transformer baseline in the needle-in-a-haystack retrieval comparison.","marker":"Jiang et al. (2023)"},{"why":"Source of the over-smoothing concept that Theorem 4.2 extends to SSMs.","marker":"Oono & Suzuki (2019)"}],"fun_headline_variants":["Polarize SSM channels to beat recency and over-smoothing","Two pinned channels fix SSM recency bias and over-smoothing","SSM recall decays exponentially; polarize to fix","Deep SSMs over-smooth; pin channels to 0 and 1","Recency vs over-smoothing: polarization wins"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that real SSM recurrence operators are diagonal with entries in $(0,1)$ and satisfy the non-expansive condition $A_t+\\Delta_t \\le 1$; if actual trained models violate either, the theorems' conclusions are not guaranteed.","fun_headline_variants_meta":{"raw":{"variants":["Polarize SSM channels to beat recency and over-smoothing","Two pinned channels fix SSM recency bias and over-smoothing","SSM recall decays exponentially; polarize to fix","Deep SSMs over-smooth; pin channels to 0 and 1","Recency vs over-smoothing: polarization wins"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000182,"raw_usage":{"total_tokens":1355,"prompt_tokens":1035,"completion_tokens":320,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":651,"completion_tokens_details":{"reasoning_tokens":230}},"tokens_in":651,"tokens_out":320,"duration_ms":3453,"temperature":1.0,"reasoning_tokens":230,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:45:04.341695+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure, on a trained Mamba, the empirical influential scores $\\log|\\partial y_t/\\partial x_s|$ and check whether they follow the predicted slope $\\log(A_{\\max}^{-1})$; separately, record whether any channel has $(A_t)_{n,n}+\\Delta_t > 1$. If the score does not decay exponentially with distance, or if over-smoothing fails to appear when many channels violate non-expansiveness, the central claims are wrong.","supporting_citations":[{"cited_title":"0 A −1000 # , At ≈","cited_arxiv_id":null,"evidence_quote":"Defines the Mamba selective SSM used throughout as the empirical testbed and source of the diagonal transition structure."}],"review_version":1}