{"id":"6af742b4-a095-4c48-875c-508c386d9162","arxiv_id":"2505.09022","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"B2S6, a block-biased variant of Mamba's S6 unit, restores universal approximation, softens input-dependent forgetting, and beats S4/S4D on Long-Range Arena while keeping Mamba-like language perplexity.","lead":"This paper proposes B2S6, a modified Mamba state space unit that splits inputs into blocks and adds a per-channel bias, and reports it outperforms S4 and S4D on Long-Range Arena while matching Mamba on language modeling. The authors also give proofs for why plain Mamba's shared weights, input-dependent forgetting, and sampling-interval parameters hurt long-range tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The LRA 'resurrection' claim is confounded: the S6 row is the causal unidirectional Mamba, while B2S6 is trained bidirectionally, so the gap may come from protocol, not block/bias.","rationale":"I identified the same load-bearing concern as the reader. The theoretical sections are carefully scoped: Theorem 1 restricts to fixed Δ and a single layer and explicitly warns against over-reading; Theorems 4 and 5 have proofs that, within their stated scopes, are consistent with the S4D reduction; and the stability result's w=0 assumption is disclosed, though it limits applicability to the input-dependent selection mechanism. However, these theorems establish capacity and bias properties of the unit in isolation; they do not establish that B2S6, rather than bidirectionality, normalization, complex parameterization, or scheduling, causes the LRA improvement. Since Appendix F omits the S6 baseline configuration and Appendix A reveals the bidirectional choice, the central empirical claim is not yet supported. The proposed concrete test—a controlled S6 run in the exact B2S6 pipeline—would settle this cleanly. I would not change the CONDITIONAL verdict; the concern is addressable and does not require rejection.","tokens_in":30200,"tokens_out":8164,"duration_ms":82420,"concrete_test":"Retrain the standard S6 unit inside the exact B2S6 LRA pipeline of Appendix F—same bidirectional wrapper, normalization, complex parameterization, reduced Δ learning rate, batch size, epochs, and seeds—replacing only the block partition and B_bias with the vanilla S6 recurrence; equivalently, run Table 4 with h=1, B_bias=0. If this controlled baseline averages above roughly 85 on Table 2's six tasks, the claimed 'resurrection' is mostly attributable to bidirectionality and training protocol, not to the proposed unit.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim (Section 1, 'B2S6 resurrects Mamba from failure') rests on Table 2, where S6 (Mamba) averages 66.59 and B2S6 averages 87.71. The weak point is a training-protocol confound. Appendix A states the authors also adopt a bidirectional structure in training the LRA tasks, while Appendix F (Table 4) gives full configurations for B2S6 and none for the S6 baseline; the S6 row matches the standard causal, unidirectional Mamba LRA numbers reported in bidirectional-SSM follow-ups such as HYDRA. B2S6 differs from that baseline not only by block partition and channel bias, but also by bidirectionality, normalization choices, complex-diagonal parameterization, and a reduced learning rate on Δ-related parameters. Since bidirectionality alone is known to add several LRA points to SSMs, any part of the 21-point gap could come from these protocol changes rather than from the proposed block-plus-bias mechanism. Consequently, the paper's attribution of the improvement and its 'Mamba fails on LRA' narrative are not established. This is addressable by publishing or controlling the S6 baseline; it is not an internal contradiction.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper analyzes why Mamba's S6 unit underperforms on Long-Range Arena (LRA) and proposes B2S6, a modification that partitions the input channels into blocks and adds a channel-specific bias term. The theoretical part claims three limitations of S6: single-layer S6 is not a universal approximator (Theorem 1), S6 imposes an exponentially input-magnitude-dependent sensitivity bias (Theorem 2), and S6 is less stable to train than S4D (Theorem 3). The paper then proves that B2S6 regains universality (Theorem 4) and exhibits a milder polynomial sensitivity (Theorem 5). Empirically, Table 2 reports an average LRA accuracy of 87.71 for B2S6 versus 66.59 for S6 (Mamba), and Appendix G reports that B2S6 roughly matches Mamba's perplexity on a SlimPajama subset.","tokens_in":30506,"tokens_out":8337,"duration_ms":84923,"significance":"If the empirical comparison were properly controlled, this paper would be a useful contribution: it identifies a concrete expressiveness bottleneck in S6, proposes a simple architectural repair, and backs the results with detailed proofs. The strengths are the self-contained Lemma 1 construction for the non-universality result, the explicit proof appendices for Theorems 1-5, the reproducible B2S6 configuration table in Appendix F, and the synthetic experiments that directly illustrate the theoretical statements. However, the headline LRA claim is currently confounded by protocol differences, and Theorem 3 explicitly fixes w=0, so the paper's central explanation of Mamba's LRA failure is not fully established. These issues substantially limit the significance of the present version.","major_comments":[{"comment":"The central empirical claim that B2S6 \"resurrects Mamba from failure\" on LRA is not established because the S6 baseline is not compared under a matched protocol. Appendix A states that the authors adopt a bidirectional structure in training the LRA tasks, and Appendix F (Table 4) provides full configurations for B2S6 but no configuration for the S6 row. B2S6 also differs from the published S6 model by using complex-valued parameters, normalization choices, and a reduced learning rate for Delta-related parameters. Since bidirectionality and these training choices are known to affect LRA accuracy substantially, the 21-point gap between S6 (66.59) and B2S6 (87.71) cannot be attributed to the proposed block partition and channel bias alone. The authors should either train a same-protocol S6 baseline, or ablate bidirectionality, complex parameterization, and the reduced learning rate to isolate the effect of the proposed mechanism.","section":"Section 7, Table 2; Appendix A; Appendix F"},{"comment":"Theorem 3 fixes w=0, which makes the sampling interval Delta_k = softplus(b) input-independent. The paper itself acknowledges this at the end of Section 5: \"We also fixed w = 0, leaving the case when w≠0 for future work.\" Consequently, Theorem 3 does not analyze the input-dependent selection mechanism that abstract and Section 1 identify as the source of instability. The statement \"this instability comes from the input-dependent selection mechanism\" is therefore not supported by the theorem. The theorem should be reframed as a statement about S6's shared B/C structure under a fixed sampling interval, or extended to the selective w≠0 case before the paper can claim to explain the role of selectivity in training instability.","section":"Section 5, Theorem 3"},{"comment":"The paper claims in Section 1 and Section 7 that B2S6 achieves \"state-of-the-art performance\" on LRA, but the extended Table 3 directly contradicts this: HOPE-SSM (87.85) and S4D-FT (88.26) have higher average accuracies, and RTF SSM ties B2S6 at 87.71. The claim should be revised to \"competitive with published SSMs\" or restricted to selective SSMs, for which the claim may still hold. This is a factual inconsistency in a stated contribution and should be corrected.","section":"Section 7, Table 3"},{"comment":"The paper interprets the relative gradient sensitivity S_k as a measure of inductive bias and memory retention, but S_k as defined in Eq. (6) measures the sensitivity of the final output to perturbations of each input, not whether information is actually stored or discarded by the recurrence. The statements that S6 \"can cause the model to discard useful long-term information too quickly\" and that B2S6 \"preserves memory\" rely on this identification. The authors should either provide direct evidence connecting S_k to long-range memory behavior (e.g., probing or reconstruction tasks), or restrict the claims to gradient sensitivity rather than memory retention.","section":"Section 4, Eq. (6); Theorems 2 and 5"}],"minor_comments":[{"comment":"The sentence \"In the next section, show how this distinction limits the 'effective width' and expressiveness of an S6 unit\" is missing a grammatical subject; it should be \"we show how\".","section":"Section 2, last paragraph"},{"comment":"References [87] and [88] cite the same paper by Wang and Xue; one of them should be removed or the two citations should be distinguished by their specific claims.","section":"References [87] and [88]"},{"comment":"The caption uses the block size p and number of blocks h before these are defined in Section 6; a brief parenthetical definition in the caption would improve readability.","section":"Figure 1 caption"},{"comment":"The sentence \"the probabilities that a given input satisfies jw^T juk0 < 0 and jw^T juk0 > 0 are equal\" refers to random initialization of w; this should be stated explicitly in the main text rather than implied, since the probability statement is over initialization, not over inputs.","section":"Section 6, paragraph after Theorem 5"},{"comment":"The hyperparameter h is described as \"not carefully fine-tuned but rather picked randomly\"; reporting a small sensitivity study over h, or at least stating the range tested, would strengthen the empirical claims.","section":"Appendix F, Table 4"},{"comment":"The language-modeling experiment reports perplexity only up to 28,830 steps and does not provide final validation perplexity, model size per variant, or token count; a one-epoch preliminary result is acceptable, but the main text should describe it as preliminary with these limitations.","section":"Appendix G"}],"recommendation":"major_revision","confidential_remarks":"The S6 (Mamba) LRA numbers in Table 2 appear to match the standard unidirectional Mamba results reported in the literature. If this is the case, the comparison with the bidirectional B2S6 is not apples-to-apples. I would ask the authors to disclose the source of the S6 numbers and to provide a same-protocol baseline. Additionally, the 'state-of-the-art' claim in Section 7 is not consistent with the extended Table 3 and should be reconciled before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about arXiv:2505.09022. The theory is mostly solid and the B2S6 idea is worth a look, but the LRA comparison that drives the paper is confounded: the \"S6 (Mamba)\" row is the standard causal unidirectional model, and their B2S6 is trained bidirectionally. They admit as much in Appendix A. So the 21-point gap they attribute to block/bias is probably part protocol, part architecture.\n\nWhat is actually new: a clean non-UAT proof for single-layer S6 (Lemma 1's quadratic-encoder argument is neat), a UAT for the block-biased variant, and a gradient-growth analysis (Theorem 3) showing why S6 gets harder to optimize as inputs or sequence length grow. The toy experiments in Figures 3–5 corroborate the theory. The ablation showing that the bias term increases effective width is also informative. They do not oversell the block idea; they credit Mamba2.\n\nNow the soft spots. Theorem 3 fixes w=0, so it does not actually analyze the input-dependent selection that defines Mamba. The claimed training-stability benefit of B2S6 is asserted rather than proven. No code is shipped, which makes Table 2 harder to reproduce. But the main problem is the baseline. The original Mamba LRA numbers are causal; B2S6 uses bidirectional structure, BatchNorm, complex-diagonal A, and a reduced learning rate on delta-related parameters. Any of those could account for a meaningful slice of the 21 points. Add a bidirectional Mamba row and the \"resurrects Mamba\" narrative could shrink to \"bidirectionality helps a lot.\"\n\nStill, the theoretical results are interesting in isolation, and the language perplexity match is a useful sanity check. The architecture is simple and plausibly useful. The paper deserves a serious referee because the theory and design are substantive, but the authors need to control the baseline before we can trust the central empirical claim. I would send it to review with a request for a proper bidirectional S6 row and, ideally, code.\n\nIn short: read this if you work on selective SSMs or wonder why Mamba underperforms on non-linguistic long-range tasks. The theory may survive contact with a better baseline; the marketing claim won't.","headline":"Strong theory, promising architecture, but the headline empirical claim is unproven because the S6 baseline is causal while B2S6 is bidirectional.","tokens_in":30982,"tokens_out":1920,"would_cite":false,"duration_ms":19233,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Mamba's S6 unit fails on long-range tasks for three structural reasons, and the paper's B2S6 unit fixes all three, beating S4 and S4D on Long-Range Arena while matching Mamba's language-modeling performance.","keywords":["state space models","Mamba","selective state spaces","long-range sequence modeling","Long-Range Arena","universal approximation","training stability","inductive bias"],"falsifier":"Take the exact Mamba/S6 configuration with bidirectional processing, complex parameters, and the same learning-rate schedule used for B2S6, and run it on all six Long-Range Arena tasks; if a properly tuned bidirectional S6 reaches or passes the 87.71 average that B2S6 reports, the paper's core claim that S6 fails on these tasks is refuted. A second check is to train single-layer S6 and S4D models on a continuous target with growing width: if S6's error falls with width in the same way as S4D's, then the non-universality theorem's practical significance fails.","tokens_in":30001,"feed_emoji":"🧩","tokens_out":7513,"duration_ms":69333,"temperature":0.7,"pith_summary":"This paper argues that Mamba's weak performance on long-range sequence tasks is not accidental but follows from three structural defects in its S6 recurrent unit: the unit shares its input and output matrices across all channels, which caps its effective width and kills universal approximation; its input-dependent sampling interval produces an exponential sensitivity bias that lets large-magnitude inputs erase earlier context; and gradients with respect to the sampling-interval parameters grow faster than S4D's, making long-sequence training unstable. To fix this, the paper introduces B2S6, which partitions the input into blocks with independent selection weights and adds an input-independent, channel-specific bias term, and proves that B2S6 is a universal approximator and has only polynomial input-magnitude bias. On the Long-Range Arena benchmark, B2S6 reports an average accuracy of 87.71, above S4D's 84.89 and S4's 86.09, while the paper's S6 row averages 66.59. If these results hold, Mamba's long-range weakness is not inherent to selective recurrence; a modest architectural change can close most of the gap without giving up language-modeling perplexity.","feed_headline":"B2S6 lifts Mamba from 66 to 87 percent on long-range tasks","feed_subtitle":"Block-wise selection and a per-channel bias restore accuracy while keeping Mamba's language-modeling perplexity.","key_machinery":"The machinery is the S6 recurrence $x^{(i)}_k = A^{(i)}_k x^{(i)}_{k-1} + B^{(i)}_k u^{(i)}_k$, $y_k = C_k x_k$, with $A^{(i)}_k=\\exp(\\Delta^{(i)}_k A)$, $\\Delta^{(i)}_k=\\mathrm{softplus}(w^\\top u_k+b^{(i)})$, $B^{(i)}_k=A^{-1}(A^{(i)}_k-I)Bu_k$, and $C_k=u_k^\\top C$. Its expressiveness defect is that $B$ and $C$ are shared across channels, making the final output a quadratic function of the input that cannot distinguish certain pairs of sequences. B2S6 modifies this by splitting $u_k$ into $h$ blocks of size $p=d/h$ and giving each block its own $B_{\\mathrm{weight}}$, $B_{\\mathrm{bias}}$, and $C$, so each block sees only its subvector but the collection of blocks behaves like a wider unit; the bias term $B^{(i)}_{\\mathrm{bias}}$ is input-independent and channel-specific, providing additional effective width.","core_discovery":"The paper's central claim is that a single-layer S6 unit is not a universal approximator of continuous sequence-to-point functions, whereas a single-layer S4D unit is, and that this difference is caused by S6 sharing the matrices $B$ and $C$ across all channels. Beyond expressiveness, the paper proves that S6's input-dependent $\\Delta$ imposes an exponentially large relative-gradient bias when any input vector grows in magnitude, and that the gradients of the S6 output with respect to $\\Delta$ parameters are $\\Omega(c^3)$ times S4D's as input magnitude $c\\to\\infty$ and grow relative to S4D as sequence length $L$ increases. The proposed B2S6 restores universal approximation either with the block structure alone or with the channel-specific bias alone, softens the bias from exponential to polynomial rates, and, together with a reduced learning rate on the $\\Delta$ parameters, trains stably on long sequences. Empirically, B2S6 achieves an average Long-Range Arena accuracy of 87.71, making it, by the paper's account, the first selective SSM to reach state-of-the-art performance on this benchmark, while language-modeling perplexity on SlimPajama-6B stays close to S6's.","pith_inferences":[],"forward_implications":["Replacing only the recurrent unit in Mamba with B2S6 lifts average Long-Range Arena accuracy from 66.59 to 87.71, so the architecture outside the S6 unit is not the bottleneck.","Because a single S6 unit is not a universal approximator, any model built only from S6 units needs many layers or unshared channel parameters to fit continuous sequence functions; B2S6 removes that requirement at the unit level.","The reduced learning rate on the input-dependent sampling-interval parameters should be adopted in other long-sequence settings, since the gradient ratios in Theorem 3 grow with sequence length regardless of task.","On language modeling, B2S6's perplexity tracks S6 closely on SlimPajama-6B, so the long-range fix does not sacrifice the utility that makes Mamba attractive.","S6's exponential input-magnitude bias predicts that large-magnitude tokens can overwrite memory; B2S6's block split makes this failure unlikely because independent blocks with opposite selection signs cancel the extreme behavior.","The paper's comparison would be strengthened by a matched S6 baseline: the S6 row in Table 2 is reported without the configuration given for B2S6 in Appendix F, so a re-run with bidirectional S6, complex parameters, and the same schedule would test whether the reported gain comes from the B2S6 changes or from the training setup.","Theorem 3 fixes $w=0$, meaning the stability analysis holds only for the length-dependent part of the selective mechanism; the input-dependent part remains unanalyzed, so the reduced-learning-rate recommendation is an engineering remedy whose theoretical justification covers a restricted regime.","The block structure alone restores universal approximation, which suggests that multihead selective models already inherit part of the expressiveness fix; this isolates B2S6's distinctive contribution to the bias term and the learning-rate treatment."],"supporting_citations":[{"why":"Defines the S6 unit and reports the Mamba baseline that B2S6 extends and compares against.","marker":"[26]"},{"why":"Introduces S4 and supplies the Long-Range Arena baseline that B2S6 outperforms.","marker":"[28]"},{"why":"Introduces S4D, the non-selective unit whose universal approximation and stability properties are contrasted with S6 and B2S6.","marker":"[27]"},{"why":"Suggests the multihead or block-wise selective structure that B2S6 adopts for its block partitioning.","marker":"[20]"},{"why":"Cited as the source of the Long-Range Arena benchmark whose results support the claim that S6 underperforms.","marker":"[3]"},{"why":"Provides S5, another non-selective SSM baseline in the LRA comparison that B2S6 exceeds.","marker":"[77]"},{"why":"Provides S7, a selective SSM baseline that B2S6 outperforms on the Long-Range Arena average.","marker":"[79]"},{"why":"Supplies the SlimPajama corpus used for the language-modeling perplexity comparison.","marker":"[78]"},{"why":"Establishes universal approximation results for state-space models that the proof of S4D's universality builds on.","marker":"[87]"}],"fun_headline_variants":["B2S6 makes Mamba a long-range contender: 87.7 on LRA","Block-wise bias lifts Mamba from 66 to 87.7 on long-range","Mamba's long-range weakness fixed with B2S6's biased blocks","B2S6 restores universal approximation, stabilizes Mamba training","Mamba meets its match: B2S6 boosts LRA to 87.7"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the S6 baseline was trained comparably to B2S6, with similar bidirectional structure, normalization, and schedule, and the stability theorem's $w=0$ assumption means the training-instability diagnosis does not cover the input-dependent selection mechanism itself.","fun_headline_variants_meta":{"raw":{"variants":["B2S6 makes Mamba a long-range contender: 87.7 on LRA","Block-wise bias lifts Mamba from 66 to 87.7 on long-range","Mamba's long-range weakness fixed with B2S6's biased blocks","B2S6 restores universal approximation, stabilizes Mamba training","Mamba meets its match: B2S6 boosts LRA to 87.7"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000213,"raw_usage":{"total_tokens":1459,"prompt_tokens":1018,"completion_tokens":441,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":634,"completion_tokens_details":{"reasoning_tokens":332}},"tokens_in":634,"tokens_out":441,"duration_ms":4147,"temperature":1.0,"reasoning_tokens":332,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:43:27.242947+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the exact Mamba/S6 configuration with bidirectional processing, complex parameters, and the same learning-rate schedule used for B2S6, and run it on all six Long-Range Arena tasks; if a properly tuned bidirectional S6 reaches or passes the 87.71 average that B2S6 reports, the paper's core claim that S6 fails on these tasks is refuted. A second check is to train single-layer S6 and S4D models on a continuous target with growing width: if S6's error falls with width in the same way as S4D's, then the non-universality theorem's practical significance fails.","supporting_citations":[],"review_version":1}