{"id":"3d41dc0d-dd72-43e6-9038-420fad2f311e","arxiv_id":"2411.19455","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"For diagonal state space models, the initialization timescale should scale with the sequence length and the largest eigenvalue of the input autocorrelation matrix, and eigenvalue placement controls memory and optimization conditioning.","lead":"This paper derives a stability bound showing that the right time step for initializing a state space model depends on the autocorrelation of the input data, not just the sequence length. It also argues that zero real-part eigenvalues can help long-memory tasks and that imaginary-part spacing controls optimization conditioning, with a frequency-alignment tradeoff.","discovery_kind":"first_principles","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 2 is false as stated: conjugate imaginary parts ±a make the Gram matrix rank-one, so the conditioning theorem needs an explicit positivity assumption.","rationale":"The reader's weakest_assumption (Theorem 1 upper bound not matched by a lower bound) is real but mainly affects interpretation of the scaling recommendation; the numerical experiments in Figures 2-4 support the claimed tightness for i.i.d. and constant inputs, making it a softer concern. The more serious internal issue is Section 4.3's Proposition 1 and Theorem 2. The basis functions are e^{-s/2} cos(v_j s), even in v_j, so a pair of conjugate imaginary parts yields identical basis functions and a rank-one Gram matrix. This is not a boundary case: it occurs for any a > 0, e.g., v = (-100, 100), where δ = 200, well inside the regime where Theorem 2 claims a positive λ_min lower bound. Thus Theorem 2 is false as stated, and Proposition 1, used to justify nonsingularity, is also false. The proof's Gershgorin argument does bound the usual positive-frequency case; it simply omits the required positivity and ordering assumptions. Since the paper's experiments use S4D-Lin with positive v_j, the empirical conclusions are not necessarily wrong, but a central theoretical claim needs a corrected statement and proof. This warrants the same conditional verdict, now with a precise revision requirement.","tokens_in":20474,"tokens_out":6775,"duration_ms":56024,"concrete_test":"Compute the 2×2 Gram matrix for w = (-0.5 + 100 i, -0.5 - 100 i) using Lemma 5: diagonal = 1/2(1 + 1/(1+40000)), off-diagonal = 1/2(1/(1+40000) + 1), which are exactly equal, so G is rank-one and λ_min(G) = 0. This contradicts Theorem 2 for δ = 200. As a second check, rerun the proof of Theorem 2 under the additional assumption 0 < v_1 < ... < v_m and confirm that the displayed inequality for the off-diagonal sum relies on positivity and ordering; no analogous bound holds for signed v_j.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Proposition 1 and Theorem 2 in Section 4.3 claim that for w_j = -0.5 + i v_j with distinct v_j ∈ R and separation δ > 0, the Gram matrix G_jk = ∫_0^∞ e^{-s} cos(v_j s) cos(v_k s) ds is positive definite and satisfies λ_min(G) > 1.19 - (3π/4δ)coth(π/δ). This is false for arbitrary real v_j because cos is even: if v_1 = -a and v_2 = a, then ℜ(e^{w_1 s}) = e^{-s/2} cos(a s) = ℜ(e^{w_2 s}), so the two basis functions coincide, G = [[β,β],[β,β]] is rank-one, and λ_min(G) = 0. For a = 100, δ = 200, the stated lower bound is positive, directly contradicting the theorem. The proof's off-diagonal bound implicitly assumes the v_j are positive and sorted; this is not stated. The same counterexample refutes Proposition 1's assertion that distinct nonzero imaginary parts guarantee positive definiteness. The practical S4D-Lin initialization uses positive v_j, so the intended application may survive, but the theorem as stated is internally inconsistent and must be repaired, e.g., by requiring 0 < v_1 < ... < v_m or by replacing v by |v|.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies initialization schemes for diagonal state space models (SSMs), focusing on three components: the timescale Delta, the real part of the eigenvalues of the state matrix, and the imaginary part. In Section 4.1, Theorem 1 gives an upper bound on the expected squared output at initialization, E_{c,x}[y_L^2] <= Delta^2 m^2 L lambda_max(E[xx^T]), under Re(w_j) <= 0 and Gaussian read-out, and the authors propose initializing Delta as O(1/sqrt(L lambda_max(E[xx^T]))). In Section 4.2, they argue that initializing Re(w)=0 can mitigate the curse of memory for fixed-length tasks. In Section 4.3, they analyze a convex least-squares problem with fixed w and show that the conditioning of the induced Gram matrix is controlled by the separation of the imaginary parts Im(w), leading to an approximation-estimation tradeoff. The paper includes numerical experiments on synthetic processes, sequential MNIST/CIFAR, copying tasks, and the LRA benchmark.","tokens_in":20771,"tokens_out":4311,"duration_ms":36570,"significance":"If the main results hold after repair, the paper provides a useful, data-dependent guideline for SSM initialization: the timescale should depend on the spectrum of the input autocorrelation matrix, not just on sequence length, and complex eigenvalues with well-separated imaginary parts can yield well-conditioned optimization problems. The derivations are self-contained, with no fitted constants, and the numerical experiments align with the sufficient bound in several cases. The paper also connects to the curse-of-memory literature and offers a concrete, falsifiable prediction about timescale scaling. However, the false statement of Proposition 1 and Theorem 2 as written, and the unproven necessity of the timescale recommendation, prevent acceptance in the current form.","major_comments":[{"comment":"Proposition 1 is false as stated. The statement allows v_j in R with all v_j distinct and nonzero, but if v_1 = -a and v_2 = a with the same real part, then Re(e^{w_1 s}) = e^{-s/2} cos(a s) = Re(e^{w_2 s}), so the two basis functions coincide and the Gram matrix G is rank-one, contradicting positive definiteness. The proof's Vandermonde argument also fails in this case because the nodes e^{gamma w_j} and e^{gamma \\bar w_k} coincide when v_j = -v_k. The proposition needs an explicit positivity condition such as 0 < v_1 < ... < v_m, or a replacement of v by |v| with distinct absolute values.","section":"Section 4.3, Proposition 1"},{"comment":"Theorem 2 inherits the same flaw as Proposition 1. Its statement assumes only v_j in R and delta := min_{j != k} |v_j - v_k| > 0, but for v_1 = -a and v_2 = a the Gram matrix is singular while delta = 2a > 0, so the claimed lower bound lambda_min(G) > 1.19 - (3 pi / (4 delta)) coth(pi/delta) is false for such choices. The proof in Appendix E explicitly uses the condition 'v_j > 0 and reordering {v_k}_{k>=1}', which is not stated in the theorem. The theorem must be restated with 0 < v_1 < ... < v_m. Additionally, the lower bound is vacuous for delta <= 2.3, a gap the authors acknowledge; the theorem should state clearly for which delta the bound is nontrivial.","section":"Section 4.3, Theorem 2"},{"comment":"The paper claims in the abstract and introduction to 'rigorously characterize the dependency' of the timescale on sequence length based on autocorrelation, but Theorem 1 provides only an upper bound E_{c,x}[y_L^2] <= Delta^2 m^2 L lambda_max(E[xx^T]). The recommendation Delta = O(1/sqrt(L lambda_max(E[xx^T]))) is sufficient for the bound to be O(1), but no matching lower bound is proved showing that larger Delta leads to instability. The tightness discussion after Theorem 1 covers only Re(w)=0 with i.i.d. inputs and the constant-sequence case; the general autocorrelation case is asserted rather than proved. To support the 'characterize' language, the authors should either prove a converse under appropriate conditions or explicitly state that only a sufficient condition is established.","section":"Section 4.1, Theorem 1 and following discussion"}],"minor_comments":[{"comment":"In the paragraph discussing sequential image datasets, the text refers to 'Figure 1 (Left)' when describing the maximal eigenvalue scaling with resize rate; this should be 'Figure 1 (Right)', since the left panel shows the copying-task accuracy and the right panel shows lambda_max(E[xx^T]) for sMNIST and sCIFAR10.","section":"Section 4.1, Figure references"},{"comment":"The LRA results in Table 1 and the sCIFAR ablations in Table 3 report single runs without standard deviations or multiple seeds. Given that the improvements over baseline are a few points and that the claimed benefit is the central empirical support for Section 4.2, error bars or at least a note on run-to-run variability would strengthen the evidence.","section":"Table 1 and Table 3"},{"comment":"The pooling-mode formula writes y = (1/L) sum_{\\ell=1}^L y_L^2, which appears to be a typo: it should be y = (1/L) sum_{\\ell=1}^L y_\\ell^2, with the average taken over output positions.","section":"Remark 2"},{"comment":"The proof of Theorem 2 uses the identity 1/(1+v_j^2+v_k^2) <= 1/(1+v_j^2+delta^2 k^2) after reordering, which relies on both v_j > 0 and an ordering of the v_k. This dependency should be made explicit in the theorem statement, as noted in the major comments; the unattributed reordering makes the proof difficult to follow.","section":"Appendix E, proof line"}],"recommendation":"major_revision","confidential_remarks":"The false statement of Proposition 1 and Theorem 2 is a genuine correctness issue, but it is local and fixable by adding a positivity assumption on the imaginary parts, which is already satisfied by the S4D-Lin and S4D-Legs initializations used in the experiments. The missing lower bound in Theorem 1 is a more delicate gap: the paper's headline claim of 'characterizing' the Delta-L dependency is stronger than what the upper bound proves. I would recommend a major revision requiring the authors to repair the theorem statements and either add a converse or soften the characterization claim. The overall line of research is relevant and the paper is likely to be of interest to the SSM community."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The genuinely new and useful piece is Theorem 1: for ZOH-discretized diagonal SSMs with Re(w)≤0 and Gaussian c, E[y_L^2] ≤ Δ² m² L λ_max(E[xx^T]). That gives practitioners a data-dependent rule Δ = O(1/√(L λ_max)), and the paper shows numerically that the right power of L shifts from 1/√L for i.i.d. to 1/L for strongly correlated sequences. The zero-real-part experiments and the curse-of-memory discussion are also worth reading; the LRA results suggest real benefit even without stable reparameterization.\n\nThe soft spot is Section 4.3, and it's not minor. Proposition 1 and Theorem 2 claim positive definiteness and a lower eigenvalue bound for any distinct real v_j, but that's false. Take v_1 = -a and v_2 = a. Since cos is even, ℜ(e^{w_1 s}) = ℜ(e^{w_2 s}); the two basis functions coincide, the Gram matrix is rank-one, and λ_min(G) = 0. That directly contradicts Theorem 2 for large a (δ = 2a, bound positive). The proof of Theorem 2 appears to assume v_j > 0 in the text ('the last inequality is because v_j > 0'), but the theorem statement never says that. Same for Proposition 1. This is a load-bearing flaw in the conditioning analysis, but it's repairable: require 0 < v_1 < ... < v_m, or replace v by |v|. The practical S4D-Lin initialization uses positive v_j, so the intended conclusion—well-separated positive frequencies give a well-conditioned Gram matrix—survives, and the comparison with the Hilbert matrix for real-valued SSMs is still convincing for the intended setting.\n\nOther concerns are smaller. Theorem 1 is an upper bound and the paper sometimes talks as if it's a characterization; the tightness discussion is honest about the i.i.d. Re(w)=0 case. LRA experiments lack error bars, and no code is shipped. Both are addressable.\n\nBottom line: the timescale result is solid, the conditioning story is right in the intended regime but the theorem as printed is false and must be fixed. I'd send it to review with a clear request for that repair. It's not desk-reject material, but the authors need to add the positivity assumption and re-check Proposition 1 before this is publishable.","headline":"Useful timescale bound and a genuine conditioning insight, but Theorem 2 is false as stated because it silently assumes positive imaginary parts.","tokens_in":21241,"tokens_out":3315,"would_cite":false,"duration_ms":25702,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","68T05","68Q32"],"pacs":[],"model":"deepseek-v4-flash","headline":"The correct timescale for initializing a state space model depends on the autocorrelation spectrum of its input, and zeroing the state eigenvalues' real part can extend memory without sacrificing stability.","keywords":["state space models","initialization schemes","timescale scaling","input autocorrelation","training stability","curse of memory","Gram matrix conditioning","approximation-estimation tradeoff"],"falsifier":"Run a zero-order-hold diagonal SSM with $\\Re(w)=0$ and inputs drawn from an Ornstein-Uhlenbeck process with a fixed correlation length, for which $\\lambda_{\\max}(\\mathbb{E}[xx^\\top])$ grows sublinearly in $L$, and measure the largest $\\Delta(L)$ at which $\\mathbb{E}[y_L^2]$ stays bounded as $L$ increases. If $\\Delta^*(L)\\sqrt{L\\,\\lambda_{\\max}(\\mathbb{E}[xx^\\top])}$ is not roughly constant—for instance, if the empirical threshold $\\Delta^*$ decays much faster or slower than $1/\\sqrt{L\\,\\lambda_{\\max}}$—then the upper bound is not tight and the paper's scaling rule does not describe the true stability frontier.","tokens_in":20278,"feed_emoji":"⏱️","tokens_out":11297,"duration_ms":86097,"temperature":0.7,"pith_summary":"This paper gives a theoretical account of how to initialize a diagonal state space model (SSM) for fixed-length sequence tasks. Its first result bounds the expected square of the initialized output by $\\Delta^2 m^2 L \\lambda_{\\max}(\\mathbb{E}[xx^\\top])$, which ties the timescale $\\Delta$ to both the sequence length $L$ and the autocorrelation spectrum of the input. The paper concludes that $\\Delta$ should scale as $O(1/\\sqrt{L\\,\\lambda_{\\max}(\\mathbb{E}[xx^\\top])})$, ranging from $L^{-1/2}$ for uncorrelated inputs to $L^{-1}$ for fully correlated ones. It then argues that with this $\\Delta$, setting the real part of the state matrix eigenvalues to zero avoids exponential decay of the SSM kernel and eases the curse of memory while keeping initialization stable. Finally, it shows that the imaginary parts of the eigenvalues control the conditioning of a convex training objective, with well-separated imaginary parts giving fast convergence but risking a frequency mismatch with the target function, producing an approximation-estimation tradeoff.","feed_headline":"SSM timescale should follow input autocorrelation, not length alone","feed_subtitle":"New bound ties safe initialization scale to data correlations; zero-real-part eigenvalues extend memory.","key_machinery":"The load-bearing objects are the input autocorrelation matrix $\\mathbb{E}[xx^\\top]$ and the SSM kernel (memory) function $\\Re(c^\\top e^{w s})$. Theorem 1 is carried by writing the final output as $y_L=\\Delta\\, c^\\top V J x$ with $V$ a Vandermonde-type matrix built from powers of the discretized state eigenvalues $e^{\\Delta w_j}$; the bound follows from the factorization $V=\\tfrac12\\Phi^H D V_L$, the estimate $|(e^z-1)/z|\\le 1$ for $\\Re(z)\\le 0$, and the observation that the exchange matrix $J$ is orthogonal, leaving $\\lambda_{\\max}(\\mathbb{E}[xx^\\top])$ as the only input-dependent scale. For the optimization analysis, the central identity is the Gram matrix $G_{jk}=\\int_0^\\infty \\Re(e^{w_j s})\\Re(e^{w_k s})\\,ds$ of the basis kernels; its extreme eigenvalues are bounded in terms of the separation distance $\\delta=\\min_{j\\ne k}|\\Im(w_j)-\\Im(w_k)|$, and the Hilbert-matrix case emerges when all imaginary parts vanish, giving exponentially growing condition numbers.","core_discovery":"The paper's central claim is that the three classic components of SSM initialization—timescale, real part of the state eigenvalues, and imaginary part of the state eigenvalues—should be chosen from the data, not from a universal rule. For a zero-order-hold discretized diagonal SSM with $\\Re(w_j)\\le 0$ and Gaussian readout, Theorem 1 gives $\\mathbb{E}_{c,x}[y_L^2]\\le \\Delta^2 m^2 L\\,\\lambda_{\\max}(\\mathbb{E}[xx^\\top])$, and the paper treats this as the right dependence of $\\Delta$ on $L$: initialize $\\Delta=O(1/\\sqrt{L\\,\\lambda_{\\max}(\\mathbb{E}[xx^\\top])})$. Because $\\lambda_{\\max}(\\mathbb{E}[xx^\\top])$ ranges from $1$ for i.i.d. inputs to $L$ for constant inputs, the recommended scaling interpolates between $1/\\sqrt{L}$ and $1/L$. With that timescale, the real part of $w$ may be initialized at zero without an output blow-up, removing the exponential decay that causes poor long-term memory. For optimization, the paper analyzes the Gram matrix $G_{jk}=\\int_0^\\infty \\Re(e^{w_j s})\\Re(e^{w_k s})\\,ds$ of the kernel basis and proves, via the Gershgorin circle theorem, that its conditioning is governed by the separation of the imaginary parts $\\Im(w_j)$; this explains the advantage of complex-valued over real-valued SSMs and reveals the tradeoff between well-conditioned optimization and accurate approximation of targets with close frequencies.","pith_inferences":["The paper does not prove that the upper bound in Theorem 1 is tight for general autocorrelated inputs; if it is loose, the recommended $\\Delta$ is an overestimate of safety, and larger timescales might be usable to retain more memory—a matching lower bound would settle this.","A practical pipeline the paper gestures at but does not automate is to estimate $\\lambda_{\\max}(\\mathbb{E}[xx^\\top])$ (or the full spectrum) from a batch of sequences, set $\\Delta$ from that estimate, then choose the imaginary frequency grid by thresholding the recovered memory function's Fourier content; this could be tested end-to-end on the LRA tasks.","The conditioning analysis fixes $w$ and optimizes only the readout $c$; extending the argument to the full non-convex dynamics where $w$ and $\\Delta$ also move would test whether the zero-real-part benefits survive gradient descent, since the paper only observes that some non-negative real parts remain after training.","Whitening or decorrelating inputs before an SSM changes $\\lambda_{\\max}(\\mathbb{E}[xx^\\top])$ to $1$, which by the paper's rule shifts the correct initialization from $\\Delta\\sim 1/L$ to $\\Delta\\sim 1/\\sqrt{L}$; this connects the initialization theory directly to data preprocessing choices."],"forward_implications":["For uncorrelated or whitened inputs, the paper's rule says to initialize $\\Delta\\sim L^{-1/2}$ rather than the common $L^{-1}$, and its copying-task experiments show this choice can turn a task that previously seemed to need large state size into a fixed-length task a small SSM can learn.","For strongly correlated inputs, $\\lambda_{\\max}(\\mathbb{E}[xx^\\top])\\sim L$, so the rule reverts to $\\Delta\\sim L^{-1}$, matching common practice but now with a data-dependent justification.","Initializing the real part of the eigenvalues at zero with the correct timescale keeps the output bounded while making the kernel non-decaying, so targets with long-range spikes (like a shift by $L$ positions) become learnable without stable reparameterization.","Complex-valued SSMs with well-separated imaginary parts have Gram matrices whose condition number stays bounded as the hidden size grows, unlike real-valued SSMs, whose Gram matrix approaches a Hilbert matrix.","Spreading the imaginary frequencies too far improves conditioning but can miss the dominant frequencies of the target memory function, so initialization should balance conditioning against approximation error."],"supporting_citations":[{"why":"Supplies the diagonal SSM parameterization, ZOH discretization, S4D-Lin/S4D-Real initialization, and the $\\Delta\\sim 1/L$ convention that Theorem 1 revises.","marker":"Gu et al. 2022c"},{"why":"Provides the HiPPO-based initialization framework whose timescale choice ignores input autocorrelation; the paper's data-dependent rule is contrasted with it.","marker":"Gu et al. 2023"},{"why":"Introduces HiPPO recurrent memory with orthogonal polynomial projections, the basis for the initialization schemes under study.","marker":"Gu et al. 2020"},{"why":"Defines the curse of memory for recurrent models; the zero-real-part result is aimed at avoiding this exponential-decay limitation.","marker":"Li et al. 2022"},{"why":"Earlier formulation of the curse of memory in approximation and optimization of RNNs, motivating the long-memory experiments.","marker":"Li et al. 2021"},{"why":"Establishes the exponential growth of the Hilbert matrix condition number, used to contrast real-valued SSM conditioning with complex-valued SSMs.","marker":"Todd 1953"},{"why":"Supplies the Long Range Arena benchmark used to test initialization with fractions of zero real parts.","marker":"Tay et al. 2021"},{"why":"Provides the Hanson-Wright inequality used to turn the expectation bound in Theorem 1 into a high-probability bound.","marker":"Rudelson and Vershynin 2013"}],"fun_headline_variants":["Autocorrelation, not length, sets SSM timescale","SSM init: scale with input autocorrelation, not length","Zero real-part SSM weights: stable memory with right timescale","SSM conditioning tied to imaginary eigenvalue spacing","For SSMs, autocorrelation picks the timescale, not length"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the upper bound on the expected squared output actually marks the edge of stable training—if that bound is loose for correlated inputs, the recommended timescale is merely a safe overestimate and the claimed link between autocorrelation and initialization loses its force.","fun_headline_variants_meta":{"raw":{"variants":["Autocorrelation, not length, sets SSM timescale","SSM init: scale with input autocorrelation, not length","Zero real-part SSM weights: stable memory with right timescale","SSM conditioning tied to imaginary eigenvalue spacing","For SSMs, autocorrelation picks the timescale, not length"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000265,"raw_usage":{"total_tokens":1665,"prompt_tokens":1058,"completion_tokens":607,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":674,"completion_tokens_details":{"reasoning_tokens":523}},"tokens_in":674,"tokens_out":607,"duration_ms":5508,"temperature":1.0,"reasoning_tokens":523,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:10:19.554853+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a zero-order-hold diagonal SSM with $\\Re(w)=0$ and inputs drawn from an Ornstein-Uhlenbeck process with a fixed correlation length, for which $\\lambda_{\\max}(\\mathbb{E}[xx^\\top])$ grows sublinearly in $L$, and measure the largest $\\Delta(L)$ at which $\\mathbb{E}[y_L^2]$ stays bounded as $L$ increases. If $\\Delta^*(L)\\sqrt{L\\,\\lambda_{\\max}(\\mathbb{E}[xx^\\top])}$ is not roughly constant—for instance, if the empirical threshold $\\Delta^*$ decays much faster or slower than $1/\\sqrt{L\\,\\lambda_{\\max}}$—then the upper bound is not tight and the paper's scaling rule does not describe the true stability frontier.","supporting_citations":[],"review_version":1}