{"id":"28c9ff9b-099f-40e1-9c13-242333748226","arxiv_id":"2501.11729","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"SeRpEnt compresses sequences for state space models by resampling around high-information elements, giving small gains on three Long Range Arena tasks and one language modeling benchmark.","lead":"This paper proposes SeRpEnt, a method that shortens sequences before feeding them into state space models by resampling around elements that matter most. It also offers a theoretical explanation for why Mamba's learned time intervals track information content, plus modest empirical gains on three Long Range Arena tasks and one language modeling benchmark.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The information-proxy claim lacks validation: Proposition 1 proves only a Euclidean state perturbation linear in Δ_m as Δ_m→0, while the paper's 'information' is KL divergence, which is quadratic in parameter distance, and SeRpEnt uses finite Δ_l ∈ [κΔ, Δ].","rationale":"The paper's central claim and its proposed compression both depend on Δ_l tracking token informativeness. The theoretical support consists of Proposition 1, which is a statement about the Euclidean norm of a state perturbation, plus a one-sentence heuristic connecting that norm to KL divergence. This is the weakest link in the argument: the proposition does not address the predictive distribution, the connection is not made rigorous, and in the standard smooth-parametric treatment the connection would predict quadratic rather than linear scaling in Δ_m, undercutting the paper's headline. In addition, the regime of the proposition (Δ_m→0) is not the regime used by SeRpEnt (Δ_m bounded below by κΔ). The proposed test would settle the matter by directly measuring whether Δ_i correlates with an actual information measure in a trained model. I do not see a deeper flaw in the architecture itself: the resampling idea is coherent and the empirical gains, while incomplete, are suggestive. The reader's conditional verdict is appropriate; the theory needs either proof or soft language, and the empirical validation should include the proposed information-correlation check.","tokens_in":11286,"tokens_out":7578,"duration_ms":73996,"concrete_test":"Train or reuse the small Mamba model described in §5.2. For a set of validation sequences, for each token i compute: (a) Δ_i; (b) ||h_L - h_L^i|| by running the recurrence with the i-th step removed; (c) the leave-one-out KL divergence between the model's predictive distribution over the next token with and without token i, holding all other inputs fixed. Then report Spearman correlations among (a), (b), and (c), and fit log(KL) vs log(Δ_i) to estimate the exponent. If KL and Δ_i are not significantly correlated, or if the log-log slope is ≈2 rather than ≈1 over the [κΔ, Δ] range actually used by SeRpEnt, the central 'linear approximator of information' claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 defines information as the change in the predictive distribution induced by deleting an element and says this is measured by KL divergence, then asserts that KL is 'related to' the Euclidean distance ||h_L - h_L^m||. That bridge is load-bearing and unproven. For smooth parametric families, KL(p_h||p_{h'}) ≈ (1/2)(h-h')^T F(h)(h-h'), so KL is quadratic, not linear, in the state perturbation. Proposition 1 (Appendix A) gives only ||h_L - h_L^m|| ~ c Δ_m as Δ_m→0, so even under the heuristic bridge the information change would scale as O(Δ_m^2), not O(Δ_m). The asymptotic constant c (Eq. A11) also contains h_{m-1} and x_m/α, so the perturbation size is context-dependent; Δ_m alone is not a sufficient statistic for it. Finally, Eq. (13) restricts Δ_l to [κΔ, Δ] with κ>0, outside the Δ_m→0 limit in which Proposition 1 is proved, so the linearity is extrapolated. No independent information measure is reported to validate the proxy. The proof also has a minor algebra slip: the norm constant C should be Σ_j c_j^2, not (Σ_j c_j)^2, though linearity survives.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper analyzes the selectivity mechanism in Mamba-style state space models and claims that the learned selective time intervals act as linear approximators of information. It proves Proposition 1, an asymptotic statement that for a discrete SSM with diagonal A and zero-order hold discretization, removing element m changes the final hidden state by a norm proportional to Δ_m as Δ_m → 0. Building on this, the paper proposes SeRpEnt, a resampling architecture that compresses input sequences by aggregating elements according to their Δ_l values, then applies an SSM to the compressed sequence and decompresses the output. The method is evaluated on a subset of the Long Range Arena benchmark and on WikiText-103 language modeling, reporting improvements over base S4, S5, Liquid-S4, and Mamba baselines.","tokens_in":11583,"tokens_out":5328,"duration_ms":54091,"significance":"The question of why selectivity helps in Mamba is important, and the proposed information-proxy interpretation is conceptually attractive. If the claim that Δ_l values track information content were properly validated, it would be a useful lens for designing SSM variants, and the SeRpEnt architecture is a reasonable practical instantiation of that idea. The paper includes a mathematical proposition with a proof, which is a strength, and the experiments are clearly described with hyperparameters. However, the central bridge between state-vector perturbation and information is only asserted as a heuristic, and the empirical support is weakened by the exclusion of image tasks and by internal inconsistencies in the reported improvements. The contribution is therefore promising but not yet fully established.","major_comments":[{"comment":"The definition of information as the change in the predictive distribution p*(y*|h_L) is reasonable, but the connection between this information and the Euclidean distance ∥h_L − h_L^m∥ is asserted rather than proved. The text says the KL divergence is 'related to' the Euclidean distance, yet for smooth parametric families KL(p_h ∥ p_h') is locally quadratic in the parameter difference, roughly (1/2)(h−h')^T F(h)(h−h'). Consequently, even granting this bridge, Proposition 1 would imply that information scales as O(Δ_m^2), not O(Δ_m). The paper does not validate the proxy with any independent information measure, so the central claim that 'selective time intervals act as linear approximators of information' is not supported by the provided evidence.","section":"§3.2"},{"comment":"Proposition 1 is proved only in the asymptotic regime Δ_m → 0, but SeRpEnt constrains Δ_l to the finite interval [κΔ, Δ] with κ > 0 and a learnable Δ. Nothing guarantees that this interval contains values close enough to zero for the asymptotic linearity to hold. The architecture relies on comparing Δ_l values across this finite range, so the linearity is extrapolated far outside the regime where it is justified. The paper should either prove a non-asymptotic bound or provide empirical evidence that the linear approximation is accurate in the operating range.","section":"§4.2, Eq. (13)"},{"comment":"The asymptotic constant c in Eq. (A11) depends on h_{m−1} and x_m, as well as on the subsequent time intervals. This means the norm of the perturbation is not determined by Δ_m alone; the same Δ_m can produce very different perturbations depending on the state and input context. Therefore, Proposition 1 does not by itself justify treating Δ_l as a sufficient proxy for token importance. The paper should clarify whether the proposed resampling relies on Δ_l as an information proxy or only as a relative ordering heuristic, and what role the context dependence plays.","section":"Appendix A, Eq. (A11)"},{"comment":"The text states that SeRpEnt improves the performance of the baseline on every task except for image ones, but Table 1 contradicts this: SeRpEnt+S5 is worse than S5 on Retrieval (89.04 vs. 90.49) and SeRpEnt+Liquid-S4 is worse than Liquid-S4 on ListOps (60.35 vs. 60.65). The average improvements are also small (e.g., +0.68 for S4, +0.63 for S5). The empirical claim should be corrected, and the paper should either report the full LRA benchmark including image tasks or clearly justify the exclusion, since the current presentation overstates the consistency of the gains.","section":"§5, Table 1"}],"minor_comments":[{"comment":"The norm constant is mis-specified: in Eq. (A15), C should be Σ_j c_j^2, not (Σ_j c_j)^2, and consequently the constant in Eq. (A16)–(A17) should be √(Σ_j c_j^2). The linearity conclusion is unaffected, but the algebra should be corrected.","section":"Appendix A, Eq. (A15)–(A17)"},{"comment":"There are typos in the caption: 'T op Right' should be 'Top Right', and the spacing in 'T op Right' and 'time variable intervals' should be fixed.","section":"Fig. 1 caption"},{"comment":"The phrase 'both empirically [8] and inad-hoc benchmarks [9]' contains a typo; it should likely be 'in ad-hoc benchmarks'.","section":"§1"},{"comment":"The symbol M is used where a summation symbol seems intended; please clarify the notation for the concatenation/summation over the neighbors.","section":"Eq. (12b)"},{"comment":"The capitalization of 'SeRpent' in Table B1 is inconsistent with 'SeRpEnt' used elsewhere.","section":"§B.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript addresses an interesting question and contains a technically sound proposition, but the central information-interpretation rests on an unproven heuristic and the empirical evidence is not as consistent as claimed. The recommended path is major revision: either validate the KL-to-Euclidean bridge with an independent information measure or explicitly reframe the claim as a heuristic, and correct the experimental reporting. The paper's scope may also be somewhat narrow for a general machine learning venue, though the topic is relevant for the SSM community."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the architectural core of SeRpEnt is genuinely new: it resamples an SSM input by nearest-neighbor interpolation with a Gaussian expansion of time offsets, and it learns the resampling timescale from the input. Second, the paper's theoretical justification for that design—that Mamba's learned Δ_l are 'linear approximators of information'—is not actually established. Proposition 1 is a correct Taylor expansion: deleting element m perturbs the final state by a norm that scales like c·Δ_m as Δ_m→0. The proof has a small algebra slip—the constant C should be Σ_j c_j^2, not (Σ_j c_j)^2—but the linearity claim survives.\n\nWhat the paper does well is propose a concrete, cleanly described mechanism that can be dropped into any SSM layer, and it reports consistent (if modest) average gains on three of the six LRA tasks, with S4, S5, and Liquid-S4 as bases. The references are appropriate and the related work is fair.\n\nThe soft spots are real but fixable. The 'information' interpretation has two problems. The paper defines information as a change in the predictive distribution and invokes KL divergence, but for smooth parametric families KL is quadratic in state distance, so even if the Euclidean state perturbation is O(Δ_m), the KL change would be O(Δ_m^2), not linear. The heuristic bridge from Euclidean distance to KL is never validated against any independent information measure, and Proposition 1 only holds as Δ_m→0 while the method forces Δ_l into [κΔ, Δ] with κ>0. So the 'linear approximator' phrasing oversells the result. You can keep the resampling as a heuristic, but you can't claim it is information-theoretically grounded.\n\nThe experimental reporting also needs work. LRA excludes image tasks, yet the text claims improvement on every task except image ones, while Table 1 shows SeRpEnt+S5 doing worse than S5 on Retrieval. The language modeling table lists 46 perplexity for both Mamba and SeRpEnt+Mamba while claiming outperformance; that is confusing. No code, no error bars, and likely single seeds.\n\nWho is this for? Researchers working on SSM efficiency and sequence compression. It is a subfield-level contribution, not a breakthrough, but it is a reasonable architectural idea with honest limitations once you get past the overclaims. I would send it to peer review rather than desk-reject, because the core mechanism is new and the issues are in interpretation and presentation. With the full LRA, corrected tables, code, and a softer or better-supported information claim, it could be a solid paper.","headline":"A plausible architectural idea for SSM compression with a correct but mislabeled local analysis; the information interpretation is unproven and the experiments need cleaning up, but it deserves review.","tokens_in":12124,"tokens_out":3983,"would_cite":false,"duration_ms":39145,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"Mamba's per-token time intervals are linear approximations of token information, and SeRpEnt turns that signal into sequence compression.","keywords":["selective state space models","Mamba","token importance","sequence compression","resampling","long-range dependencies","language modeling","learned time intervals"],"falsifier":"On a trained Mamba model, rank tokens by their learned $\\Delta_l$ and compare the ranking with rankings from direct leave-one-out KL divergence between output distributions; substantial disagreement, especially for tokens with small $\\Delta_l$ and large KL shift, would falsify the information-proxy claim.","tokens_in":11077,"feed_emoji":"⏱️","tokens_out":6953,"duration_ms":71669,"temperature":0.7,"pith_summary":"The paper tries to explain why selectivity works in Mamba-style state space models and to turn that explanation into a practical compression layer. Its core claim is that the per-token time interval $\\Delta_l$ learned by Mamba acts as a linear approximator of information: removing a token changes the final hidden state by an amount asymptotically proportional to that token's interval. The authors then build SeRpEnt, which resamples a sequence by aggregating tokens with related information content, feeds the compressed sequences through SSM branches at several compression rates, and restores the original length. On non-image Long Range Arena tasks and WikiText-103 language modeling, SeRpEnt improves accuracy or loss over the base SSM it wraps, suggesting the selectivity signal is usable for compression.","feed_headline":"Mamba's time steps measure how much each token matters","feed_subtitle":"A new layer, SeRpEnt, resamples by information content and lifts SSM accuracy on long-range and language tasks.","key_machinery":"The load-bearing object is the asymptotic identity in Proposition 1, derived from the recurrence $h_l = \\exp(\\Delta_l A) h_{l-1} + (\\exp(\\Delta_l A)-I) x_l / \\alpha$ for a diagonal state matrix $A$. Diagonalizing $A$ separates the recurrence into scalar components, so the difference between the full and reduced final states reduces to a single leftover term whose factor $\\exp(\\alpha \\Delta_m)-1$ is linear in $\\Delta_m$ for small $\\Delta_m$; this is what licenses reading $\\Delta_l$ as an information score. On the architecture side, the compression mechanism is selective resampling: time intervals are clamped to $[\\kappa \\Delta, \\Delta]$ by a sigmoid, the sequence is interpolated at uniform times $\\Delta$ through a nearest-neighbors linear map with a Gaussian basis expansion of time differences, and multiple compression rates run in parallel branches before being concatenated back with a residual connection.","core_discovery":"The central discovery is Proposition 1: for a discrete state space model with diagonal matrix $A$ and zero-order hold discretization, deleting element $m$ alters the last hidden state by $\\lVert h_L - h_L^m\\rVert \\sim c \\Delta_m$ as $\\Delta_m \\to 0$, where the constant $c$ depends on later intervals and the state before $m$. Because the target distribution is parametric in $h_L$, the paper interprets this as saying that learned time intervals measure how much information each sequence element carries for the model's prediction. The paper's proposed SeRpEnt mechanism exploits that interpretation: it computes time intervals, resamples the input at a coarser uniform grid by interpolating among nearest neighbors weighted by Gaussian expansions of time differences, and processes the compressed sequence with an SSM, then decompresses by copying the closest output. The broad conclusion is that selectivity in Mamba is not just an empirical heuristic but an information-processing mechanism, and that sequence compression can be guided by the same signal.","pith_inferences":["A natural extension the paper does not pursue is to use $\\Delta_l$ as an importance signal for pruning or early-exit in non-SSM architectures, since Proposition 1's mechanism is specific to the discrete SSM recurrence but the ranking intuition is general.","The paper reports that image-based LRA tasks do not benefit from resampling; one testable explanation is that interpolation in time is a poor model of spatial locality, which would suggest a spatial resampling rule for vision inputs.","Because the proof is asymptotic in $\\Delta_m$ while SeRpEnt operates in a clamped finite range, measuring whether the linear proportionality actually holds at the operating $\\Delta$ values would directly test the extrapolation.","One could validate the KL-to-Euclidean bridge independently by comparing $\\Delta_l$ rankings with leave-one-out KL estimates on real token sequences; the paper does not report such a comparison."],"forward_implications":["If $\\Delta_l$ measures information, then compressing by aggregating tokens with similar time-interval values should preserve task-relevant content better than uniform pooling.","SeRpEnt should improve any compatible SSM layer, and the paper shows gains over S4, S5, and Liquid-S4 on LRA and over Mamba on WikiText-103.","Multiple compression rates let a single block model both short- and long-range dependencies, since each branch sees a different trade-off between length and detail.","The time-interval signal can serve as an input-dependent importance score without requiring attention weights or extra supervision.","Because the compression is orthogonal to the SSM layer, future SSM variants can inherit the same resampling mechanism without changing their internal updates."],"supporting_citations":[{"why":"Introduces Mamba's selectivity mechanism and the learned per-token time intervals $\\Delta_l$ that the paper reinterprets as information scores.","marker":"[13]"},{"why":"Provides the diagonal structured SSM layer whose recurrence the proof of Proposition 1 analyzes.","marker":"[12]"},{"why":"Establishes the continuous-time interpretation and initialization of SSM states that motivates treating sequences as samples from an underlying function.","marker":"[11]"},{"why":"Supplies the zero-order hold discretization formula $\\exp(\\Delta_l A)$ used to derive the asymptotic linearity.","marker":"[34]"},{"why":"Defines the nearest-neighbors interpolation procedure used for resampling compressed sequences.","marker":"[41]"},{"why":"Contributes the Gaussian basis expansion of time differences used in the resampling map.","marker":"[42]"},{"why":"Defines the Long Range Arena benchmark used to evaluate the compression mechanism.","marker":"[9]"},{"why":"Provides the WikiText-103 dataset and splits used for the language modeling evaluation.","marker":"[47]"},{"why":"Gives the KL divergence as the information-theoretic measure linking hidden-state distance to information content.","marker":"[40]"}],"fun_headline_variants":["Selective time steps in Mamba gauge token importance","SeRpEnt resamples sequences by information content","Mamba selectivity demystified and used for compression","Time interval in Mamba reveals what each token contributes","Resampling by information content boosts SSM accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the Euclidean change in the final hidden state when a token is removed faithfully measures that token's information content, a link the paper asserts through a continuity heuristic without validating it against an independent information measure.","fun_headline_variants_meta":{"raw":{"variants":["Selective time steps in Mamba gauge token importance","SeRpEnt resamples sequences by information content","Mamba selectivity demystified and used for compression","Time interval in Mamba reveals what each token contributes","Resampling by information content boosts SSM accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000187,"raw_usage":{"total_tokens":1325,"prompt_tokens":936,"completion_tokens":389,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":552,"completion_tokens_details":{"reasoning_tokens":315}},"tokens_in":552,"tokens_out":389,"duration_ms":4616,"temperature":1.0,"reasoning_tokens":315,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T17:55:39.007080+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a trained Mamba model, rank tokens by their learned $\\Delta_l$ and compare the ranking with rankings from direct leave-one-out KL divergence between output distributions; substantial disagreement, especially for tokens with small $\\Delta_l$ and large KL shift, would falsify the information-proxy claim.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the diagonal structured SSM layer whose recurrence the proof of Proposition 1 analyzes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the continuous-time interpretation and initialization of SSM states that motivates treating sequences as samples from an underlying function."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the zero-order hold discretization formula $\\exp(\\Delta_l A)$ used to derive the asymptotic linearity."},{"cited_title":"& Friedman, J.The Elements of Statistical Learning: Data Mining, Inference, and Prediction, Second Edition(Springer New York, 2009)","cited_arxiv_id":null,"evidence_quote":"Defines the nearest-neighbors interpolation procedure used for resampling compressed sequences."},{"cited_title":"SchNet: A continuous-filter convolutional neural network for modeling quantum interactions.NeurIPS (2017)","cited_arxiv_id":null,"evidence_quote":"Contributes the Gaussian basis expansion of time differences used in the resampling map."},{"cited_title":"Long range arena: A benchmark for efficient transformers.ICLR (2021)","cited_arxiv_id":null,"evidence_quote":"Defines the Long Range Arena benchmark used to evaluate the compression mechanism."},{"cited_title":"& Thomas, J.Elements of Information Theory(Wiley, 2012)","cited_arxiv_id":null,"evidence_quote":"Gives the KL divergence as the information-theoretic measure linking hidden-state distance to information content."}],"review_version":1}