{"id":"8566c07d-f3f9-4f14-88d9-d53e7506e702","arxiv_id":"2501.00692","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"The paper derives an adjoint-based gradient sharding algorithm for SSMs and claims up to 3X memory reduction, but provides no experimental evidence for the central empirical claims.","lead":"Adjoint sharding is a proposed technique for training state space models on very long context sequences by splitting gradient computation into many small vector-Jacobian products. The paper claims this reduces memory by up to three times, but the preprint does not include the experiments that would support the headline results.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The abstract's memory-reduction claims (up to 3X, orders of magnitude) are unsupported: the paper contains no experiments, and its own Algorithm 1 stores O(T K N) per-token states whose footprint at 1M context may dominate the supposed savings.","rationale":"I agree with the reader's verdict of REJECT. The theoretical derivation of adjoint sharding is internally consistent and may compute equivalent gradients, but the paper's central claims—up to 3X memory reduction, 1M-token training, and 100K-token context on five P4 instances—are entirely unsupported by any experimental measurement. The reader's weakest assumption correctly identifies that the storage of per-token states (Algorithm 1, line 10) grows as O(T K N) and is never measured. My analysis adds that this storage, when computed using the paper's own formulas at the reported scale, already approaches or exceeds the memory of a single P4 GPU, making the claimed reduction questionable even in principle. The inconsistency between 'orders of magnitude' and 'up to 3X' further weakens the empirical statement. I therefore see no reason to change the reader's verdict; the manuscript needs a real experimental section or substantive revision before its central claims can be accepted.","tokens_in":19139,"tokens_out":7543,"duration_ms":73593,"concrete_test":"Compute the peak memory required by Algorithm 4 for the configuration quoted in the abstract (T=1M, 1.27B parameters, K layers) using the paper's own storage lists in Algorithm 1 and Section 4.5, adding vjp workspaces (Table 1) and optimizer states. Compare this number to the 40 GB per-GPU memory of the five AWS P4 (A100-based) instances. If the total exceeds the per-GPU budget, the claimed 3X reduction and the ability to increase context from 35K to 100K on that infrastructure are numerically impossible.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that adjoint sharding reduces memory usage by up to 3X for a 1.27B parameter SSM at 1M context length, and by 'orders of magnitude' (abstract, conclusion), but the manuscript contains no experimental section, no memory measurements, and no code. The only support is the memory analysis in Section 4.5, which states that storing {A_t^k, C_t^k, h_t^k, y_t^k} over T tokens and K layers costs T K (2N+P) + T P FP16 numbers. This grows linearly in T K N, so for the reported scale (say T=1M, K=100, N=225, P=128) the stored states alone are roughly 5.8e10 FP16 values, or ~116 GB, before vjp workspaces or optimizer states are counted. That is already larger than the per-GPU memory of a single AWS P4 instance (A100 40GB); it only fits on the full 5-instance cluster if sharded across all GPUs, which would erase part of the claimed per-GPU reduction. The paper never compares this peak to a BPTT baseline with activation checkpointing, so the 'up to 3X' figure is an assertion, not a demonstrated result. Furthermore, the abstract says 'orders of magnitude' while the empirical sentence says 'up to 3X', and the conclusion repeats 'orders of magnitude'—an internal inconsistency. The gradient-equivalence proofs (Propositions 2, 3) are plausible, but theoretical equivalence to backpropagation does not imply lower memory; indeed, the adjoint formulation still requires storing the per-token states needed to evaluate vjps. Thus the most load-bearing, least-secure condition is the unverified memory reduction, and it fails as stated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes adjoint sharding, a method that computes gradients for state-space model (SSM) language models by decomposing backpropagation into independent vector-Jacobian products via the adjoint method. It claims this reduces training memory by up to 3X (abstract) or \"orders of magnitude\" (abstract and conclusion), enabling training on 1M-token contexts for a 1.27B parameter model on five AWS P4 instances. The paper derives gradient-equivalence results in Propositions 2 and 3, introduces a truncated variant and a distributed algorithm, and presents an analytical memory/FLOP analysis. The manuscript contains no experimental section, no measured memory numbers, and no code.","tokens_in":1365,"tokens_out":1914,"duration_ms":69410,"significance":"If the memory-reduction claims were established, adjoint sharding would be a practically valuable technique for long-context SSM training. The gradient-equivalence derivations in Propositions 2 and 3 appear internally consistent and provide a useful formalization of the adjoint approach for recurrent SSM layers. However, the practical impact is entirely unverified: the paper's central empirical assertions are unsupported, and its own memory accounting raises serious doubts about the claimed savings. For a cs.LG systems submission, the absence of experiments is a fundamental gap.","major_comments":[{"comment":"The abstract states \"Empirical results show the proposed adjoint sharding algorithm reduces memory usage by up to 3X with a 1.27B parameter large language model on 1M context length training,\" and the conclusion repeats \"orders of magnitude of memory reduction,\" yet the manuscript contains no experimental section, no measurement protocol, no memory numbers, and no code. Figure 1 is a plot without axes or experimental setup, and Figure 6 is based on assumed speedups. This is load-bearing because the paper's contribution is a training method whose benefit is memory reduction; the gradient-equivalence proofs alone do not establish any memory savings.","section":"Abstract and Conclusion"},{"comment":"The memory analysis states that storing {A_t^k, C_t^k, h_t^k, y_t^k} over T tokens and K layers costs T K(2N+P) + T P FP16 numbers. For the claimed scale (T=1M, K=100, N=225, P=128), this is approximately 5.8e10 FP16 values, or about 116 GB, which exceeds the 40 GB memory of a single AWS P4 (A100) instance. Even distributed over five GPUs, this is roughly 23 GB per GPU before vjp workspaces, model weights, and optimizer states are counted. The paper never compares this peak memory with a backpropagation baseline using activation checkpointing, so the \"up to 3X\" and \"orders of magnitude\" claims are unsupported.","section":"Section 4.5"},{"comment":"Algorithm 1, line 10, requires storing per-token, per-layer states {h_t^k, C_t^k, y_hat_t^k, A_t^k} on each device. This storage grows as O(T K N) with context length and layer count and is not a constant-memory scheme. The claimed advantage of adjoint sharding over backpropagation therefore depends on comparing this O(T K N) state storage against the memory cost of the autograd graph in BPTT, a comparison the paper does not provide. The theoretical gradient equivalence does not imply lower memory.","section":"Algorithm 1"},{"comment":"Truncated adjoint sharding is introduced with the phrase \"empirically showcase its performance,\" but no empirical evaluation appears anywhere in the manuscript. The same section later states \"We leave investigation of bar(T)'s impact on performances for future works,\" which explicitly acknowledges that the performance-maintenance claim is not established. The statement that truncated adjoint sharding achieves \"similar results\" is therefore unsupported.","section":"Section 4.3"}],"minor_comments":[{"comment":"The abstract reports \"up to 3X\" memory reduction, while both the abstract's earlier sentence and the conclusion claim \"orders of magnitude\" reduction; these are inconsistent quantitative claims.","section":"Abstract and Conclusion"},{"comment":"In the proof of Proposition 3, the vjpBi terms are written with x_hat_i^k and later changed to y_hat_i^{k-1}; the notation should be harmonized to match Proposition 3 and the main text.","section":"Appendix A.2"},{"comment":"Figure 1 and Figure 6 lack axis labels, units, and any description of the experimental or simulated setup, so they cannot be interpreted or reproduced.","section":"Figures 1 and 6"},{"comment":"The bibliography contains duplicate entries: reference [4] is repeated, and references [52] and [53] both cite the ZeRO paper; these should be consolidated.","section":"References"}],"recommendation":"reject","confidential_remarks":"The paper's abstract and conclusion make empirical claims (up to 3X memory reduction, 1M-token training, 100K-token fine-tuning) that are not supported by any experimental section or reproducible artifact. The figures appear to be illustrative or simulated, and the paper's own memory analysis in Section 4.5 suggests the proposed method stores O(T K N) per-token states, which at the claimed scale already approaches or exceeds the per-GPU memory of the cited hardware. The theoretical derivations in Propositions 2 and 3 are a reasonable starting point, but the current submission does not meet the evidentiary bar for a cs.LG systems paper. The editor may wish to verify whether the claimed empirical results exist in any external form, since the manuscript as written does not permit verification."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — the short version: this is a theory paper with one genuinely useful idea, but the abstract sells it as an empirical result, and there are zero experiments in the manuscript. The reader's reject verdict is correct.\n\nWhat's actually new: Propositions 2 and 3 re-derive the standard adjoint gradient for SSMs carefully, and the real contribution is observing that the sum over tokens and layers can be broken into independent VJPs, plus the truncated variant. That is a legitimate algorithmic idea for parallelizing SSM gradient computation, and the proofs are careful enough that I don't doubt the gradient equivalence. The distributed version and the memory/FLOP accounting in Section 4.5 are also constructive.\n\nThe soft spots are in the marketing. The abstract and conclusion claim up to 3X memory reduction on a 1.27B model at 1M context, and \"orders of magnitude\" appears earlier in the abstract. There is no memory measurement, no training curve, no code, and no baseline against BPTT with activation checkpointing. Section 4.5's own storage formula is T K (2N+P) + T P in FP16, which is linear in T K N; at the claimed scale (T=1M, K around 100, N=225, P=128) that is on the order of 100 GB before any VJP workspace, already larger than a single A100's 40 GB. So the \"up to 3X\" and \"orders of magnitude\" figures are assertions, not findings. The internal inconsistency between the two numbers doesn't help. The truncated variant is explicitly left without convergence analysis, which is acceptable for a theory contribution but not for an abstract that claims it \"maintains performance.\"\n\nIf the paper were reframed as a theory/systems-design paper and all empirical claims removed or clearly labeled as back-of-envelope estimates, it would be a reasonable workshop or theory-track paper. As written, the headline claims are unsupported and the memory analysis actually cuts against them. A serious referee should ask for major revision: add real measurements with a checkpointing baseline, report peak memory, or drop the empirical claims entirely. I'd send it to review because the core decomposition is worth engaging with, but the current version should not be accepted anywhere.","headline":"A clean adjoint-method decomposition for SSM gradients with a real parallelization idea, but the abstract's memory-reduction claims are unsupported by any experiments and the paper's own storage analysis undermines them.","tokens_in":20048,"tokens_out":3236,"would_cite":false,"duration_ms":31604,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adjoint sharding computes backpropagation-equivalent gradients for state-space models by decomposing them into independent vector-Jacobian products, cutting training memory by up to 3X at 1M-token context.","keywords":["adjoint method","state-space models","long-context training","gradient sharding","vector-Jacobian products","memory-efficient training","truncated backpropagation through time","distributed training"],"falsifier":"Train a 1.27B-parameter state-space model on a 1M-token sequence with adjoint sharding and record peak GPU memory, comparing with backpropagation under activation checkpointing on the same hardware; if adjoint sharding's peak memory is not roughly 3X lower, the abstract's central memory claim is contradicted.","tokens_in":18923,"feed_emoji":"⚡","tokens_out":12476,"duration_ms":99021,"temperature":0.7,"pith_summary":"Training large language models on very long contexts is bottlenecked by GPU memory, because backpropagation stores activations for the whole sequence, so memory grows rapidly with context length. This paper proposes adjoint sharding, which replaces backpropagation with the adjoint method for the recurrent state-space layers, decomposing the gradient into many independent vector-Jacobian products (one per token per layer) that can be computed, stored, and discarded separately. The paper proves that summing these products reproduces exactly the backpropagation gradient, and argues that sharding them across time and layers reduces memory enough to train a 1.27B-parameter state-space model on contexts above 100K tokens with a small cluster. A truncated variant cuts the number of vector-Jacobian products from quadratic to linear in context length. If the memory reduction holds, long-context training and fine-tuning of state-space architectures becomes tractable on far fewer GPUs than today.","feed_headline":"Adjoint sharding cuts long-context training memory 3X","feed_subtitle":"Replacing backpropagation with independent vector-Jacobian products lets a 1.27B model train past 100K-token contexts.","key_machinery":"The load-bearing object is the adjoint state $\\lambda_{t,\\tau}^k = C_t^k \\prod_{i=1}^{t-\\tau} A_{t+1-i}^k$ for layer $k$, which replaces the chained derivatives of backpropagation. Proposition 1 (the adjoint method for recurrences) supplies the recurrence $\\lambda_{i-1}=\\lambda_i\\,\\partial f(i,h_{i-1},\\theta)/\\partial h_{i-1}$ that makes these states computable by a backward pass without storing the forward computation graph. The residual identity $y_t^K = y_t^0 + \\sum_k \\tilde{y}_t^k$ then lets the gradient separate into a sum over layers. Each term in the resulting sum is a vector-Jacobian product of the form $\\mathrm{vjp}(v) = v \\cdot \\partial \\mathrm{Net}_\\theta(\\mathrm{Input}_i)$, which can be evaluated with reverse-mode automatic differentiation initialized with the vector $v$, and which the paper treats as costing about one forward pass. The independence of these VJPs is what allows memory to be sharded and computation to be parallelized.","core_discovery":"The central claim is that for a residual stack of state-space layers, the gradient of the training loss with respect to every parameter can be written as a sum (over tokens $t$ and layers $k$) of independent vector-Jacobian products, with no cross-dependencies between terms. For a single SSM layer, Proposition 2 gives $$\\frac{d l_t}{d\\$\\theta$} = \\Big[\\sum_{i=1}^t \\mathrm{vjp}^{A_i}\\big(\\tfrac{dl_t}{dy_t}\\lambda_{t,i}\\otimes h_{i-1}\\big)\\Big] \\oplus \\Big[\\sum_{i=1}^t \\mathrm{vjp}^{B_i}\\big(\\tfrac{dl_t}{dy_t}\\lambda_{t,i}\\otimes \\hat{x}_i\\big)\\Big] \\oplus \\mathrm{vjp}^{C_t}\\big(\\tfrac{dl_t}{dy_t}\\otimes h_t\\big),$$ with adjoint state $\\lambda_{t,\\tau}=C_t\\prod_{i=1}^{t-\\tau} A_{t+1-i}$; Proposition 3 extends this to $K$ stacked residual layers, so the full gradient is a sum of such terms over $t$ and $k$. Because each vector-Jacobian product depends only on the saved forward states and the adjoint state, the products can be sharded by time and layer across devices, computed in parallel, and discarded immediately. The paper also introduces truncated adjoint sharding, which limits the time sum to the last $\\bar{T}$ states, cutting the number of VJPs from $O(T^2)$ to $O(T\\bar{T})$ while retaining implicit dependence on all earlier states through the recurrence.","pith_inferences":["The paper does not report measured memory for the 1M-token, 1.27B-parameter case its abstract cites, only smaller-scale curves; whether the stored states keep peak memory below backpropagation at that scale is an open empirical question.","If the VJP decomposition is computed in FP16, rounding may accumulate differently than in backpropagation's sequential chain, so 'equivalent gradients' likely holds only up to numerical precision; a mixed-precision gradient-equality check would clarify this.","Truncated adjoint sharding is effectively a truncated-backpropagation-through-time variant for SSMs; its convergence and the optimal truncation length $\\bar{T}$ are left for future work, and likely depend on how fast the products $\\prod A_t$ decay, which is a testable prediction.","The paper's parallel-speedup estimate assumes VJPs are compute-bound at roughly one forward pass each; if memory bandwidth is the binding constraint, the practical speedup on GPU instances will fall below the analyzed figure."],"forward_implications":["Training or fine-tuning a 1.27B-parameter SSM on contexts above 100K tokens becomes possible on a small cluster of five instances, where backpropagation tops out near 35K tokens.","Gradient computation for residual and recurrent models becomes embarrassingly parallel across time and layers, so wall-clock training time can drop if the independent VJPs are scheduled across GPUs or GPU instances.","Truncated adjoint sharding computes an approximate gradient in linear time, making very long contexts computationally feasible where the full method's $O(T^2)$ VJP count would be prohibitive.","Because the gradient decomposition is exact for the full method, training with adjoint sharding reproduces backpropagation's training results while using less memory.","Activations, gradients, and optimizer states can all be sharded across devices, so per-GPU memory falls roughly as $1/\\Upsilon$ with $\\Upsilon$ devices."],"supporting_citations":[{"why":"Supplies the adjoint sensitivity equations for recurrence relations (Proposition 1) from which the adjoint states are derived.","marker":"[8]"},{"why":"Provides the residual sum structure y_t^K = y_t^0 + sum_k y_t^k that lets the gradient be separated layer-by-layer.","marker":"[28]"},{"why":"Defines Mamba/state-space models, the linear-time recurrent architecture whose long-context training the method targets.","marker":"[22]"},{"why":"Supports the claim that vector-Jacobian products cost about one forward pass, so sharding into VJPs does not multiply compute.","marker":"[2]"},{"why":"The truncated backpropagation-through-time precedent that truncated adjoint sharding generalizes to SSMs.","marker":"[57]"},{"why":"The activation-checkpointing memory-reduction baseline that adjoint sharding claims to improve upon.","marker":"[56]"}],"fun_headline_variants":["Adjoint sharding unlocks 100K-token SSM training","Adjoint sharding: train SSMs on 100K context","Shard gradients to extend SSM context to 100K","3X memory cut enables long-context SSM training","Memory-efficient training for long-context SSMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The promised memory savings depend on the assumption that storing the intermediate values the method needs for every token and every layer—the transition and output matrices, hidden states, and normalized inputs—costs less memory than backpropagation's stored activations, so peak memory still drops sharply.","fun_headline_variants_meta":{"raw":{"variants":["Adjoint sharding unlocks 100K-token SSM training","Adjoint sharding: train SSMs on 100K context","Shard gradients to extend SSM context to 100K","3X memory cut enables long-context SSM training","Memory-efficient training for long-context SSMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001395,"raw_usage":{"total_tokens":5777,"prompt_tokens":1211,"completion_tokens":4566,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":827,"completion_tokens_details":{"reasoning_tokens":4482}},"tokens_in":827,"tokens_out":4566,"duration_ms":32549,"temperature":1.0,"reasoning_tokens":4482,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:44:16.763900+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a 1.27B-parameter state-space model on a 1M-token sequence with adjoint sharding and record peak GPU memory, comparing with backpropagation under activation checkpointing on the same hardware; if adjoint sharding's peak memory is not roughly 3X lower, the abstract's central memory claim is contradicted.","supporting_citations":[{"cited_title":"Adjoint sensitivity analysis for differential-algebraic equations: algorithms and software","cited_arxiv_id":null,"evidence_quote":"Supplies the adjoint sensitivity equations for recurrence relations (Proposition 1) from which the adjoint states are derived."}],"review_version":1}