{"id":"7e24ba8e-c63d-493a-ae6a-afa72e7a0de8","arxiv_id":"2412.08894","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Reshaping momentum tensors into near-square matrices before rank-1 factorization, plus a binary sign matrix for the first momentum, cuts optimizer memory by up to 96% versus Adafactor, SM3, and CAME on the reported benchmarks.","lead":"This paper introduces SMMF, an optimizer that compresses the momentum data used by Adam-style training into a pair of vectors plus a binary sign map, lowering optimizer memory on the tested models. The authors report competitive accuracy and perplexity on CNN and Transformer benchmarks, and include a regret-bound analysis to argue for convergence.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 4.1's regret bound is unsupported: Lemma E.5 replaces \\tilde{g}_m^2 with \\tilde{g}_v via an unproven proportionality, and Lemma E.7's zero-sum error property fails for the sign-restored first momentum, so compression errors are never bounded.","rationale":"The paper's central theoretical contribution is the regret bound in Theorem 4.1, stated as evidence that SMMF converges like AdamNC. Reading Appendix E in detail, the proof is internally inconsistent at two linked points. First, Lemma E.5 requires a bound relating the squared effective first-moment gradient to the effective second-moment gradient, but no such bound is proven; the text simply asserts positivity of both terms. Second, Lemma E.7, which is used to control the compression errors, establishes only that the total sum of the NNMF reconstruction error is zero for a non-negative matrix. SMMF's first momentum is not non-negative after sign restoration, so the lemma does not apply to e_m,t. Even for the second momentum, zero total error does not imply the elementwise or sum-of-squares bounds needed in the proof. These are not minor gaps: without a handle on e_m,t and e_v,t, the entire AdamNC-style chain from Reddi et al. cannot be imported. The reader's weakest assumption identifies exactly this issue, and I agree. The empirical campaign is broad and the code appears to be provided, but the paper explicitly advertises the regret analysis as a theoretical basis for SMMF's competitive capability. Since that analysis is unsupported, the REJECT verdict is appropriate. I do not see a need to adjust the verdict; the theory would need to be repaired before the central claim can be accepted.","tokens_in":42190,"tokens_out":5124,"duration_ms":53058,"concrete_test":"Run SMMF's exact compression/decompression (Algorithms 3-4) on a random matrix M with entries drawn from N(0,1), and compute the error E = decompress(compress(M)) - M. Check whether sum_{ij} E_{ij} = 0 and whether the ratio \\tilde{g}_m^2/\\tilde{g}_v (with \\tilde{g}_m = g + e_m/(1-beta_1), \\tilde{g}_v = g^2 + e_v/(1-beta_2)) stays bounded for a fixed constant across steps of the actual SMMF update on a simple convex problem, e.g., logistic regression on synthetic data. If the zero-sum equality fails for the first momentum or the ratio exceeds any reasonable zeta_1^2 (say 100) at any step, the proof step in Lemma E.5 is invalid and Theorem 4.1's bound does not follow.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The regret proof for Theorem 4.1 (Appendix E) is the sole theoretical justification for SMMF's claimed O(sqrt(T)) convergence. The argument breaks at Lemma E.5. After applying Cauchy-Schwarz and the AdamNC conditions, the proof must bound sum eta_t m_t^2/sqrt(v_t). At line (70) it replaces \\tilde{g}_m^2 with \\tilde{g}_v times constants, implicitly assuming \\tilde{g}_{m,j,i}^2 <= zeta_1^2 * \\tilde{g}_{v,j,i}, and justifies this only with 'since \\tilde{g}_{m,j,i}^2 and \\tilde{g}_{v,j,i} are positive.' Positivity does not imply proportionality, and no bound on e_{m,t} or e_{v,t} is given that would make such an inequality hold with a fixed zeta_1. Lemma E.7 only proves that the total sum of the error matrix from Adafactor-style NNMF is zero for non-negative matrices. SMMF's first momentum is sign-restored: it factorizes |M| and reapplies signs, so the error E = \\hat{M} - M is not the error of NNMF on a non-negative matrix; sum E_{ij} is generally nonzero and individual errors can be large even when the sum is zero. The v-momentum is non-negative, but zero-sum still does not bound \\tilde{g}_m^2 or \\tilde{g}_v elementwise, which the proof requires. Thus the AdamNC regret chain does not carry over to SMMF, and the advertised convergence guarantee is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SMMF, an optimizer that square-matricizes arbitrary-rank momentum tensors and compresses the first and second moments into row and column vectors by one NNMF step, storing a binary sign matrix for the first moment. It claims up to 96% memory reduction relative to Adafactor/CAME/SM3, comparable empirical performance across CNN and Transformer tasks, and a regret bound over convex objectives, Theorem 4.1, matching AdamNC's O(sqrt(T)) rate. The theoretical analysis is centered on bounding regret terms after absorbing NNMF decompression-compression errors e_m,t and e_v,t. The empirical section reports extensive experiments with optimizer memory, end-to-end memory, and task metrics.","tokens_in":42583,"tokens_out":10052,"duration_ms":104903,"significance":"If the claims were established, SMMF would fill a useful niche: a factorization-based optimizer that works for any tensor rank and comes with a regret analysis, something Adafactor and CAME lack. The square-matricization idea in Section 3.1 is simple and Theorems 3.1 and 3.2 appear correct. The empirical study is broad, includes many models and tasks, and code is provided. However, the central theoretical guarantee is not supported by the proof as written, and the formal memory-complexity statement ignores the sign matrix. The paper's significance is therefore currently limited to a practical memory-saving proposal with strong empirical results but unsupported asymptotic claims.","major_comments":[{"comment":"Equation (70) replaces sum_j beta^{T-j} tilde{g}_{m,j,i}^2 with zeta_1^2 sum_j beta^{T-j} tilde{g}_{v,j,i} on the justification 'since tilde{g}_{m,j,i}^2 and tilde{g}_{v,j,i} are positive.' Positivity alone does not imply an elementwise inequality of the form tilde{g}_{m,j,i}^2 <= zeta_1^2 tilde{g}_{v,j,i} for a fixed zeta_1. No bound on the compression-error terms e_{m,t} and e_{v,t} is given, and Lemma E.7's zero-sum property cannot be used for the first momentum because SMMF applies NNMF to |M| and restores signs, so the reconstruction error of M is not the error of NNMF on a non-negative matrix. Consequently the chain (69)-(75) does not prove Lemma E.5, and Theorem 4.1's O(sqrt(T)) regret bound is unsupported.","section":"Appendix E, Lemma E.5, Eq. (70)"},{"comment":"Independently of the zeta_1 issue, the proof drops the accumulated sum: Eq. (70) has sum_{t=1}^{T-1} eta_t m_{t,i}^2/sqrt(v_{t,i}) on the right-hand side, but Eq. (71) only keeps the newly bounded term. Unless an induction over T with a dominating inequality is supplied, the displayed inequalities do not imply the claimed bound for S_T = sum_{t=1}^T eta_t m_{t,i}^2/sqrt(v_{t,i}).","section":"Appendix E, Lemma E.5, Eqs. (70)-(71)"},{"comment":"The formal memory complexity O_M(hat n + hat m) and O_V(hat n + hat m) in Section 3 ignores the sign matrix S in {0,1}^{hat n x hat m} stored by Algorithm 4. For a square-matricized tensor with hat n approximately hat m approximately sqrt(N), the sign matrix alone is N bits, so the asymptotic optimizer-state size is Theta(N) bits, not O(sqrt(N)) as implied. Including the sign matrix may still give a constant-factor saving over the original 32-bit momentum, and Table 1 does count S in the measured memory, but the formal claim as stated is false and should be revised.","section":"Section 3 and Algorithm 4"},{"comment":"The theorem assumes the Reddi et al. conditions (a)-(b) 'can be satisfied by properly scheduling beta_{2,t}', but the paper never proves that the default schedule used in Algorithm 8, beta_{2,t}=1-t^gamma, satisfies them. In addition, condition (b) as printed has the factor (1 - beta_{2,j} tilde{g}_{v,j,i}), which is structurally different from the Reddi et al. condition (1 - beta_{2,j}) tilde{g}_{v,j,i}, and zeta_1 is not connected to any inequality in the conditions. Thus even if Lemma E.5 were repaired, the theorem would not be linked to the algorithm as implemented.","section":"Section 4, Theorem 4.1"}],"minor_comments":[{"comment":"The recurrence for hat v_t writes hat g_{m,j}; it should presumably be hat g_{v,j}.","section":"Appendix E, Lemma E.3, Eq. (40)"},{"comment":"The table reports NaN for GPT-2 with Adafactor; this is an unusual result and the caption or text should explain the configuration or the reason this cell is excluded.","section":"Table 3"},{"comment":"The notation 1_d is defined but never used; the algorithm actually uses 1_{hat n} and 1_{hat m}, which should be defined.","section":"Algorithm 4"},{"comment":"There are typos such as 'full-trining' in Section 5 and 'SCMF' in Appendix L tables, and a duplicated citation in Appendix J ('Il-Post (Landro et al. 2022a) (Landro et al. 2022a)').","section":"General presentation"}],"recommendation":"reject","confidential_remarks":"To the editor: the manuscript has a useful empirical idea and an unusually broad set of experiments, but the theoretical narrative is central to the paper's claims and the proof has load-bearing gaps. In addition, the theoretical memory-complexity statement is incorrect because it omits the sign matrix. I do not see a minor-revision path: either the regret claim must be substantially weakened or the algorithm/proof changed. The empirical results might support a shorter empirical paper at a different venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The practical core of this paper is real. Square-matricizing an arbitrary-rank momentum tensor into a near-square matrix and then applying the Adafactor-style rank-1 NNMF to both momenta, with a bitmask for the first momentum's sign, is a simple combination that does what it claims: it cuts optimizer memory substantially on CNNs and Transformers. The decompress-before-compress update order is a sensible difference from Adafactor, and the authors give credit to the prior factorization work they build on. The empirical campaign is unusually broad—image classification, object detection, full pre-training, fine-tuning, with code released—and the memory numbers in the tables are reported including the sign matrix, which is the honest way to do it. The limitations section also openly flags the sign-matrix overhead and the loss-spike behavior. That is a paper worth engaging with.\n\nWhere the paper falls down is the theory. Theorem 4.1 claims an AdamNC-style O(sqrt(T)) regret bound, but the proof in Appendix E does not establish it. Lemma E.5 replaces \\tilde{g}_m^2 with \\tilde{g}_v and justifies it only as 'since both are positive,' which is not a proportionality argument. Lemma E.7 shows that the total error of NNMF on a non-negative matrix sums to zero, but the first momentum is sign-restored; the error matrix there need not sum to zero, and even when it does, zero sum does not bound the elementwise error that the regret chain requires. So the central theoretical guarantee is unsupported. This is a load-bearing flaw for the paper as written, and the authors should either repair the argument (perhaps with a bounded-error assumption) or clearly present the convergence claim as a heuristic rather than a proof.\n\nThe memory complexity claim O(\\hat n + \\hat m) is also the kind of statement that will mislead casual readers, since the stored sign matrix is O(\\hat n \\hat m) bits. To the authors' credit, the text and experiments do account for this overhead, but the abstract and Section 3 headline the lower order. I would also want to see multiple seeds and a check on whether the baselines are equally tuned; the GPT-2 divergence for Adafactor, for example, suggests the comparison may not be apples-to-apples everywhere.\n\nWho this is for: anyone working on memory-efficient training or edge deployment of optimizers. The method is simple enough to be adopted quickly, and the empirical results make it a candidate baseline for future work. My recommendation: send it to peer review. The theory needs major revision or removal, but the empirical contribution and the algorithmic idea justify referee time.","headline":"The empirical optimizer is practical and the experimental campaign is serious, but the regret analysis in Theorem 4.1 does not go through as stated; the paper deserves review, not rejection out of hand.","tokens_in":43079,"tokens_out":1377,"would_cite":true,"duration_ms":17889,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SMMF claims that square-matricizing any rank-d momentum tensor and factorizing it into two vectors cuts optimizer memory by up to 96% while retaining AdamNC-comparable regret.","keywords":["square-matricization","memory-efficient optimizer","non-negative matrix factorization","momentum factorization","Adam","regret bound","adaptive learning rate"],"falsifier":"Run SMMF on a convex problem and record the compression errors $e_{m,t}$ and $e_{v,t}$ from Lemma E.3. If the substitution $\\tilde{g}_{m,t}^2 \\le \\zeta_1^2 \\tilde{g}_{v,t}$ used in Lemma E.5 fails at any step, or the error sum for the sign-restored first momentum is not zero, Theorem 4.1's derivation breaks; one can also check directly whether the empirical regret stays below the bound in Equation (2).","tokens_in":1586,"feed_emoji":"💾","tokens_out":1477,"duration_ms":76818,"temperature":0.7,"pith_summary":"This paper proposes SMMF, an optimizer that stores the first and second momentum tensors of Adam-style training in factorized form, reducing optimizer memory by up to 96% relative to existing memory-efficient optimizers such as Adafactor, CAME, and SM3. The central idea is square-matricization: any rank-d momentum tensor is reshaped into the matrix closest to a square, then factorized into two vectors by one non-negative matrix factorization, with a binary sign matrix preserving the signs of the first momentum. The paper proves a regret bound for SMMF matching the AdamNC bound, $O(\\sqrt{T})$ under two scheduling conditions, arguing that factorization need not sacrifice convergence. Experiments on CNNs and Transformers report competitive accuracy and perplexity with drastically lower memory.","feed_headline":"Square reshaping cuts Adam memory by up to 96 percent","feed_subtitle":"SMMF factorizes any-shape momentum tensors into two vectors and keeps convex regret on par with AdamNC.","key_machinery":"Square-matricization is the central mechanism: given a rank-$d$ tensor with $N$ elements, SMMF chooses $\\hat{n}, \\hat{m}$ with $\\hat{n}\\hat{m} = N$ that minimize $|\\hat{n} - \\hat{m}|$, reshapes the tensor into $\\mathbb{R}^{\\hat{n} \\times \\hat{m}}$, and applies one-time rank-1 NNMF to obtain row and column vectors whose outer product approximates the matrix. A decompression-to-compression loop reconstructs the momentum, updates it with the current gradient, and factorizes again; a binary sign matrix restores sign information for the first momentum. Theorems 3.1 and 3.2 establish that minimizing $|n-m|$ also minimizes $n+m$, which is what makes the square shape memory-optimal.","core_discovery":"On its own terms, SMMF claims that the memory cost of adaptive optimization can be reduced from $O(n_1 \\cdots n_d)$ to $O(\\hat{n} + \\hat{m})$ per momentum for tensors of any rank, by reshaping each momentum into the closest-to-square matrix $\\mathbb{R}^{\\hat{n} \\times \\hat{m}}$ and factorizing it into two vectors via non-negative matrix factorization. The first momentum's signs are kept in a 1-bit binary matrix, and the second momentum is non-negative and factorized directly. Theorem 4.1 states that, under the two scheduling conditions (a) and (b) imported from AdamNC, the regret satisfies the same upper bound as AdamNC, namely $O(\\sqrt{T})$ for convex objectives, with the explicit bound given in Equation (2). The paper takes this bound, together with experiments on CNNs and Transformers, as evidence that the factorization does not sacrifice convergence while saving up to 96% of optimizer memory.","pith_inferences":["If the compression-error gap in the proof is closed, the same square-matricization trick could likely combine with 8-bit quantization of the factor vectors to push memory even lower.","Because square reshaping deliberately discards the original coordinate layout, a stress test comparing SMMF against reshaping into a non-square shape would reveal whether the memory benefit is purely from factorization or from the square geometry itself.","SMMF's decompression-first scheme keeps the full current gradient before compression, so it may be particularly suited to fine-tuning tasks where small precise updates matter, such as instruction tuning of large language models.","The 1-bit sign matrix dominates memory for small tensors like biases; the paper itself notes that binary matrix factorization could reduce that overhead further."],"forward_implications":["Optimizer memory becomes $O(\\hat{n} + \\hat{m})$ per momentum, so high-rank tensors such as CNN kernels no longer force memory to grow with the product of all tensor dimensions.","Under the two AdamNC scheduling conditions, SMMF inherits an $O(\\sqrt{T})$ regret bound in the convex setting, matching Adam-family optimizers.","Transformer-base, which needs about 1.4 GiB with Adam, trains with roughly 0.8 GiB end-to-end under SMMF, bringing it closer to memory-constrained devices.","SMMF applies to vectors, matrices, and rank-$d$ tensors alike, unlike Adafactor and CAME which target matrix-shaped momentums.","Training time rises modestly, about 1.2 to 1.6 times Adam, while optimizer memory drops by up to 96%."],"supporting_citations":[{"why":"Provides the rank-1 NNMF factorization and the Adafactor baseline whose memory and performance SMMF is compared against.","marker":"Shazeer and Stern 2018"},{"why":"Supplies the AdamNC regret bound and the two conditions (a) and (b) that Theorem 4.1 imports.","marker":"Reddi, Kale, and Kumar 2019"},{"why":"Defines Adam, the adaptive optimizer whose momentum storage SMMF reduces, and the regret measure used in the analysis.","marker":"Kingma and Ba 2014"},{"why":"Gives the non-negative matrix factorization theory underlying SMMF's compression step.","marker":"Finesso and Spreij 2006"},{"why":"Defines SM3, a pattern-based memory-efficient optimizer used as a baseline.","marker":"Anil et al. 2019"},{"why":"Defines CAME, a confidence-guided memory-efficient optimizer used as a baseline.","marker":"Luo et al. 2023"}],"fun_headline_variants":["Square-matrix factorization cuts Adam memory by up to 96%","SMMF: any-rank momentum factorization with up to 96% memory cut","Memory-efficient optimizer: up to 96% less Adam memory, comparable convergence","Optimizer memory drops up to 96% with square reshaping"],"cache_read_input_tokens":45056,"weakest_assumption_plain":"The regret proof assumes that the errors introduced by compressing and decompressing the momentums are small enough that the AdamNC convergence argument still applies, but the proof never bounds those errors.","fun_headline_variants_meta":{"raw":{"variants":["Square-matrix factorization cuts Adam memory by up to 96%","SMMF: any-rank momentum factorization with up to 96% memory cut","Memory-efficient optimizer: up to 96% less Adam memory, comparable convergence","Optimizer memory drops up to 96% with square reshaping"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0017,"raw_usage":{"total_tokens":6750,"prompt_tokens":983,"completion_tokens":5767,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":599,"completion_tokens_details":{"reasoning_tokens":5688}},"tokens_in":599,"tokens_out":5767,"duration_ms":38806,"temperature":1.0,"reasoning_tokens":5688,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:27:59.551166+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run SMMF on a convex problem and record the compression errors $e_{m,t}$ and $e_{v,t}$ from Lemma E.3. If the substitution $\\tilde{g}_{m,t}^2 \\le \\zeta_1^2 \\tilde{g}_{v,t}$ used in Lemma E.5 fails at any step, or the error sum for the sign-restored first momentum is not zero, Theorem 4.1's derivation breaks; one can also check directly whether the empirical regret stays below the bound in Equation (2).","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the non-negative matrix factorization theory underlying SMMF's compression step."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines SM3, a pattern-based memory-efficient optimizer used as a baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines CAME, a confidence-guided memory-efficient optimizer used as a baseline."}],"review_version":1}