{"id":"bf0988a7-9139-4781-8fb6-014beb1368e6","arxiv_id":"2512.07112","paper_version":2,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"FOAM folds Adam optimizer states into block-wise gradient means with residual correction, cutting memory overhead by up to 90% while matching vanilla Adam convergence rates under standard non-convex assumptions.","lead":"FOAM compresses optimizer states during LLM training by replacing per-parameter moments with block-wise gradient means plus a residual correction term. This targets the memory bottleneck of Adam-style optimizers while claiming to preserve convergence speed.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Convergence equivalence hinges on whether block-wise mean + residual exactly preserves Adam's moment dynamics without extra bias/variance terms","rationale":"The reader's weakest assumption is precisely the load-bearing step for the theoretical claim. Because the full proof is now available, the concrete test above directly checks whether that assumption holds inside the paper's own derivation rather than relying on the abstract alone.","tokens_in":1664,"tokens_out":336,"duration_ms":27245,"concrete_test":"Extract the exact statement of the convergence theorem (likely Theorem 4.x) and the definition of the FOAM moment update (Eq. around 3.x). Substitute the block-mean operator into the standard Adam recurrence, expand the error term introduced by the mean, and check whether the proof still closes with the same rate; if an extra O(1/block-size) bias term survives in the final bound, the equivalence claim fails.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The headline theoretical claim requires that FOAM's first- and second-moment estimates remain statistically equivalent to full Adam (up to terms that do not change the O(1/sqrt(T)) rate). The block-wise mean replaces per-parameter moments with a single scalar per block; the residual correction is claimed to recover the lost information. If this correction is only first-order or leaves a non-vanishing bias in the exponential moving averages, the standard Adam proof (which relies on unbiased or bounded-variance gradient estimates) no longer applies directly. The paper must show that any residual error is absorbed into the existing constants or decays fast enough; otherwise the rate guarantee is invalidated.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper proposes FOAM, a method that compresses Adam optimizer states for LLM training via block-wise gradient means plus a residual correction term. It claims this yields convergence rates equivalent to vanilla Adam in standard non-convex settings, eliminates up to 90% of optimizer-state memory overhead, accelerates convergence, and remains compatible with other memory-efficient optimizers.","tokens_in":1816,"tokens_out":467,"duration_ms":34344,"significance":"If the equivalence claim is rigorously supported, FOAM would provide a practical route to lower memory footprints during large-model training while retaining Adam-level guarantees and throughput. The reported compatibility with existing methods and availability of code are positive indicators of utility.","major_comments":[{"comment":"The central theoretical claim (convergence equivalence to Adam) rests on the assertion that block-wise means plus residual correction preserve the first- and second-moment dynamics without introducing non-vanishing bias or extra variance terms that would invalidate the standard O(1/sqrt(T)) rate. No derivation sketch, bias bound, or variance analysis is supplied to confirm this; the residual correction is described only at a high level.","section":"Theoretical analysis section (around the statement of equivalence to Adam)"},{"comment":"§4 (experiments): reported memory reductions and convergence speed-ups are presented without error bars, number of independent runs, or explicit data-exclusion rules. This makes it impossible to judge whether the 90% memory saving and faster convergence are statistically reliable or sensitive to particular hyper-parameter choices.","section":"§4"}],"minor_comments":[{"comment":"Abstract: the phrase 'accelerates convergence' should be qualified by the baseline and the metric (steps to target loss, wall-clock time, etc.).","section":"Abstract"},{"comment":"Notation for the residual correction term is introduced without an explicit equation number or clear definition of the block partitioning; readers must infer the exact update rule.","section":"Method section"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a reasonable fit for the journal but would benefit from a more thorough discussion of related block-wise or low-rank optimizer work to clarify novelty."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for their constructive comments and for recognizing the potential utility of FOAM. We address each major comment below and will revise the manuscript to strengthen the presentation of both the theoretical analysis and the experimental results.","responses":[{"response":"We agree that a more explicit derivation would improve clarity. The current manuscript states the equivalence under standard non-convex assumptions but presents the residual correction at a high level in the main text. In the revision we will add a concise proof sketch to the theoretical analysis section that (i) shows the block-wise mean operator introduces a bias term whose expectation vanishes under the standard bounded-gradient assumption, (ii) bounds the additional variance introduced by the residual correction, and (iii) demonstrates that these terms do not alter the O(1/sqrt(T)) rate obtained by vanilla Adam. The full proof will remain in the appendix for completeness.","revision_made":"yes","referee_comment":"[Theoretical analysis section (around the statement of equivalence to Adam)] The central theoretical claim (convergence equivalence to Adam) rests on the assertion that block-wise means plus residual correction preserve the first- and second-moment dynamics without introducing non-vanishing bias or extra variance terms that would invalidate the standard O(1/sqrt(T)) rate. No derivation sketch, bias bound, or variance analysis is supplied to confirm this; the residual correction is described only at a high level."},{"response":"We acknowledge that the experimental section would benefit from additional statistical reporting. In the revised manuscript we will (i) report all key metrics with error bars computed over at least three independent random seeds, (ii) explicitly state the number of runs and the random-seed protocol, (iii) describe any data-exclusion or outlier-handling rules, and (iv) add a short paragraph discussing sensitivity to the block-size hyper-parameter.","revision_made":"yes","referee_comment":"[§4] §4 (experiments): reported memory reductions and convergence speed-ups are presented without error bars, number of independent runs, or explicit data-exclusion rules. This makes it impossible to judge whether the 90% memory saving and faster convergence are statistically reliable or sensitive to particular hyper-parameter choices."}],"tokens_in":1304,"tokens_out":476,"duration_ms":32842,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main thing to know is that FOAM folds optimizer states into blocks using per-block gradient means and adds a residual correction to recover information, claiming this keeps convergence rates the same as vanilla Adam while dropping 90% of the state memory. They also say it speeds things up and works alongside other memory tricks. Code is out on GitHub, which is helpful for checking the claims directly. This combination of blocked means with an explicit correction term looks distinct from the SVD, projection, and freezing baselines they mention, so there is a genuine new angle in the memory-efficient optimizer space. If the empirical gains hold across models, it could let teams fit bigger LLMs on existing hardware without major slowdowns. The practical side is straightforward and addresses a real scaling pain point. The soft spot sits in the theory. The equivalence claim rests on the residual term keeping the first- and second-moment estimates close enough to full Adam that no extra bias or variance invalidates the standard non-convex rate. The abstract asserts this but gives no derivation sketch or bound on the approximation error, so it is not obvious whether the block-wise replacement preserves the necessary properties or introduces terms the proof cannot absorb. The stress-test note flags exactly this issue. Without seeing the full steps or error analysis, the headline theoretical result stays unverified. Experiments are described as matching or beating baselines, but details on run-to-run variance, data splits, and exact controls would strengthen the case. This paper is aimed at researchers working on LLM training efficiency and memory-constrained optimization. It has enough of a concrete technique and released code to deserve a serious referee, even if the theory section needs tightening and more explicit checks on the moment preservation. I would send it out for peer review rather than desk reject.","headline":"FOAM cuts optimizer memory via block-wise means plus residual correction but the convergence equivalence needs a clear derivation to hold up.","tokens_in":2305,"tokens_out":421,"would_cite":false,"duration_ms":31525,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":{"model":"grok-4.3","evidence":[{"relation":"unclear","rs_module":"IndisputableMonolith/Cost/FunctionalEquation.lean","rs_theorem":"washburn_uniqueness_aczel","paper_passage":"FOAM replaces each group of 2^l consecutive elements with their mean value... Mt = ~Mt E(l) + Rt, Vt = ~Vt E(l) + R^2_t"}],"headline":"FOAM block-folding + residual correction for Adam states has no structural overlap with RS J-cost or distinction forcing","alignment":"orthogonal","rationale":"The paper's core machinery (block-wise gradient averaging via A(l)/E(l), residual Rt = Gt − Gt P(l), unfolded moments Mt/Vt) is a practical compression technique for stochastic optimizers. RS derives J(x), φ-ladders, 8-tick periodicity and constants from a single distinction with zero adjustable parameters; the two domains share no theorems, cost functions or structural primitives.","tokens_in":65478,"confidence":"high","tokens_out":229,"duration_ms":17176,"cache_read_input_tokens":32896,"cache_creation_input_tokens":0},"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"FOAM compresses Adam optimizer states via block-wise gradient means and residual corrections to match full convergence while cutting memory overhead by up to 90 percent.","keywords":["memory-efficient optimizers","LLM training","Adam optimizer","optimizer state compression","block-wise approximation","residual correction","non-convex convergence","training memory reduction"],"falsifier":"Measure the number of steps or wall-clock time required for FOAM and vanilla Adam to reach the same validation loss on a fixed benchmark model; if the curves diverge beyond statistical noise while using identical learning rates and batch sizes, the equivalence claim is refuted.","tokens_in":2565,"feed_emoji":"💾","tokens_out":720,"duration_ms":73816,"temperature":0.7,"pith_summary":"The paper presents FOAM as a way to train large language models with far less memory when using memory-heavy optimizers like Adam. It replaces the full first and second moment buffers with averages computed over blocks of parameters and adds a residual term that restores the information discarded by the averaging. The authors show that this construction preserves the standard convergence rate of Adam under typical non-convex stochastic optimization assumptions. The practical result is that the dominant memory cost of optimizer states drops sharply without requiring extra projection matrices or freezing weights. If the claim holds, practitioners can train larger models or longer schedules on the same hardware while keeping training dynamics and final quality comparable to the uncompressed baseline.","feed_headline":"FOAM cuts LLM optimizer memory by 90% while matching Adam convergence","feed_subtitle":"Block-wise gradient means plus residual corrections keep convergence rates identical to full Adam while freeing up substantial GPU memory.","key_machinery":"Block-wise gradient mean with residual correction: the mechanism that approximates Adam's first and second moments by averaging gradients inside each parameter block and subtracting the induced approximation error to keep the update direction faithful.","core_discovery":"FOAM folds optimizer states by replacing per-parameter moment estimates with block-wise gradient means and recovers lost detail through an explicit residual correction. Under standard non-convex optimization settings the method attains the same convergence rate as vanilla Adam. Experiments confirm that the approach eliminates up to 90 percent of optimizer-state memory, accelerates convergence in wall-clock time, and remains compatible with other memory-saving techniques while matching or exceeding the performance of both full-rank and prior compressed baselines.","pith_inferences":["The same folding pattern could be applied to other adaptive methods that maintain per-parameter statistics, such as RMSprop or Lion.","Adaptive block sizing per layer or attention head might further reduce memory while controlling approximation error.","In distributed settings the compressed states could lower communication volume when synchronizing optimizer buffers across nodes."],"forward_implications":["Optimizer-state memory becomes a much smaller fraction of total training footprint, allowing larger batch sizes or model scales on fixed hardware.","Training dynamics remain governed by the same non-convex rate guarantees as Adam, so hyper-parameter schedules transfer with little adjustment.","The method stacks directly with other memory reducers such as low-rank adapters or quantization for additive savings.","Wall-clock throughput rises because reduced memory traffic and fewer state updates free up compute resources."],"fun_headline_variants":["FOAM matches full Adam convergence while cutting optimizer memory 90%","Residual corrections in FOAM recover detail after 90% state compression","FOAM folds blocks to match Adam rates using only 10% of state memory","LLM training memory drops 90% with FOAM without losing Adam performance"],"cache_read_input_tokens":64,"weakest_assumption_plain":"The combination of block-wise averaging and residual correction must preserve enough gradient information that the standard Adam convergence proof continues to apply without new bias or variance terms that would change the rate.","fun_headline_variants_meta":{"raw":{"variants":["FOAM matches full Adam convergence while cutting optimizer memory 90%","Residual corrections in FOAM recover detail after 90% state compression","FOAM folds blocks to match Adam rates using only 10% of state memory","LLM training memory drops 90% with FOAM without losing Adam performance"]},"model":"grok-4.3","cost_usd":0.007565,"raw_usage":{"total_tokens":3459,"prompt_tokens":651,"num_sources_used":0,"completion_tokens":77,"cost_in_usd_ticks":75649500,"prompt_tokens_details":{"text_tokens":651,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2731,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":651,"tokens_out":77,"duration_ms":40137,"temperature":1.0,"reasoning_tokens":2731,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-05-17T00:43:16.536911+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Measure the number of steps or wall-clock time required for FOAM and vanilla Adam to reach the same validation loss on a fixed benchmark model; if the curves diverge beyond statistical noise while using identical learning rates and batch sizes, the equivalence claim is refuted.","supporting_citations":[],"review_version":1}