{"id":"01cc86ea-7da0-4a25-8e8e-237ffaa50fe3","arxiv_id":"2504.18590","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A multilevel scheme that alternates fine transformer training with two half-depth coarse models reaches the single-level training loss with 44 percent fewer FLOPs on one small language-model setup.","lead":"This paper tests a multilevel training scheme for transformer decoders, using an ODE view of the network to define cheaper half-depth versions and cycling their weights back. If the measured 44 percent FLOP saving holds outside this single small setup, it points to a practical way to cut language-model training cost.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 44% FLOPs saving may be caused by the high constant learning-rate phase on coarse models, not by the multilevel coarsening; a fine-only control with the same LR schedule is needed.","rationale":"The paper's central empirical claim is a 44% FLOPs saving to reach the single-level minimum loss. The reader's verdict is CONDITIONAL, with the weakest assumption being that the half-depth coarse models faithfully approximate the fine ODE discretization. My read agrees that the coarse-model construction lacks a fidelity guarantee, but I see a more decisive and testable confound: the multilevel algorithm differs from the baseline not only in model coarsening but also in the learning-rate schedule. The coarse models are trained with a constant learning rate of 1.2e-3, whereas the fine baseline uses cosine decay from 1.2e-3 to 1.2e-4. Since coarse parameters are copied back into the fine network, this gives the multilevel run an effective high-LR warm-start phase. A simple fine-only control with the same early high-LR phase would isolate whether the multilevel mechanism itself provides any benefit. The paper does not include such a control, so the attribution of the speedup to 'multilevel' is not yet established. This does not invalidate the paper; it identifies a missing experiment that a revision could supply. The verdict should remain CONDITIONAL, because the claim is plausible but currently under-supported by the experimental design. I agree partially with the reader: the coarse-fidelity issue is related, but the LR-schedule confound is more directly actionable and can be settled with one numerical experiment.","tokens_in":7197,"tokens_out":8741,"duration_ms":89852,"concrete_test":"Run the single-level fine model with the same data order and total optimizer steps, but replace the early coarse phase with fine steps at learning rate 1.2e-3 for the same number of steps (e.g., 35 steps after the first fine step), then apply the reported cosine decay. Compare training loss vs FLOPs. If this control reaches the target loss with equal or better FLOPs than multilevel, the acceleration is due to the LR schedule, not to the multilevel construction.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section 3, the multilevel method trains two half-depth coarse models with a constant learning rate of 1.2e-3, while the single-level baseline uses a cosine decay from 1.2e-3 to 1.2e-4 over 16k steps. Because the coarse parameters are prolongated into the fine network, the multilevel run effectively receives a high-learning-rate warm-start phase that the baseline does not. The reported comparison therefore conflates two changes: the multilevel layer coarsening (Eqs. 3-5) and a different LR schedule. No control is run in which the fine model is trained directly with the same early high-LR phase. Since the paper's strongest claim is about reducing FLOPs, this confound is load-bearing: if a fine-only run with the same schedule achieves the same loss-vs-FLOPs curve, the multilevel mechanism contributes nothing. Moreover, the coarse construction (dropping every other layer) has no ODE-consistency guarantee, so there is no reason to expect the speedup beyond what the schedule alone would give.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a multilevel training scheme for transformer decoders, motivated by an ODE interpretation of residual transformer blocks. Two half-depth coarse models are constructed by selecting even-indexed and odd-indexed transformer layers (Eqs. 3 and 5); after each fine-level optimization step, the coarse models are trained for 100 steps with a constant learning rate of 1.2e-3 and their parameters are prolongated back into the fine network with an averaging constant delta = 0.25 (Eq. 4). The method is tested on a 22M-parameter decoder trained with SGD on FineWeb-Edu, and the training loss is compared against a single-level baseline with a cosine decay schedule. The authors report that the multilevel approach reaches the same training loss in 16000 fine steps while reducing total FLOPs by 44%.","tokens_in":7399,"tokens_out":5007,"duration_ms":53570,"significance":"If the 44% FLOP saving is robust, the paper makes a useful practical contribution: it proposes a simple, operator-free coarsening for transformers and reports carefully calibrated FLOP estimates. The use of six seeds and a standard single-level baseline is commendable, as is the explicit treatment of FLOP accounting. However, the experimental evidence is currently narrow, and the main quantitative claim is confounded by a learning-rate schedule mismatch between the multilevel and single-level runs. The significance of the result therefore depends on the outcome of the controls proposed below.","major_comments":[{"comment":"The reported 44% FLOP reduction conflates the multilevel coarsening with a different learning-rate schedule. The single-level baseline uses 715 warm-up steps followed by cosine decay from 1.2e-3 to 1.2e-4, whereas both coarse models are trained at a constant learning rate of 1.2e-3. Since coarse parameters are prolongated into the fine network by Eq. (4), the multilevel run effectively receives a high-learning-rate warm-start phase that the baseline does not receive. A control experiment is needed in which the fine model is trained directly with the same effective learning-rate schedule, for example constant 1.2e-3 for the same number of steps or FLOPs before switching to cosine decay. Without this control, the speedup in Figure 2 cannot be attributed to the multilevel mechanism rather than to the schedule alone.","section":"Section 3, Hyperparameters and Results; Section 2.2, Eq. (4)"},{"comment":"The central claim is supported only by training loss. No held-out evaluation, validation perplexity, or downstream task metric is reported, and Section 4 explicitly acknowledges that robustness to test tasks remains to be checked. A training-loss-only comparison is insufficient for an acceleration claim in language modeling, since a method that reduces training loss faster may still generalize worse. Please add held-out loss or a standard evaluation metric for both the single-level and multilevel runs, ideally across the same six seeds.","section":"Section 3, Comparison metrics and Results; Section 4"},{"comment":"The construction of coarse models by taking every other layer has no stated consistency with the ODE in Eq. (2). In standard multigrid or multilevel ODE solvers, the coarse discretization approximates the same continuous trajectory with a larger step, but here the coarse network is a different, non-nested vector field: dropping every other layer changes the evolution defined by Eq. (1), and no approximation bound or transfer guarantee is provided. At minimum, the paper should provide empirical evidence that the coarse models remain aligned with the fine objective, such as measuring loss or gradient alignment between fine and coarse levels. Without this, the method's mechanism is unverified, which further strengthens the need for the schedule-control experiment.","section":"Section 2.2, Eqs. (3)-(5)"}],"minor_comments":[{"comment":"The description of the coarse-model learning-rate schedule is ambiguous: the text says the hyperparameters are identical to the fine model except for a constant learning rate, but it does not state whether the coarse models also receive the 715 warm-up steps or how the fine learning rate behaves during the 35 coarse cycles. Please specify the exact schedule for both levels.","section":"Section 3, Multilevel hyperparameters"},{"comment":"The prolongation rule for the odd-indexed coarse model is only described as 'symmetric' to Eq. (4). For reproducibility, write out the complete update for both even- and odd-indexed coarse models.","section":"Section 2.2, Eq. (4)"},{"comment":"The annotation 'Multilevel FLOPS to reach Single Level Min Loss: -44%' appears only in the figure; the main text should define how this number is computed and state the corresponding FLOP totals for both methods.","section":"Figure 2"},{"comment":"The FLOP estimate relies on a factor of 3 forward passes per training step, cited to a blog report; please give the exact reference and, if possible, a sensitivity check of the reported saving to this factor.","section":"Section 3, Comparison metrics"}],"recommendation":"major_revision","confidential_remarks":"The main result is currently confounded by the learning-rate schedule difference, but this is fixable with a control experiment. I saw no evidence of circularity: the baseline is a standard single-level method, not a target derived from the proposed method. The self-citations [16,17] are relevant background and do not unduly affect the assessment. The absence of a comparison with [30] is a limitation but not fatal, since the paper's claim is relative to standard training. I would ask the authors to provide the schedule control, held-out evaluation, and a clearer statement of the coarse-level construction before accepting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuinely new but thinly validated multilevel trick, and the 44% FLOP saving is probably inflated by a learning-rate schedule difference. Worth a serious referee, but the schedule-matched control is the missing experiment.\n\nWhat's actually new: the even/odd subsampling of transformer blocks, without any width/depth transfer operators, is a construction I don't recall from the ResNet multilevel papers. Training two complementary half-depth coarse models and averaging the prolongation is a sensible way to avoid the asymmetry of touching only even layers. The FLOP accounting is honest and reproducible, with six seeds and a standard single-level baseline. The paper is clearly written and does not oversell.\n\nSoft spots, in order. First, the LR confound. Coarse models train at constant 1.2e-3, while the fine baseline uses cosine decay to 1.2e-4. For the first 35 cycles, the multilevel run gives the shared parameters a long high-LR phase that the baseline never has. No control runs on the fine model alone with the same early schedule, so the reported gain conflates coarsening with schedule. This is load-bearing, not a nit.\n\nSecond, the evaluation is training loss only, on one 22M decoder with SGD. No test loss, no held-out evaluation, no comparison to [30], which the authors acknowledge. No code or data released. Given the short format, these are moderate, not fatal, but they limit the claim to this specific setting.\n\nThird, the ODE justification is thin: dropping every other layer has no consistency guarantee, so the method reduces to a heuristic multilevel scheme. That's fine for an empirical paper, but it means the ODE-interpretation framing adds little.\n\nThe authors are honest about most of this; the discussion explicitly calls for test evaluation and more experiments. The comparison is not circular and the paper is self-contained. I'd accept it for peer review and require the schedule-matched control, test evaluation, and ideally a comparison with [30] or at least a second architecture. As is, treat 44% as promising but unconfirmed.","headline":"A clean but thin empirical paper; the 44% FLOP saving is likely inflated by a learning-rate schedule difference and needs a control before the claim is credible.","tokens_in":7963,"tokens_out":4377,"would_cite":false,"duration_ms":40457,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Interleaving training with two half-depth copies of a transformer reaches the same loss with 44% fewer FLOPs.","keywords":["multilevel optimization","ODE transformers","training acceleration","transformer decoder","depth coarsening","layer subsampling","FLOP reduction","SGD training"],"falsifier":"Run the identical 22M-parameter experiment but with the two coarse models formed from a random permutation of the 12 transformer blocks instead of the exact even and odd subsets, keeping the same δ=0.25 and schedule. If the 44% FLOP saving at the 16,000-step mark persists, then layer-wise fidelity to the ODE discretization is not what drives the acceleration, and the paper's explanation would need revision.","tokens_in":6981,"feed_emoji":"⚡","tokens_out":7912,"duration_ms":71819,"temperature":0.7,"pith_summary":"This paper tries to establish that the cost of training a transformer decoder can be cut by treating its depth as a discretization parameter. It builds two coarse networks by keeping only the even-indexed or only the odd-indexed transformer blocks, trains those cheap half-depth models, and periodically mixes their updated weights back into the full network with an averaging constant of δ=0.25. On a 22-million-parameter decoder trained with SGD on a slice of FineWeb-Edu, the multilevel run reaches the same training loss as the standard single-level run in the same 16,000 fine-level steps while using 44% fewer FLOPs. The paper's own conclusion states that the experimental setting is limited and calls for more experiments and theory, especially about the interaction between fine- and coarse-level optimizer momentum, so the 44% figure is a first validation rather than a general guarantee.","feed_headline":"Half-depth helpers cut transformer training FLOPs by 44%","feed_subtitle":"Two cheaper every-other-layer models are trained alongside the full network and mixed back in, saving compute.","key_machinery":"The central object is the ODE interpretation of a transformer layer: $x_{t+1} = x_t + F(x_t, \\theta_t)$ is Euler's method with step $\\Delta t=1$ for $\\frac{dx(t)}{dt}=F(x(t),\\theta(t))$. Coarse levels are obtained by halving the number of layers: coarse model 1 takes the even-indexed fine layers ($\\theta^H_i = \\theta^h_{2i}$), coarse model 2 the odd-indexed ones ($\\theta^H_i = \\theta^h_{2i-1}$), with no correction operator. After coarse training, parameters are prolonged back by $\\tilde{\\theta}^h_{2i}=\\tilde{\\theta}^H_i$ and $\\tilde{\\theta}^h_{2i+1}=(1-\\delta)\\theta^h_{2i+1}+\\delta\\tilde{\\theta}^H_i$ with $\\delta=0.25$. This simple subsampling-and-averaging transfer is what carries the acceleration: half-depth models cost roughly half a forward pass, and their updates nudge the full model without gradient sharing between levels.","core_discovery":"The central claim is that a transformer-decoder can be trained faster by alternating the full network with two deliberately cheap, half-depth copies that are exact subsets of its layers. Viewing a stack of transformer blocks as the Euler discretization with step Δt=1 of an ODE, the paper makes a coarser level of the same ODE by halving the number of blocks: one coarse model uses the even-indexed blocks, the other the odd-indexed blocks. After a block of coarse training, the updated weights are copied back into the corresponding fine layers and mixed with the untouched opposite layers using the averaging rule $\\tilde{\\theta}^h_{2i}=\\tilde{\\theta}^H_i$ and $\\tilde{\\theta}^h_{2i+1}=(1-\\delta)\\theta^h_{2i+1}+\\delta\\tilde{\\theta}^H_i$ with $\\delta=0.25$. On a 22,368,512-parameter decoder trained with SGD on a slice of FineWeb-Edu for 16,000 steps, the multilevel schedule reaches the same training loss as standard training while reducing the total FLOP count by 44%. The result is presented as a first validation that ODE-based depth coarsening works for transformers, with the coarse level needing no learned operators or gradient sharing.","pith_inferences":["The even/odd split is a discretization choice; coarsening by every third layer or by random subsets would test whether the transfer is tied to the ODE structure or simply to exposure to extra small-network updates.","The momentum interaction the paper flags suggests a concrete next experiment: use AdamW for both levels and reset the coarse optimizer state on prolongation, which would tell whether the method's acceleration survives the optimizer used in current practice.","Since the shared input and output layers are not coarsened, the relative FLOP saving grows with depth; the method may become more attractive for very deep decoders than for the 12-layer testbed.","The paper compares only training loss; evaluating the trained checkpoints on held-out text or downstream tasks would show whether the FLOP saving comes with any generalization cost."],"forward_implications":["The 44% FLOP saving is measured at the same fine-level step count as single-level training, so the acceleration is a real compute reduction, not just faster wall-clock due to smaller batches.","Because the coarse models are built purely by subsampling existing layers, no learned expansion or projection operators are needed, which makes the method simple to implement on top of an existing transformer training loop.","The coarse levels are used only during the first 35 fine steps, so the practical benefit is concentrated in the warm-up phase of training.","Matching the single-level loss in the same number of fine steps while spending fewer FLOPs implies that, in the SGD regime tested, coarse-model updates transfer useful information to the fine network rather than merely adding extra compute."],"supporting_citations":[{"why":"Supplies the ODE formulation of transformer decoders that motivates halving depth as a coarser discretization.","marker":"[18]"},{"why":"Introduces the multilevel training scheme for deep residual networks that this paper adapts to transformers.","marker":"[15]"},{"why":"Earlier multilevel minimization for ResNets that grounds the coarse-level construction and prolongation strategy.","marker":"[10]"},{"why":"Recent multilevel transformer training framework whose averaging constant δ and baseline the paper follows and compares against.","marker":"[30]"},{"why":"Provides the FineWeb-Edu dataset used in the experiments.","marker":"[22]"},{"why":"Provides the three-forward-pass-per-training-step FLOP estimate used for the fair comparison.","marker":"[14]"},{"why":"Defines the GPT-2 tokenizer, architecture style, and learning-rate schedule the experiments adopt.","marker":"[24]"}],"fun_headline_variants":["Multilevel training cuts transformer FLOPs by 44%","ODE-based coarsening speeds up transformer training","Half-depth copies cut transformer training compute by 44%","Alternating fine and coarse levels saves 44% FLOPs","Cheaper coarse levels trim transformer FLOPs by 44%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a coarse model formed by keeping every other transformer layer, with no correction operator, is faithful enough to the full model that its SGD updates mixed in with δ=0.25 accelerate training; no approximation bound or transfer guarantee is given.","fun_headline_variants_meta":{"raw":{"variants":["Multilevel training cuts transformer FLOPs by 44%","ODE-based coarsening speeds up transformer training","Half-depth copies cut transformer training compute by 44%","Alternating fine and coarse levels saves 44% FLOPs","Cheaper coarse levels trim transformer FLOPs by 44%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000471,"raw_usage":{"total_tokens":2293,"prompt_tokens":846,"completion_tokens":1447,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":462,"completion_tokens_details":{"reasoning_tokens":1363}},"tokens_in":462,"tokens_out":1447,"duration_ms":8781,"temperature":1.0,"reasoning_tokens":1363,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:41:56.496020+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the identical 22M-parameter experiment but with the two coarse models formed from a random permutation of the 12 transformer blocks instead of the exact even and odd subsets, keeping the same δ=0.25 and schedule. If the 44% FLOP saving at the 16,000-step mark persists, then layer-wise fidelity to the ODE discretization is not what drives the acceleration, and the paper's explanation would need revision.","supporting_citations":[{"cited_title":"ODE Transformer: An Ordinary Differential Equation-Inspired Model for Sequence Generation","cited_arxiv_id":"2203.09176","evidence_quote":"Supplies the ODE formulation of transformer decoders that motivates halving depth as a coarser discretization."},{"cited_title":"Kopaniˇcáková and R","cited_arxiv_id":null,"evidence_quote":"Introduces the multilevel training scheme for deep residual networks that this paper adapts to transformers."},{"cited_title":"Gaedke-Merzhäuser, A","cited_arxiv_id":null,"evidence_quote":"Earlier multilevel minimization for ResNets that grounds the coarse-level construction and prolongation strategy."},{"cited_title":"A Multi-Level Framework for Accelerating Training Transformer Models","cited_arxiv_id":"2404.07999","evidence_quote":"Recent multilevel transformer training framework whose averaging constant δ and baseline the paper follows and compares against."},{"cited_title":"Penedo, H","cited_arxiv_id":null,"evidence_quote":"Provides the FineWeb-Edu dataset used in the experiments."}],"review_version":1}