Pith. sign in

REVIEW 3 major objections 4 minor 30 references

A multilevel approach to accelerate the training of Transformers

T0 review · 3 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Interleaving training with two half-depth copies of a transformer reaches the same loss with 44% fewer FLOPs.

desk verdict A clean but thin empirical paper; the 44% FLOP saving is likely inflated by a learning-rate schedule difference and needs a control before the claim is credible. read the letter →

arxiv 2504.18590 v1 pith:UI6BJPZM submitted 2025-04-24 cs.LG cs.AImath.OC

Guillaume Lauga (OCKHAM) , Maël Chaumette (OCKHAM) , Edgar Desainte-Maréville (OCKHAM) , Étienne Lasalle (OCKHAM) , Arthur Lebeurrier (OCKHAM) This is my paper · ORCID
classification cs.LGcs.AImath.OC
keywords multileveloptimizationODEtransformerstrainingaccelerationtransformerdecoderdepthcoarseninglayersubsamplingFLOPreductionSGD
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the cost of training a transformer decoder can be cut by treating its depth as a discretization parameter. It builds two coarse networks by keeping only the even-indexed or only the odd-indexed transformer blocks, trains those cheap half-depth models, and periodically mixes their updated weights back into the full network with an averaging constant of δ=0.25. On a 22-million-parameter decoder trained with SGD on a slice of FineWeb-Edu, the multilevel run reaches the same training loss as the standard single-level run in the same 16,000 fine-level steps while using 44% fewer FLOPs. The paper's own conclusion states that the experimental setting is limited and calls for more experiments and theory, especially about the interaction between fine- and coarse-level optimizer momentum, so the 44% figure is a first validation rather than a general guarantee.

What carries the argument

The central object is the ODE interpretation of a transformer layer: $x_{t+1} = x_t + F(x_t, \theta_t)$ is Euler's method with step $\Delta t=1$ for $\frac{dx(t)}{dt}=F(x(t),\theta(t))$. Coarse levels are obtained by halving the number of layers: coarse model 1 takes the even-indexed fine layers ($\theta^H_i = \theta^h_{2i}$), coarse model 2 the odd-indexed ones ($\theta^H_i = \theta^h_{2i-1}$), with no correction operator. After coarse training, parameters are prolonged back by $\tilde{\theta}^h_{2i}=\tilde{\theta}^H_i$ and $\tilde{\theta}^h_{2i+1}=(1-\delta)\theta^h_{2i+1}+\delta\tilde{\theta}^H_i$ with $\delta=0.25$. This simple subsampling-and-averaging transfer is what carries the acceleration: half-depth models cost roughly half a forward pass, and their updates nudge the full model without gradient sharing between levels.

What would settle it

Run the identical 22M-parameter experiment but with the two coarse models formed from a random permutation of the 12 transformer blocks instead of the exact even and odd subsets, keeping the same δ=0.25 and schedule. If the 44% FLOP saving at the 16,000-step mark persists, then layer-wise fidelity to the ODE discretization is not what drives the acceleration, and the paper's explanation would need revision.

Watch

Extended reading notes

Core claim

The central claim is that a transformer-decoder can be trained faster by alternating the full network with two deliberately cheap, half-depth copies that are exact subsets of its layers. Viewing a stack of transformer blocks as the Euler discretization with step Δt=1 of an ODE, the paper makes a coarser level of the same ODE by halving the number of blocks: one coarse model uses the even-indexed blocks, the other the odd-indexed blocks. After a block of coarse training, the updated weights are copied back into the corresponding fine layers and mixed with the untouched opposite layers using the averaging rule $\tilde{\theta}^h_{2i}=\tilde{\theta}^H_i$ and $\tilde{\theta}^h_{2i+1}=(1-\delta)\theta^h_{2i+1}+\delta\tilde{\theta}^H_i$ with $\delta=0.25$. On a 22,368,512-parameter decoder trained with SGD on a slice of FineWeb-Edu for 16,000 steps, the multilevel schedule reaches the same training loss as standard training while reducing the total FLOP count by 44%. The result is presented as a first validation that ODE-based depth coarsening works for transformers, with the coarse level needing no learned operators or gradient sharing.

Load-bearing premise

The method assumes that a coarse model formed by keeping every other transformer layer, with no correction operator, is faithful enough to the full model that its SGD updates mixed in with δ=0.25 accelerate training; no approximation bound or transfer guarantee is given.

Editorial extensions

If this is right

  • The 44% FLOP saving is measured at the same fine-level step count as single-level training, so the acceleration is a real compute reduction, not just faster wall-clock due to smaller batches.
  • Because the coarse models are built purely by subsampling existing layers, no learned expansion or projection operators are needed, which makes the method simple to implement on top of an existing transformer training loop.
  • The coarse levels are used only during the first 35 fine steps, so the practical benefit is concentrated in the warm-up phase of training.
  • Matching the single-level loss in the same number of fine steps while spending fewer FLOPs implies that, in the SGD regime tested, coarse-model updates transfer useful information to the fine network rather than merely adding extra compute.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The even/odd split is a discretization choice; coarsening by every third layer or by random subsets would test whether the transfer is tied to the ODE structure or simply to exposure to extra small-network updates.
  • The momentum interaction the paper flags suggests a concrete next experiment: use AdamW for both levels and reset the coarse optimizer state on prolongation, which would tell whether the method's acceleration survives the optimizer used in current practice.
  • Since the shared input and output layers are not coarsened, the relative FLOP saving grows with depth; the method may become more attractive for very deep decoders than for the 12-layer testbed.
  • The paper compares only training loss; evaluating the trained checkpoints on held-out text or downstream tasks would show whether the FLOP saving comes with any generalization cost.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes a multilevel training scheme for transformer decoders, motivated by an ODE interpretation of residual transformer blocks. Two half-depth coarse models are constructed by selecting even-indexed and odd-indexed transformer layers (Eqs. 3 and 5); after each fine-level optimization step, the coarse models are trained for 100 steps with a constant learning rate of 1.2e-3 and their parameters are prolongated back into the fine network with an averaging constant delta = 0.25 (Eq. 4). The method is tested on a 22M-parameter decoder trained with SGD on FineWeb-Edu, and the training loss is compared against a single-level baseline with a cosine decay schedule. The authors report that the multilevel approach reaches the same training loss in 16000 fine steps while reducing total FLOPs by 44%.

Significance. If the 44% FLOP saving is robust, the paper makes a useful practical contribution: it proposes a simple, operator-free coarsening for transformers and reports carefully calibrated FLOP estimates. The use of six seeds and a standard single-level baseline is commendable, as is the explicit treatment of FLOP accounting. However, the experimental evidence is currently narrow, and the main quantitative claim is confounded by a learning-rate schedule mismatch between the multilevel and single-level runs. The significance of the result therefore depends on the outcome of the controls proposed below.

major comments (3)
  1. [Section 3, Hyperparameters and Results; Section 2.2, Eq. (4)] The reported 44% FLOP reduction conflates the multilevel coarsening with a different learning-rate schedule. The single-level baseline uses 715 warm-up steps followed by cosine decay from 1.2e-3 to 1.2e-4, whereas both coarse models are trained at a constant learning rate of 1.2e-3. Since coarse parameters are prolongated into the fine network by Eq. (4), the multilevel run effectively receives a high-learning-rate warm-start phase that the baseline does not receive. A control experiment is needed in which the fine model is trained directly with the same effective learning-rate schedule, for example constant 1.2e-3 for the same number of steps or FLOPs before switching to cosine decay. Without this control, the speedup in Figure 2 cannot be attributed to the multilevel mechanism rather than to the schedule alone.
  2. [Section 3, Comparison metrics and Results; Section 4] The central claim is supported only by training loss. No held-out evaluation, validation perplexity, or downstream task metric is reported, and Section 4 explicitly acknowledges that robustness to test tasks remains to be checked. A training-loss-only comparison is insufficient for an acceleration claim in language modeling, since a method that reduces training loss faster may still generalize worse. Please add held-out loss or a standard evaluation metric for both the single-level and multilevel runs, ideally across the same six seeds.
  3. [Section 2.2, Eqs. (3)-(5)] The construction of coarse models by taking every other layer has no stated consistency with the ODE in Eq. (2). In standard multigrid or multilevel ODE solvers, the coarse discretization approximates the same continuous trajectory with a larger step, but here the coarse network is a different, non-nested vector field: dropping every other layer changes the evolution defined by Eq. (1), and no approximation bound or transfer guarantee is provided. At minimum, the paper should provide empirical evidence that the coarse models remain aligned with the fine objective, such as measuring loss or gradient alignment between fine and coarse levels. Without this, the method's mechanism is unverified, which further strengthens the need for the schedule-control experiment.
minor comments (4)
  1. [Section 3, Multilevel hyperparameters] The description of the coarse-model learning-rate schedule is ambiguous: the text says the hyperparameters are identical to the fine model except for a constant learning rate, but it does not state whether the coarse models also receive the 715 warm-up steps or how the fine learning rate behaves during the 35 coarse cycles. Please specify the exact schedule for both levels.
  2. [Section 2.2, Eq. (4)] The prolongation rule for the odd-indexed coarse model is only described as 'symmetric' to Eq. (4). For reproducibility, write out the complete update for both even- and odd-indexed coarse models.
  3. [Figure 2] The annotation 'Multilevel FLOPS to reach Single Level Min Loss: -44%' appears only in the figure; the main text should define how this number is computed and state the corresponding FLOP totals for both methods.
  4. [Section 3, Comparison metrics] The FLOP estimate relies on a factor of 3 forward passes per training step, cited to a blog report; please give the exact reference and, if possible, a sensitivity check of the reported saving to this factor.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the FLOPs saving is an empirical comparison against a single-level baseline, not forced by construction.

full rationale

The paper makes no formal derivation from the ODE interpretation to the reported speedup; its central claim (Section 3) is an empirical comparison of training loss versus fine-level steps and FLOPs between the multilevel algorithm and a standard single-level baseline. The coarse models are constructed by layer subsampling (Eqs. 3 and 5) and prolongation (Eq. 4), but no fitted parameter is renamed as a prediction and no result is defined in terms of the quantity it is said to explain. The 44% FLOPs saving is measured, not derived. The self-citations [16,17] appear only as background for multilevel acceleration and for a convex momentum remark; they are not load-bearing for the transformer-specific empirical result, which is validated against an external single-level baseline on the same data and architecture. The constant-LR coarse phase versus cosine-decay fine baseline is a possible experimental confound, but that is a threat to internal validity, not a circular reduction. The manuscript itself flags its limitations ('the setting of these experiments is limited... more experiments... more theoretical developments'), and no passage asserts a circular step or omitted proof. Therefore no significant circularity is present.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The method depends on the ODE interpretation of transformer layers and on an untested approximation: half-depth coarse models, obtained by skipping every other block and later mixing weights with delta, are faithful enough for gradient transfer. The numerical gains also depend on several hand-tuned hyperparameters (delta, coarse step counts, coarse learning rate). No new physical or mathematical entities are introduced.

free parameters (4)
  • averaging constant delta = 0.25
    Chosen following [30] and the authors' own experiments (Section 3, Multilevel hyperparameters); controls how much coarse weights replace fine weights on prolongation.
  • coarse training length = 100 steps
    Each coarse model is trained for 100 steps before propagation (Section 3, Multilevel hyperparameters); value is hand-set with no ablation or theory.
  • coarse cycle count = 35 steps
    Coarse models are used during the first 35 fine steps, then stopped because later use gave diminishing returns (Section 3, Multilevel hyperparameters); this is a tuned early-stopping choice.
  • coarse learning rate = 1.2e-3 constant
    Coarse models use a constant learning rate while the fine model uses warm-up and cosine decay; no justification beyond hyperparameter choice is given.
assumptions (3)
  • domain assumption A transformer block update (Eq. 1) is the Euler discretization of the ODE dx/dt = F(x,theta(t)) with step dt=1 (Eq. 2).
    This is the standard ODE interpretation of residual networks, cited from [18]; it is an approximation because layer norms and attention nonlinearities are time-dependent and no convergence in depth is shown.
  • ad hoc to paper Coarse networks formed by taking even (Eq. 3) or odd (Eq. 5) layers approximate the fine network well enough for training transfer.
    No operator or correction is used; the paper relies on ODE discretization intuition and does not prove a bound on the approximation error. This is the main load-bearing heuristic.
  • domain assumption Gradient information from coarse models, trained on the same data with SGD, is useful for the fine model.
    This is the core multilevel hypothesis, inherited from [15,10] and not established for transformers.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A multilevel approach to accelerate the training of Transformers." pith.science (2026). https://pith.science/paper/UI6BJPZM

@misc{pith2026250418590,
  author       = {Pith},
  title        = {Pith review of: A multilevel approach to accelerate the training of Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UI6BJPZM}},
  note         = {Machine review of arXiv:2504.18590}
}
read the original abstract

In this article, we investigate the potential of multilevel approaches to accelerate the training of transformer architectures. Using an ordinary differential equation (ODE) interpretation of these architectures, we propose an appropriate way of varying the discretization of these ODE Transformers in order to accelerate the training. We validate our approach experimentally by a comparison with the standard training procedure.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

30 extracted references · 17 canonical work pages

  1. [30]

    L. Zou, H. Zhang, and Y . Deng. A Multi-Level Framework for Acceler- ating Training Transformer Models, April 2024. arXiv:2404.07999. 4

    work page Pith review arXiv 2024

  2. [1]

    Avelin and K

    B. Avelin and K. Nyström. Neural ODEs as the deep limit of ResNets with constant weights. Analysis and Applications , 19(03):397–437, 2021

    work page 2021

  3. [2]

    Baier-Reinio and H

    A. Baier-Reinio and H. De Sterck. N-ode transformer: A depth-adaptive variant of the transformer using neural ordinary differential equations. Preprint arXiv:2010.11358, 2020

    arXiv 2010

  4. [3]

    Brown, B

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. NeurIPS, 33:1877–1901, 2020

    work page 1901

  5. [4]

    Chang, W

    X. Chang, W. Zhang, Y . Qian, J. Le Roux, and S. Watanabe. End-to-end multi-speaker speech recognition with transformer. In ICASSP 2020, pages 6134–6138. IEEE, 2020

    work page 2020

  6. [5]

    C. Chen, Y . Yin, L. Shang, X. Jiang, Y . Qin, F. Wang, Z. Wang, X. Chen, Z. Liu, and Q. Liu. bert2bert: Towards reusable pretrained language models. Preprint arXiv:2110.07143, 2021

    arXiv 2021

  7. [6]

    R. T. Q. Chen, Y . Rubanova, J. Bettencourt, and D. Duvenaud. Neural Ordinary Differential Equations, December 2019. arXiv:1806.07366

    arXiv 2019

  8. [7]

    T. Chen, I. Goodfellow, and J. Shlens. Net2net: Accelerating learning via knowledge transfer. Preprint arXiv:1511.05641, 2015

    arXiv 2015

Show all 30 references
  1. [8]

    N. Ding, Y . Tang, K. Han, C. Xu, and Y . Wang. Network expansion for practical training acceleration. In IEEE/CVF CVPR, pages 20269– 20279, 2023

    2023

  2. [9]

    Dosovitskiy, L

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. Preprint arXiv:2010.11929, 2020

    2010 arXiv

  3. [10]

    Gaedke-Merzhäuser, A

    L. Gaedke-Merzhäuser, A. Kopaniˇcáková, and R. Krause. Multilevel minimization for deep residual networks. ESAIM: Proceedings and Surveys, 71:131–144, August 2021

    2021

  4. [11]

    L. Gong, D. He, Z. Li, T. Qin, L. Wang, and T. Liu. Efficient training of bert by progressively stacking. In ICML, pages 2337–2346. PMLR, 2019

    2019

  5. [12]

    Gratton, V

    S. Gratton, V . Mercier, E. Riccietti, and P. L. Toint. A block-coordinate approach of multi-level optimization with an application to physics- informed neural networks. Computational Optimization and Applica- tions, 89(2):385–417, 2024

    2024

  6. [13]

    X. Gu, L. Liu, H. Yu, J. Li, C. Chen, and J. Han. On the transformer growth for progressive bert training. Preprint arXiv:2010.12562, 2020

    2010 arXiv

  7. [14]

    Kaplan, S

    J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. Scaling Laws for Neural Language Models. CoRR, abs/2001.08361, 2020

    2001 arXiv

  8. [15]

    Kopaniˇcáková and R

    A. Kopaniˇcáková and R. Krause. Globally Convergent Multilevel Train- ing of Deep Residual Networks, June 2022

    2022

  9. [16]

    G. Lauga. Multilevel proximal methods and application to image restora- tion. phdthesis, École Normale Supérieure de Lyon, December 2024

    2024

  10. [17]

    Lauga, E

    G. Lauga, E. Riccietti, N. Pustelnik, and P. Gonçalves. IML FISTA: A Multilevel Framework for Inexact and Inertial Forward-Backward. Application to Image Restoration. SIAM Journal on Imaging Sciences , 17(3):1347–1376, 2024

    2024

  11. [18]

    B. Li, Q. Du, T. Zhou, Y . Jing, S. Zhou, X. Zeng, T. Xiao, J. Zhu, X. Liu, and M. Zhang. ODE Transformer: An Ordinary Differen- tial Equation-Inspired Model for Sequence Generation, March 2022. arXiv:2203.09176

    2022 arXiv

  12. [19]

    L. Liu, X. Liu, J. Gao, W. Chen, and J. Han. Understanding the difficulty of training transformers. Preprint arXiv:2004.08249, 2020

    2004 arXiv

  13. [20]

    Z. Liu, Y . Lin, Y . Cao, H. Hu, Y . Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In IEEE/CVF ICCV, pages 10012–10022, 2021

    2021

  14. [21]

    L. Lu, C. Liu, J. Li, and Y . Gong. Exploring transformers for large-scale speech recognition. Preprint arXiv:2005.09684, 2020

    2005 arXiv

  15. [22]

    Penedo, H

    G. Penedo, H. Kydlíˇcek, L. Ben allal, A. Lozhkov, M. Mitchell, C. Raffel, L. V on Werra, and T. Wolf. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale, 2024

    2024

  16. [23]

    Quemener and M

    E. Quemener and M. Corvellec. SIDUS—the Solution for Extreme Deduplication of an Operating System. Linux Journal, 2013

    2013

  17. [24]

    Radford, J

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019

    2019

  18. [25]

    Touvron, T

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.A. Lachaux, et al. Llama: Open and efficient foundation language models. Preprint arXiv:2302.13971, 2023

    2023 arXiv

  19. [26]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is All you Need. 2017

    2017

  20. [27]

    P. Wang, R. Panda, L. T. Hennigen, P. Greengard, L. Karlinsky, R. Feris, D. D. Cox, Z. Wang, and Y . Kim. Learning to grow pretrained models for efficient transformer training. Preprint arXiv:2303.00980, 2023

    2023 arXiv

  21. [28]

    S. Yang, L. Hou, X. Song, Q. Liu, and D. Zhou. Speeding up deep model training by sharing weights and then unsharing. Preprint arXiv:2110.03848, 2021

    2021 arXiv

  22. [29]

    R. Zhao, D. Morwani, D. Brandfonbrener, N. Vyas, and S. Kakade. Deconstructing what makes a good optimizer for language models. Preprint arXiv:2407.07972, 2024

    2024 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.