Pith. sign in

REVIEW 2 major objections 5 minor 2 cited by

This paper claims that any non-uniform distribution of layer sizes beats an equal-sized uniform model at equal parameter count, and that the exact shape of the distribution barely matters.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

At 180M parameters and 5B tokens, all layer-wise scaling variants beat the paper's 18-layer uniform baseline, yet the 12-layer uniform baseline remains best.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection An honest small-scale ablation of LWS variants whose 'heterogeneity helps' conclusion is undercut by a depth-confounded baseline, though the negative result on LWS alone is a useful data point. the 2 major comments →

arxiv 2509.06518 v1 pith:VI4SBPIM submitted 2025-09-08 cs.CL cs.AI

Crown, Frame, Reverse: Layer-Wise Scaling Variants for LLM Pre-Training

Andrei Baroian , Kasper Notebomer This is my paper
classification cs.CL cs.AI
keywords Layer-Wise Scalingheterogeneous architecturesLLM pre-trainingvalidation perplexityparameter allocationgrouped-query attentionlayer importancepruning-inspired design
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper isolates Layer-Wise Scaling (LWS) from other architectural changes and tests four ways to redistribute feed-forward widths and attention heads across layers at a fixed 180M-parameter budget and 5B training tokens. All four non-uniform profiles — linearly increasing, framed, reverse, and crown-shaped — reach validation perplexities meaningfully below an 18-layer isotropic baseline, with gains around 5–6%. The authors conclude that heterogeneity itself, not the specific profile, drives the improvement, and that LWS reallocates rather than creates representational capacity. They also find that LWS alone does not deliver the roughly 2x data-efficiency gain previously attributed to it, suggesting that earlier claims depend on other parts of the training recipe.

Core claim

On the paper's own terms, the central discovery is that pre-training a Transformer with non-uniform per-layer parameter allocation reliably improves validation perplexity over an equal-cost isotropic baseline, while the choice among heterogeneous profiles is a second-order concern. The authors compare an 18-layer uniform model with four LWS variants — Vanilla (linearly growing), Framed (maximum first and last layers), Reverse (largest early layers, framed), and Crown (peak in the middle, framed) — all at around 180M parameters trained on 5B tokens with grouped-query attention applied throughout. Every variant beats the uniform baseline, and the three best variants cluster within 0.04 perplex

What carries the argument

The machinery is a per-layer linear interpolation of two architectural widths: the feed-forward expansion factor beta_i (FFN hidden size) and the query-head count alpha_i, with key/value heads fixed by grouped-query attention. Profiles are defined by the endpoints (and, for Crown, a middle point) of the interpolation, and all variants are built to the same total parameter budget. This device converts a qualitative hypothesis about differential layer importance into concrete architectures that can be trained and compared, and it is what allows the paper to attribute the observed perplexity gap to heterogeneity per se.

Load-bearing premise

The 18-layer uniform baseline, with its particular width scalars (FFN scale 2.5, QKV scale 0.75), fairly represents isotropic architectures at the 180M budget; if a better-tuned uniform model were used, the all-variants-beat-baseline result could weaken.

What would settle it

Train the same 18-layer uniform model with the same per-layer compute but tuned width scalars (e.g., matching the average FLOPs of the LWS variants) and show that a well-tuned uniform model closes the perplexity gap; or scale the comparison to a 7B-parameter model trained on >=100B tokens and check whether the perplexity advantage of the crown, reverse, and vanilla profiles persists or reverses.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Architectures that vary layer sizes at fixed parameter count can improve validation perplexity by roughly 5–6% over a uniform model of the same size, without slowing training.
  • The shape of the layer-wise profile is a second-order choice: Vanilla, Reverse, and Crown cluster together, so practitioners may pick a profile for hardware or pipeline convenience.
  • LWS alone does not reproduce the ~2x data-efficiency advantage claimed by OpenELM; that advantage likely arises from the interaction of LWS with other recipe components.
  • Because the gains are reallocative rather than additive, the practical value is in matching compute, not in increasing capacity.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the heterogeneity effect survives scaling, it would let model builders trade a uniform architecture for one matched to the layer-importance profile discovered by pruning studies, potentially getting the same perplexity at fewer parameters or tokens.
  • The near-equivalence of all non-uniform profiles suggests a saturation or regularity phenomenon: once the network is released from exact width uniformity, the optimizer adapts to a variety of shapes. A testable extension would train the same profiles at 7B parameters and >=100B tokens to see whether the gap to isotropic models widens or collapses.
  • The paper's own perplexity values are unusually low (around 5) relative to typical 10–15 for larger models; if that reflects an artifact of the validation set, the relative ranking of variants could still hold while the absolute gains might not transfer.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper studies whether non-uniform (layer-wise) parameter allocation can improve language model pre-training at a fixed parameter budget. Building on OpenELM's Layer-Wise Scaling (LWS), it introduces three variants -- Framed, Reverse, and Crown -- that redistribute FFN widths and attention-head counts via linear interpolation profiles, and compares them against an isotropic baseline. Experiments use a 180M-parameter model trained on 5B tokens, with validation perplexity as the main metric. The authors report that all 18-layer LWS variants outperform the 18-layer isotropic baseline, that the exact profile matters little, and that LWS alone does not reproduce OpenELM's claimed data-efficiency gains. The central evidence is Table 3 and Figures 4-5.

Significance. If the main claim were robust, this would be a useful contribution: it is the first controlled isolation of LWS from the other changes in OpenELM, and it suggests a cheap architectural improvement at fixed parameter count. The paper has genuine strengths: a fixed 180M parameter budget with parameter accounting in Table 2, a baseline repeated five times, explicit acknowledgment of limitations, and open-source code. However, the current evidence does not establish the headline claim because the isotropic comparator is confounded with depth and tuning. The central 'heterogeneity helps' conclusion should therefore be treated as conditional pending a fairer baseline and multi-seed evaluation.

major comments (2)
  1. [Section 4 / Table 3] The paper's central comparison is confounded by depth and baseline tuning. The original 12-layer isotropic baseline (PPL 4.962) beats every 18-layer LWS variant (Crown 5.057, Reverse 5.087, Vanilla 5.093, Framed 5.205), and 12-layer vanilla LWS (4.984) does not beat the 12-layer baseline. The 18-layer baseline was introduced only after the authors saw that 12-layer LWS did not help, and it is configured as a deeper, narrower uniform model (FFN 2.5, QKV 0.75) without reported tuning; its PPL is 5.400. The conclusion that 'heterogeneity helps' therefore rests on an isotropic comparator that under-performs the original 12-layer model. The authors should either tune the 18-layer baseline, include the 12-layer comparison in the central claim, or explicitly restrict the conclusion to the 18-layer deep-narrow regime.
  2. [Section 5 / Section 5.1] The claim that 'the exact shape matters little' is not supported by the reported statistics. Only the 18-layer baseline was run five times (std 0.0595 on final PPL, Table 3); each LWS variant was run once. Framed LWS (PPL 5.205) differs from Crown LWS (5.057) by about 2.5 baseline standard deviations, and Figure 5 shows a late-training PPL bump for Framed. Without per-variant error bars or a formal significance test, the apparent convergence among variants -- and some of the improvement over baseline -- could be run-to-run variation. The authors acknowledge the single-run limitation in Section 5.1, but this is precisely the point on which the 'shape does not matter' conclusion depends.
minor comments (5)
  1. [Table 1] 'Training steps 1 3000' appears to be a typo; the table should also state explicitly that it is for the 12-layer baseline, since later models use 18 layers.
  2. [Section 3.2, Eqs. (1)-(4)] The index range '0 ≤ i ≤ N' should likely be '0 ≤ i < N' or the interpolation should be defined consistently for N layers; there is also a typo in Eq. (1) ('di ffn').
  3. [Section 4] 'Baseline 12L has 180M tokens instead of the 190M from olmo2-190M configuration' should read 'parameters', not 'tokens'.
  4. [Section 5.1] The heading 'Weakneseess' is misspelled. More substantively, the text acknowledges single-run training but frames it as future work rather than qualifying the abstract's and conclusion's definitive claims.
  5. [Figures 4-5 and references] Figures 4 and 5 would be clearer with shaded error bars or confidence bands for the baseline; the zoomed plots should state the exact x-axis range. The DOLMino citation is malformed and should be cleaned up.

Circularity Check

0 steps flagged

No significant circularity: empirical ablation with hand-chosen scaling vectors and externally cited motivations.

full rationale

This paper is a purely empirical ablation. The claimed results are validation perplexities measured from trained models, not quantities derived from the scaling-formula inputs. Equations (1)-(4) define layer FFN widths and attention-head counts from interpolation scalars chosen to equalize parameter counts, and Table 3 reports the resulting measured PPL values. There is no step in which an output quantity is defined in terms of the claimed conclusion, and no fitted parameter is renamed as a prediction. The citations to LWS (Mehta et al. 2024) and to pruning work (He et al. 2024, Pan et al. 2025, Askari et al. 2025, etc.) are external prior work, not self-citations by the present authors, and the paper explicitly treats OpenELM's LWS benefit as an open question rather than importing it as a premise. The mid-study switch from 12 to 18 layers and the untuned 18-layer isotropic baseline are threats to the validity of the 'heterogeneity helps' conclusion, but they are experimental confounds, not circular reasoning: the LWS scalars were not fitted to validation perplexity, and the baseline was not constructed to make the variants win by definition. Therefore no circular step is present.

Axiom & Free-Parameter Ledger

7 free parameters · 4 axioms · 0 invented entities

The central empirical comparison rests on hand-picked scaling vectors and a held-out set drawn from the same data source. These are not fitted to the outcome, but they are free choices that affect the conclusions. No new entities are introduced.

free parameters (7)
  • Baseline 18L FFN/QKV scaling vectors = [2.5, 2.5] / [0.75, 0.75]
    Chosen by hand to hold 180M parameters at 18 layers; this uniform baseline is the comparison anchor.
  • Vanilla LWS 12L scaling vectors = [2.0, 5.3] / [0.5, 2.0]
    Chosen to match the 12-layer parameter budget.
  • Vanilla LWS 18L scaling vectors = [1.0, 4.0] / [0.5, 1.0]
    Chosen to hold 179.7M parameters at 18 layers.
  • Framed LWS scaling vectors = [0.5, 4.0] / [0.5, 1.0]
    Chosen to keep first and last layers at maximum size.
  • Reverse LWS scaling vectors = [4.0, 0.5] / [1.0, 0.5]
    Chosen to allocate more parameters to early layers.
  • Crown LWS scaling vectors = [0.5, 3.8, 0.5] / [0.5, 1.0, 0.5]
    Chosen to peak in the middle layers.
  • Model depth = 18 layers
    Increased from OLMo2's 12 layers to make interpolation smoother (Section 4); this choice directly affects the comparison.
axioms (4)
  • domain assumption Validation PPL on a 10M held-out set from the same DOLMino/DCLM data source measures generalization
    The paper itself notes PPL 5 is unusually low and possible distribution overlap (Section 5.1).
  • domain assumption Small-model results extrapolate to larger scales
    Invoked to give meaning to 180M/5B results, citing Kaplan and Hoffmann.
  • standard math Linear interpolation equations (2) and (4) define per-layer sizes
    Used to construct all LWS variants; no alternative scaling law is considered.
  • domain assumption Deeper models tend to have lower perplexity
    Cited to Petty et al. to justify switching from 12 to 18 layers (Section 4).

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Crown, Frame, Reverse: Layer-Wise Scaling Variants for LLM Pre-Training." pith.science (2026). https://pith.science/paper/VI4SBPIM

@misc{pith2026250906518,
  author       = {Pith},
  title        = {Pith review of: Crown, Frame, Reverse: Layer-Wise Scaling Variants for LLM Pre-Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VI4SBPIM}},
  note         = {Machine review of arXiv:2509.06518}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Transformer-based language models traditionally use uniform (isotropic) layer sizes, yet they ignore the diverse functional roles that different depths can play and their computational capacity needs. Building on Layer-Wise Scaling (LWS) and pruning literature, we introduce three new LWS variants - Framed, Reverse, and Crown - that redistribute FFN widths and attention heads via two or three-point linear interpolation in the pre-training stage. We present the first systematic ablation of LWS and its variants, on a fixed budget of 180M parameters, trained on 5B tokens. All models converge to similar losses and achieve better performance compared to an equal-cost isotropic baseline, without a substantial decrease in training throughput. This work represents an initial step into the design space of layer-wise architectures for pre-training, but future work should scale experiments to orders of magnitude more tokens and parameters to fully assess their potential.

Figures

Figures reproduced from arXiv: 2509.06518 by Andrei Baroian, Kasper Notebomer.

Figure 1
Figure 1. Figure 1: Figure from He et al. (2024) showing similarity score across training. Block number on x-axis, darker shades indicate higher block importance Another insight from the same paper comes from the shifting of similarity score during pre-training, as it starts relatively flat across all layers and gradually transitions into a ”crown”- shaped pattern toward the end of training. It is important to note that in po… view at source ↗
Figure 3
Figure 3. Figure 3: Layer-Wise Scaling (LWS) variants used in our study. Parameter-allocation profiles for 18 layers : Baseline is an uniform (isotropic) model; (a) Vanilla LWS linearly increases from shallow to deep; (b) Framed LWS keeps the first and last layers at maximum size while linearly scaling; (c) Reverse LWS more parameters allocated to initial layers, decreasing it toward later layers, using framing; (d) Crown LWS… view at source ↗
Figure 4
Figure 4. Figure 4: Validation Perplexity for LWS vs Baseline at 12 and 18 layers [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: shows the results of all the variants. Unexpectedly, all the variants performed almost the same, all significant better than the baseline. To exclude the possibility that this convergence arose from implementation error, for example, inadvertently training of identical architectures, we manu￾ally inspected the run logs and confirmed that the correct dimensions were instantiated in every run. Framed LWS tho… view at source ↗
Figure 6
Figure 6. Figure 6: Training Cross Entropy Loss [PITH_FULL_IMAGE:figures/full_fig_p010_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Training Cross Entropy Loss Zoomed in 10 [PITH_FULL_IMAGE:figures/full_fig_p010_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Tapered Language Models

    cs.LG 2026-06 unverdicted novelty 7.0

    Tapered Language Models monotonically decrease MLP width across depth with a cosine schedule, yielding better perplexity and downstream performance than uniform-width baselines across multiple architectures and scales...

  2. Variable-Width Transformers

    cs.CL 2026-06 conditional novelty 6.0

    ×-shaped variable-width transformers outperform parameter-matched uniform baselines on language modeling loss with 22% fewer FLOPs and 15% smaller KV cache.

Reference graph

Works this paper leans on

30 extracted references · 8 canonical work pages · cited by 2 Pith papers · 3 internal anchors

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Gqa: Training generalized multi-query transformer models from multi-head checkpoints, 2023

    Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebrón, F., and Sanghai, S. Gqa: Training generalized multi-query transformer models from multi-head checkpoints, 2023. URL https://arxiv.org/abs/2305.13245

    Pith/arXiv arXiv 2023

  3. [3]

    Layerif: Estimating layer quality for large language models using influence functions, 2025

    Askari, H., Gupta, S., Wang, F., Chhabra, A., and Chen, M. Layerif: Estimating layer quality for large language models using influence functions, 2025. URL https://arxiv.org/abs/2505.23811

    work page arXiv 2025

  4. [4]

    A., Purohit, S., Prashanth, U

    Biderman, S., Schoelkopf, H., Anthony, Q., Bradley, H., O'Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., Skowron, A., Sutawika, L., and van der Wal, O. Pythia: A suite for analyzing large language models across training and scaling, 2023. URL https://arxiv.org/abs/2304.01373

    Pith/arXiv arXiv 2023

  5. [5]

    Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, ...

    Pith/arXiv arXiv 2020

  6. [6]

    Is bigger and deeper always better? probing llama across scales and layers, 2024

    Chen, N., Wu, N., Liang, S., Gong, M., Shou, L., Zhang, D., and Li, J. Is bigger and deeper always better? probing llama across scales and layers, 2024. URL https://arxiv.org/abs/2312.04333

    Pith/arXiv arXiv 2024

  7. [7]

    Dolmino Mix 1124 , year = 2024 , n. . O. h. . https://huggingface.co/datasets/allenai/dolmino-mix-1124 . Dolmino mix 1124

    work page 2024

  8. [8]

    Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Training

    Du, W., Luo, T., Qiu, Z., Huang, Z., Shen, Y., Cheng, R., Guo, Y., and Fu, J. Stacking your transformers: A closer look at model growth for efficient llm pre-training, 2024. URL https://arxiv.org/abs/2405.15319

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  9. [9]

    M o LA : M o E L o RA with layer-wise expert allocation

    Gao, C., Chen, K., Rao, J., Liu, R., Sun, B., Zhang, Y., Peng, D., Guo, X., and Subrahmanian, V. M o LA : M o E L o RA with layer-wise expert allocation. In Chiruzzo, L., Ritter, A., and Wang, L. (eds.), Findings of the Association for Computational Linguistics: NAACL 2025, pp.\ 5097--5112, Albuquerque, New Mexico, April 2025. Association for Computationa...

    doi:10.18653/v1/2025.findings-naacl.284 2025

  10. [10]

    What matters in transformers? not all attention is needed, 2024

    He, S., Sun, G., Shen, Z., and Li, A. What matters in transformers? not all attention is needed, 2024. URL https://arxiv.org/abs/2406.15786

    Pith/arXiv arXiv 2024

  11. [12]

    A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Rae, J

    Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Rae, J. W., Vinyals, O., and Sifre, L. Training compute-optimal large language models, 2022 b ....

    Pith/arXiv arXiv 2022

  12. [13]

    Determining Layer-wise Sparsity for Large Language Models Through a Theoretical Perspective

    Huang, W., Zhang, Y., Zheng, X., Chao, F., and Ji, R. Determining layer-wise sparsity for large language models through a theoretical perspective, 2025. URL https://arxiv.org/abs/2502.14770

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  13. [14]

    Exploring concept depth: How large language models acquire knowledge and concept at different layers?, 2025

    Jin, M., Yu, Q., Huang, J., Zeng, Q., Wang, Z., Hua, W., Zhao, H., Mei, K., Meng, Y., Ding, K., Yang, F., Du, M., and Zhang, Y. Exploring concept depth: How large language models acquire knowledge and concept at different layers?, 2025. URL https://arxiv.org/abs/2404.07066

    Pith/arXiv arXiv 2025

  14. [15]

    Johannes Welbl, Nelson F. Liu, M. G. Crowdsourcing multiple choice science questions. 2017

    work page 2017

  15. [16]

    and Martin, J

    Jurafsky, D. and Martin, J. H. Chapter 3: N-gram language models. In Speech and Language Processing. 3rd\,ed.\ (draft) edition, 2025. Draft dated 12 Jan 2025. URL: https://web.stanford.edu/ jurafsky/slp3/3.pdf

    work page 2025

  16. [18]

    B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models, 2020 b . URL https://arxiv.org/abs/2001.08361

    Pith/arXiv arXiv 2020

  17. [19]

    W., Jitsev, J., Kollar, T., Dimakis, A

    Li, J., Fang, A., Smyrnis, G., Ivgi, M., Jordan, M., Gadre, S., Bansal, H., Guha, E., Keh, S., Arora, K., Garg, S., Xin, R., Muennighoff, N., Heckel, R., Mercat, J., Chen, M., Gururangan, S., Wortsman, M., Albalak, A., Bitton, Y., Nezhurina, M., Abbas, A., Hsieh, C.-Y., Ghosh, D., Gardner, J., Kilian, M., Zhang, H., Shao, R., Pratt, S., Sanyal, S., Ilharc...

    Pith/arXiv arXiv 2025

  18. [20]

    Delight: Deep and light-weight transformer, 2021

    Mehta, S., Ghazvininejad, M., Iyer, S., Zettlemoyer, L., and Hajishirzi, H. Delight: Deep and light-weight transformer, 2021. URL https://arxiv.org/abs/2008.00623

    Pith/arXiv arXiv 2021

  19. [21]

    H., Cao, Q., Horton, M., Jin, Y., Sun, C., Mirzadeh, I., Najibi, M., Belenko, D., Zatloukal, P., et al

    Mehta, S., Sekhavat, M. H., Cao, Q., Horton, M., Jin, Y., Sun, C., Mirzadeh, I., Najibi, M., Belenko, D., Zatloukal, P., et al. Openelm: An efficient language model family with open training and inference framework. arXiv preprint arXiv:2404.14619, 2024

    Pith/arXiv arXiv 2024

  20. [22]

    W., Liu, J., Malik, S., Merrill, W., Miranda, L

    OLMo, T., Walsh, P., Soldaini, L., Groeneveld, D., Lo, K., Arora, S., Bhagia, A., Gu, Y., Huang, S., Jordan, M., Lambert, N., Schwenk, D., Tafjord, O., Anderson, T., Atkinson, D., Brahman, F., Clark, C., Dasigi, P., Dziri, N., Guerquin, M., Ivison, H., Koh, P. W., Liu, J., Malik, S., Merrill, W., Miranda, L. J. V., Morrison, J., Murray, T., Nam, C., Pyatk...

    Pith/arXiv arXiv 2025

  21. [23]

    Adapt-pruner: Adaptive structural pruning for efficient small language model training, 2025

    Pan, R., Wang, B., Diao, S., Pan, X., Zhang, J., Pi, R., and Zhang, T. Adapt-pruner: Adaptive structural pruning for efficient small language model training, 2025. URL https://arxiv.org/abs/2502.03460

    arXiv 2025

  22. [24]

    The impact of depth on compositional generalization in transformer language models, 2024

    Petty, J., van Steenkiste, S., Dasgupta, I., Sha, F., Garrette, D., and Linzen, T. The impact of depth on compositional generalization in transformer language models, 2024. URL https://arxiv.org/abs/2310.19956

    Pith/arXiv arXiv 2024

  23. [25]

    Fast transformer decoding: One write-head is all you need, 2019

    Shazeer, N. Fast transformer decoding: One write-head is all you need, 2019. URL https://arxiv.org/abs/1911.02150

    Pith/arXiv arXiv 2019

  24. [26]

    R., Zhao, D., Patel, N., Naghiyev, J., LeCun, Y., and Shwartz-Ziv, R

    Skean, O., Arefin, M. R., Zhao, D., Patel, N., Naghiyev, J., LeCun, Y., and Shwartz-Ziv, R. Layer by layer: Uncovering hidden representations in language models, 2025. URL https://arxiv.org/abs/2502.02013

    Pith/arXiv arXiv 2025

  25. [27]

    Snellius supercomputer

    SURF . Snellius supercomputer. https://www.surf.nl/en/compute/snellius-supercomputer, 2025. Accessed: 2025-06-30

    work page 2025

  26. [28]

    Dlo: Dynamic layer operation for efficient vertical scaling of llms, 2024

    Tan, Z., Dong, D., Zhao, X., Peng, J., Cheng, Y., and Chen, T. Dlo: Dynamic layer operation for efficient vertical scaling of llms, 2024. URL https://arxiv.org/abs/2407.11030

    Pith/arXiv arXiv 2024

  27. [29]

    N., Kaiser, L., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need, 2023. URL https://arxiv.org/abs/1706.03762

    Pith/arXiv arXiv 2023

  28. [30]

    Model compression and efficient inference for large language models: A survey, 2024

    Wang, W., Chen, W., Luo, Y., Long, Y., Lin, Z., Zhang, L., Lin, B., Cai, D., and He, X. Model compression and efficient inference for large language models: A survey, 2024. URL https://arxiv.org/abs/2402.09748

    Pith/arXiv arXiv 2024

  29. [31]

    Brainformers: Trading Simplicity for Efficiency

    Zhou, Y., Du, N., Huang, Y., Peng, D., Lan, C., Huang, D., Shakeri, S., So, D., Dai, A., Lu, Y., Chen, Z., Le, Q., Cui, C., Laudon, J., and Dean, J. Brainformers: Trading simplicity for efficiency, 2024. URL https://arxiv.org/abs/2306.00008

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  30. [32]

    A survey on model compression for large language models

    Zhu, X., Li, J., Liu, Y., Ma, C., and Wang, W. A survey on model compression for large language models. Transactions of the Association for Computational Linguistics, 12: 0 1556--1577, 11 2024. ISSN 2307-387X. doi:10.1162/tacl_a_00704. URL https://doi.org/10.1162/tacl\_a\_00704

    doi:10.1162/tacl_a_00704 2024

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.