Pith. sign in

REVIEW 3 major objections 5 minor 54 references

Which student layers you swap for teacher blocks decides how well intermediate-size models interpolate, and a KL-greedy order often finds a near-optimal path.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-10 11:56 UTC pith:YULMOCWG

load-bearing objection Clean formalization and a practical O(N^{2}) greedy algorithm for the patching-order problem in boomerang distillation; solid extension, not a paradigm shift. the 3 major comments →

arxiv 2607.08170 v1 pith:YULMOCWG submitted 2026-07-09 cs.LG

Understanding Layer Patching in Model Size Interpolation

Sara Kangaslahti , Jonathan Geuter , Nihal V. Nayak , Marco Fumero , Francesco Locatello , David Alvarez-Melis This is my paper
classification cs.LG
keywords model size interpolationboomerang distillationlayer patchingshortest pathKL divergenceknowledge distillationlanguage models
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Zero-shot model size interpolation builds intermediate language models by patching a distilled student with contiguous teacher layer blocks, without further training. This paper shows that the order of those patches is not a minor implementation detail: different orders produce very different performance-versus-size curves, and a naive larger model can even underperform a smaller one. The authors cast choosing the best nested sequence of patches as a combinatorial optimization problem and prove that, under equidistant size steps, it is equivalent to a shortest path on a lattice whose edges are weighted by expected KL divergence to the teacher. Exhaustive checks on small models and large random samples on bigger ones show that simple last-to-first (or first-to-last) orders are often competitive, while a greedy algorithm called KLPatch frequently matches or beats them and approaches the empirical optimum. The practical payoff is a cheap recipe for building smooth families of intermediate-size models from one teacher–student pair.

Core claim

Finding the optimal nested patching order for model-size interpolation is equivalent, under equidistant consecutive sizes, to a shortest path in the Boolean lattice of partially patched models whose edge weights are expected KL divergence from the teacher’s next-token distribution. A greedy algorithm that always patches the layer that most reduces that KL produces near-optimal interpolation trajectories on several language-model families.

What carries the argument

The interpolation graph: the Boolean lattice of student-layer subsets, with an edge from a model to the same model after one more teacher block is patched, weighted by expected KL(p_teacher || p_patched) on a calibration set. Shortest paths in this graph are optimal permutations for teacher-relative log-area-under-perplexity-curve; KLPatch greedily follows the cheapest outgoing edge at each step.

Load-bearing premise

The optimality claim needs consecutive patched models to grow by roughly the same number of parameters, and needs KL distance to the teacher to track real data perplexity and downstream accuracy closely enough.

What would settle it

On a fixed teacher–student pair, compute the true minimum AUPIC (or AUIC) over all nested orders—or a large random sample—and check whether the KLPatch order’s score is near that minimum; a large, systematic gap on a new model family would falsify the claim that the KL path is near-optimal in practice.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies student-layer selection for zero-shot model-size interpolation via boomerang distillation. It formalizes optimal nested patching as a combinatorial problem over subsets/permutations (Problems 1–2), shows that under equidistant consecutive sizes the optimal permutation w.r.t. teacher-relative log-AUPIC is exactly a shortest path in the Boolean lattice whose edge weights are expected KL(p_T || p_M) (Definition 1, Proposition 1), and introduces the greedy O(N^{2}) KLPatch algorithm that selects the cheapest outgoing edge at each step. Exhaustive enumeration on DistilBERT/DistilGPT2 (N=6), 200 random orders on Qwen3-4B/8B and Pythia-6.9B, and ablations (cosine vs KL, iterative vs non-iterative, calibration size) show that last-to-first is often strong, that good orders cluster near the optimum in footrule distance, and that KLPatch recovers best or near-best AUPIC/AUIC trajectories on those families (with a hand-crafted first-layer constraint needed for Llama).

Significance. If the reduction and the empirical near-optimality of KLPatch hold, the work supplies the first principled account of the combinatorial design space of layer patching and a practical O(N^{2}) recipe that improves on the fixed sequential baselines of the prior boomerang-distillation paper. Strengths include a short, correct proof of Proposition 1 under the stated assumptions, exhaustive enumeration on the small models, transparent ablations (Appendix K), and explicit acknowledgment of the Llama exception and the approximate equidistance of real student initializations (Remark 2). The contribution is incremental relative to Kangaslahti et al. (2026) but fills a genuine user-facing gap and is immediately usable for constructing intermediate-size models without retraining.

major comments (3)
  1. Proposition 1 (and the optimality claim for KLPatch) rests on equidistant consecutive model sizes. Remark 2 notes that the student initializations keep first/last layers, so the assumption is only approximate. The manuscript should quantify the size variation across edges for the actual Qwen/Pythia/Llama students (e.g., max relative deviation of |M_{A∪{i}}|−|M_A|) and either restate the guarantee for non-uniform edge lengths or show that the greedy path remains near-optimal under the observed deviations.
  2. Appendix J shows that plain KLPatch underperforms first-to-last on Llama-3.2-3B and recovers only after the hand-crafted constraint “always patch layer 1 first.” Because the abstract and §6 claim that KLPatch “often improves over last-to-first \ldots across several language models,” the main text should either (i) present the Llama result as a clear family-specific failure mode tied to student initialization, or (ii) supply a diagnostic (e.g., first-layer KL gap) that predicts when the unconstrained algorithm will fail.
  3. The proxy chain teacher-log-PPL → data-PPL → downstream AUIC is only partially validated. Propositions 2–3 give additive/multiplicative bounds, and Appendix G shows strong Pearson correlations on DistilBERT/GPT2, yet §5.3 already notes that last-to-first is best on Wikitext AUPIC but leaves a substantial gap on downstream AUIC for the large models. A short table of Spearman rank correlations between KL-path length, data-AUPIC and AUIC on the 200-order samples would make the strength of the proxy transparent rather than leaving it to the reader to infer from the figures.
minor comments (5)
  1. Figure 1 caption and the surrounding text use both “block” and “layer” for the same student unit; a single consistent term would reduce confusion.
  2. In §4.2 the AUIC/AUPIC formulas are written with a trapezoidal average; the text never states whether the reported “normalized AUPIC” simply divides by total size span or also by the student–teacher gap. One clarifying sentence would help.
  3. Table 3 (DistilGPT2) shows that the global shortest KL path coincides with first-to-last, while the minimum-AUPIC path is different; a one-sentence remark on why the KL optimum and the data-AUPIC optimum diverge on this model would be useful.
  4. The calibration-set ablation (Table 7) is thorough; moving the |D_cal|=64 recommendation into the main-text description of Algorithm 1 would make the practical recipe self-contained.
  5. A few typos: “boomerang distillation [Kangaslahti et al., 2026]” appears with a future year in the abstract; “AUPIC Percentile” is defined twice (G.3 and I.1) with slightly different wording.

Circularity Check

0 steps flagged

No significant circularity: shortest-path equivalence is a direct algebraic identity from the KL decomposition and AUPIC definition; self-citation supplies only the experimental setup.

full rationale

Proposition 1 equates argmin of path length E(π) in the KL-weighted Boolean lattice to argmin of AUPIClog_T(π) by the elementary identity Ex KL(pT ∥ pM) = log PPLT(M) - log PPLT(T) together with the equidistant-size assumption that converts the sum into a scaled trapezoidal area. This is a definitional rewriting, not a prediction forced by a fit or by an external uniqueness claim. Propositions 2–3 merely bound the gap to data-referenced and raw-scale AUPIC; they do not close a loop. KLPatch is an explicit greedy heuristic on the same graph and is validated empirically against exhaustive enumeration (DistilBERT/GPT2) and random baselines (Qwen/Pythia), not by construction. The only self-citation is to Kangaslahti et al. 2026 for the boomerang-distillation procedure and the released student checkpoints; those supply the experimental substrate, not a load-bearing uniqueness or ansatz that forces the optimality result. No fitted parameter is later called a prediction, no uniqueness theorem is imported from the same authors, and no known empirical pattern is merely renamed. Score 1 reflects only the ordinary reliance on the authors’ prior models for the empirical sections; the formal claim is self-contained.

Axiom & Free-Parameter Ledger

2 free parameters · 3 axioms · 2 invented entities

The central claims rest on standard KL/CE identities, the Boolean-lattice path interpretation, and the empirical fidelity of teacher KL as a proxy. Free parameters are mainly experimental choices (calibration size, number of random orders). No new physical entities are invented.

free parameters (2)
  • calibration set size |D_cal| = 64
    Fixed at 64 Pile samples after an ablation; performance stabilizes around this value but is still a hand-chosen hyper-parameter of KLPatch.
  • number of random patching orders sampled for large models = 200
    200 orders used to estimate the AUIC/AUPIC distribution; the reported percentiles are relative to this finite sample.
axioms (3)
  • standard math Cross-entropy decomposition H(p_T, p_M) = H(p_T) + KL(p_T || p_M) holds for the next-token distributions of any intermediate model M.
    Used in the proof of Proposition 1 to equate path length with teacher-relative log-AUPIC.
  • domain assumption Consecutive interpolated models differ by a constant number of parameters (equidistant sizes).
    Required for the exact equality of shortest paths and optimal permutations in Proposition 1; only approximately true for the student initializations used (Remark 2).
  • domain assumption Teacher-relative KL (or log-perplexity) is a sufficiently faithful proxy for data perplexity and downstream accuracy that greedy minimization of the former yields good values of the latter.
    Supported by high Pearson correlations in Appendix G but fails enough on Llama to require a manual constraint.
invented entities (2)
  • Interpolation graph G (Boolean lattice with KL edge weights) no independent evidence
    purpose: Turns the combinatorial patching-order problem into a shortest-path problem that KLPatch approximates.
    Standard construction once the edge weight is chosen; no independent physical existence claimed.
  • KLPatch algorithm independent evidence
    purpose: Greedy O(N^{2}) procedure that selects the next layer minimizing expected KL to the teacher.
    New algorithmic contribution; independent evidence is the empirical near-optimality on the studied models.

pith-pipeline@v1.1.0-grok45 · 33388 in / 2739 out tokens · 31187 ms · 2026-07-10T11:56:18.105834+00:00 · methodology

0 comments
read the original abstract

Zero-shot model size interpolation aims to create new models of intermediate target sizes by combining existing models without additional training. Recent work on boomerang distillation [Kangaslahti et al., 2026] shows that a student language model distilled from a larger teacher can be expanded by iteratively patching its layers, replacing student layers with contiguous blocks of teacher layers to obtain models whose size and performance interpolate between the student and the teacher. In this work, we provide the first systematic study of student-layer selection for model size interpolation. We cast finding the optimal layer subset for each model size as an optimization problem and prove it can be viewed as a shortest-path problem in a certain acyclic graph. In experiments, we show that patching strongly shapes interpolation behavior, with effects that vary substantially across model families. We find that simple sequential strategies--patching either from the first layer to the last or from the last to the first--often achieve surprisingly strong performance in practice. We further introduce KLPatch, a greedy patching algorithm based on KL divergence, which often improves over last-to-first patching and approximately solves the optimization problem. Together, our results provide a principled understanding of how layer patching affects model size interpolation and offer practical guidance for constructing near-optimal interpolated models.

Figures

Figures reproduced from arXiv: 2607.08170 by David Alvarez-Melis, Francesco Locatello, Jonathan Geuter, Marco Fumero, Nihal V. Nayak, Sara Kangaslahti.

Figure 1
Figure 1. Figure 1: Patching order in model size interpolation. We show boomerang distillation with a six￾layer teacher and a three-block distilled student, where each student block corresponds to a contiguous pair of two-layer teacher blocks. A patching order specifies the sequence in which student layers are replaced by their corresponding teacher blocks, inducing a trajectory of intermediate-size models from the student to… view at source ↗
Figure 2
Figure 2. Figure 2: All permutations of patching order for DistilBERT and DistilGPT2. Patching order significantly affects interpolation performance, and the best ordering yields nearly linear interpolation in (pseudo-)perplexity between the students (DistilBERT, DistilGPT2) and teachers (BERT, GPT2). Shaded bands show the (25-75) inter-quartile range over all 720 orderings. 0 5 10 15 Spearman’s Footrule Distance 0.50 0.75 1.… view at source ↗
Figure 3
Figure 3. Figure 3: AUPIC vs. Spearman footrule distance from the minimum-AUPIC permutation. On both DistilBERT and DistilGPT2, mean AUPIC rises with distance from the optimum and then flattens, indicating a cluster of near-optimal orderings around the empirical minimum. pseudo-perplexity (DistilBERT) or perplexity (DistilGPT2), while the worst ordering produces only two interpolated models that improve on the student. The la… view at source ↗
Figure 4
Figure 4. Figure 4: Downstream interpolation curves across patching orders. For each of Qwen3-4B, Qwen3-8B, and Pythia-6.9B, the mean downstream accuracy is plotted against patched model size, across the 200 sampled orderings together with the first-to-last and last-to-first baselines. Shaded bands show the (25-75) inter-quartile range for the 200 orderings. First-to-last remains competitive but does not always match the best… view at source ↗
Figure 5
Figure 5. Figure 5: KLPatch interpolation curves on Qwen3-4B, Qwen3-8B, and Pythia-6.9B. For each model, mean downstream accuracy is plotted against the patched student size, comparing KLPatch (Algorithm 1) against first-to-last and last-to-first baselines. Across all three models, KLPatch tracks or outperforms last-to-first on both classification and generation tasks. students, and downstream task suite as in Section 5.3 (Qw… view at source ↗
Figure 6
Figure 6. Figure 6: KL divergence on Wikitext versus Wikitext perplexity. When KL divergence is computed on Wikitext, there is a strong correlation between the KL divergence between interpolated models and the teacher and Wikitext perplexity for DistilBERT and DistilGPT2 models. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: KL Path Length on Wikitext versus Wikitext AUPIC. For all possible patching orders, when KL divergence and perplexity are both computed on Wikitext, the KL path length in the interpolation graph correlates strongly with AUPIC. 0 1 2 KL Divergence 6 8 Pseudo-Perplexity DistilBERT Pearson r=0.975 0.0 0.2 0.4 0.6 0.8 KL Divergence 40 50 60 70 80 Perplexity DistilGPT2 Pearson r=0.934 [PITH_FULL_IMAGE:figures/… view at source ↗
Figure 8
Figure 8. Figure 8: KL divergence on the Pile versus Wikitext perplexity. When KL divergence is computed on a 64 example calibration set from the Pile, there is a strong correlation between the KL divergence between interpolated models and the teacher and Wikitext perplexity for DistilBERT and DistilGPT2 models. 4 6 8 10 12 KL Path Length 0.50 0.75 1.00 1.25 AUPIC DistilBERT Pearson r=0.930 2.5 3.0 3.5 KL Path Length 1.0 1.5 … view at source ↗
Figure 9
Figure 9. Figure 9: KL Path Length on the Pile versus Wikitext AUPIC. For all possible patching orders, when KL divergence is computed on a 64 example calibration set of the Pile and perplexity is computed on Wikitext, the KL path length in the interpolation graph correlates strongly with AUPIC. G.2 Optimal Patching Order Closely Approximates Optimal Interpolation In Section 4.1, we introduce the optimal interpolation problem… view at source ↗
Figure 10
Figure 10. Figure 10: Comparison of minimum AUPIC permutation vs interpolation for DistilBERT and DistilGPT2. The minimum permutation produces the same perplexity as the minimum interpolation for DistilBERT and perplexity close to the minimum interpolation for DistilGPT2. Shaded bands show the (25-75) inter-quartile range over all 720 orderings. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Comparison of maximum AUIC permutation vs interpolation for Qwen3-4B-Base, Qwen3-8B-Base, and Pythia-6.9b over 200 random orderings. Across models, the maximum AUIC permutation has classification accuracy very close to that of the maximum interpolation. Shaded bands show the (25-75) inter-quartile range over all 200 orderings. 3.0 3.5 4.0 4.5 Parameter Count (Billions) 10 15 20 Wikitext Perplexity ( ↓ ) Q… view at source ↗
Figure 12
Figure 12. Figure 12 [PITH_FULL_IMAGE:figures/full_fig_p023_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: KLPatch finds patching orders that are close to optimal: Visual representation of the optimal patching order between the student S and teacher T along the KL divergence to the teacher model, compared to the KLPatch order, front-to-back and back-to-front, and random permutations. H Additional Permutation Results We present extended results from Sections 5.1 and 5.3 on permutations of patching orders for mo… view at source ↗
Figure 14
Figure 14. Figure 14: AUPIC distribution across patching order permutations for DistilBERT and Distil￾GPT2. Across the 720 orderings, last-to-first patching is at or near the optimum on both models, suggesting it is a simple and reliable default recipe for model size interpolation. 0.3 0.4 0.5 AUIC (↑) 0 5 10 15 20 Count Qwen3-4B-Base AUIC Distribution 0.4 0.5 AUIC (↑) Qwen3-8B-Base AUIC Distribution 0.0 0.2 0.4 AUIC (↑) Pythi… view at source ↗
Figure 15
Figure 15. Figure 15: Distribution of AUIC for different patching orders. Across 200 orderings, first to last and last to first patching have relatively high AUIC, but last-to-first patching does not have the highest AUIC for Qwen models despite having the lowest AUPIC. 0.6 0.8 AUPIC (↓) 0 5 10 15 Count Qwen3-4B-Base AUPIC Distribution 1 2 AUPIC (↓) Qwen3-8B-Base AUPIC Distribution 0.75 1.00 1.25 1.50 AUPIC (↓) Pythia-6.9b AUP… view at source ↗
Figure 16
Figure 16. Figure 16: Distribution of AUPIC for different patching orders. Across the 200 orderings, either last to first or first to last patching often yields the highest AUPIC, suggesting it is a simple and reliable default recipe for model size interpolation. 3.0 3.5 4.0 4.5 Parameter Count (Billions) 10 15 20 Wikitext Perplexity ( ↓ ) Qwen3-4B-Base 5 6 7 8 Parameter Count (Billions) 20 40 Wikitext Perplexity ( ↓ ) Qwen3-8… view at source ↗
Figure 17
Figure 17. Figure 17: Wikitext perplexity for different patching orders. Across 200 orderings, either last to first patching and first to last patching achieve smooth interpolation performance, suggesting that they are generalizable recipes that work well across models. On the other hand, we observe that a naive patching order can sometimes result in poor interpolation performance. Shaded bands show the (25-75) inter-quartile … view at source ↗
Figure 18
Figure 18. Figure 18: shows that KLPatch closely tracks the best performing permutation and achieves better model size interpolation compared to last to first and first to last patching when we patch fewer student layers, but eventually converges to the best performing patching order trajectory on the generation tasks. 3.0 3.5 4.0 4.5 Parameter Count (Billions) 0.1 0.2 0.3 0.4 Generation Accuracy ( ↑ ) Qwen3-4B-Base 5 6 7 8 Pa… view at source ↗
Figure 19
Figure 19. Figure 19: shows that KLPatch often tracks closely to the best performing patching order among last to first and first to last patching orders on Wikitext. 3.0 3.5 4.0 4.5 Parameter Count (Billions) 10.0 12.5 15.0 17.5 Wikitext Perplexity ( ↓ ) Qwen3-4B-Base 5 6 7 8 Parameter Count (Billions) 10 15 20 25 Wikitext Perplexity ( ↓ ) Qwen3-8B-Base 4 5 6 7 Parameter Count (Billions) 15 20 25 Wikitext Perplexity ( ↓ ) Pyt… view at source ↗
Figure 20
Figure 20. Figure 20: shows that KLPatch severely underperforms first to last patching, suggesting that KLPatch might not universally improve over naive patching orders. However, after enforcing a heuristic constraint that the first student layer (layer 1) be patched first, then the KLPatch algorithm recovers the first to last patching. One possible reason for this discrepancy could be the initialization of the student model. … view at source ↗
Figure 21
Figure 21. Figure 21: KLPatch for Llama-3.2-3B by first patching Layer 1. Enforcing layer 1 as the first patched layer allows KLPatch to recover the optimal patching order and improves performance by avoiding an initially suboptimal patching choice. K KLPatch Ablations We show additional ablations on the KLPatch algorithm, including using metrics other than KL Divergence (K.1) and a non-iterative greedy approach (K.2) K.1 KLPa… view at source ↗
Figure 22
Figure 22. Figure 22: Change in classification accuracy with respect to KLPatch for ablated metrics. Ablating the metrics used to compute ℓi in Algorithm 1 produces interpolation curves worse than or on par with KLPatch on classification tasks. 3.0 3.5 4.0 4.5 Parameter Count (Billions) −0.10 −0.05 0.00 ∆ Generation Accuracy ( ↑ ) Qwen3-4B-Base 5 6 7 8 Parameter Count (Billions) −0.10 −0.05 0.00 ∆ Generation Accuracy ( ↑ ) Qwe… view at source ↗
Figure 23
Figure 23. Figure 23: Change in generation accuracy for KLPatch with ablated metrics. For generation tasks, cosine patching and perplexity patching baselines have worse or similar performance to KLPatch across models. 3.0 3.5 4.0 4.5 Parameter Count (Billions) −1 0 1 ∆ Wikitext Perplexity ( ↓ ) Qwen3-4B-Base 5 6 7 8 Parameter Count (Billions) 0 5 ∆ Wikitext Perplexity ( ↓ ) Qwen3-8B-Base 4 5 6 7 Parameter Count (Billions) 0 5 … view at source ↗
Figure 24
Figure 24. Figure 24: Change in wikitext perplexity with ablated metrics for KLPatch. KLPatch has similar or better wikitext perplexity to other metrics. 29 [PITH_FULL_IMAGE:figures/full_fig_p029_24.png] view at source ↗
Figure 25
Figure 25. Figure 25: Difference in classification accuracy between non-iterative approaches and KLPatch. KLPatch has comparable or better performance than non-iterative approaches on classification tasks. 3.0 3.5 4.0 4.5 Parameter Count (Billions) −0.1 0.0 ∆ Generation Accuracy ( ↑ ) Qwen3-4B-Base 5 6 7 8 Parameter Count (Billions) −0.15 −0.10 −0.05 0.00 ∆ Generation Accuracy ( ↑ ) Qwen3-8B-Base 4 5 6 7 Parameter Count (Billi… view at source ↗
Figure 26
Figure 26. Figure 26: Difference in generation accuracy between non-iterative approaches and KLPatch. KLPatch has better performance than non-iterative approaches on generation tasks. 30 [PITH_FULL_IMAGE:figures/full_fig_p030_26.png] view at source ↗
Figure 27
Figure 27. Figure 27: Difference in wikitext perplexity between non-iterative approaches and KLPatch. KLPatch has similar or better wikitext perplexity to non-iterative approaches. K.3 Ablating the calibration set We test varying calibration set sizes |Dcal| used for evaluating KL divergence in the KLPatch algorithm and report the mean and standard deviation over 5 random seeds in [PITH_FULL_IMAGE:figures/full_fig_p031_27.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

54 extracted references · 54 canonical work pages · 18 internal anchors

  1. [1]

    Ainsworth, Jonathan Hayase, and Siddhartha S

    Samuel K. Ainsworth, Jonathan Hayase, and Siddhartha S. Srinivasa. Git re-basin: Merging models modulo permutation symmetries. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023. URL https://openreview.net/pdf?id=CQsmMYmlP5T

    work page 2023

  2. [2]

    Pythia: A suite for analyzing large language models across training and scaling

    Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling. In Andreas Krause, Emma Brunskill, Kyung...

    work page 2023

  3. [3]

    PIQA: reasoning about physical commonsense in natural language

    Yonatan Bisk, Rowan Zellers, Ronan LeBras, Jianfeng Gao, and Yejin Choi. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in ...

    work page 2020

  4. [4]

    LL amaflex: Many-in-one LLM s via generalized pruning and weight sharing

    Ruisi Cai, Saurav Muralidharan, Hongxu Yin, Zhangyang Wang, Jan Kautz, and Pavlo Molchanov. LL amaflex: Many-in-one LLM s via generalized pruning and weight sharing. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=AyC4uxx2HW

    work page 2025

  5. [5]

    B ool Q : Exploring the surprising difficulty of natural yes/no questions

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. B ool Q : Exploring the surprising difficulty of natural yes/no questions. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics:...

    work page doi:10.18653/v1/n19-1300 2019

  6. [6]

    Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018. URL https://arxiv.org/abs/1803.05457

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  7. [7]

    Training Verifiers to Solve Math Word Problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL https://arxiv.org/abs/2110.14168

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  8. [8]

    BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT : Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, V...

    work page doi:10.18653/v1/n19-1423 2019

  9. [9]

    The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks

    Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks. arXiv preprint arXiv:2110.06296, 2021

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  10. [10]

    Linear mode connectivity and the lottery ticket hypothesis

    Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research, pages 3259--3269. PMLR , 2020. URL http://proc...

    work page 2020

  11. [11]

    The Pile: An 800GB Dataset of Diverse Text for Language Modeling

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling, 2021. URL https://arxiv.org/abs/2101.00027

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  12. [12]

    A framework for few-shot language model evaluation, 2023

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework...

    work page arXiv 2023

  13. [13]

    The Llama 3 Herd of Models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. ArXiv preprint, abs/2407.21783, 2024. URL https://arxiv.org/abs/2407.21783

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  14. [14]

    Song Han, Jeff Pool, John Tran, and William J. Dally. Learning both weights and connections for efficient neural network. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett, editors, Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 201...

    work page 2015

  15. [15]

    Amc: Automl for model compression and acceleration on mobile devices

    Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. Amc: Automl for model compression and acceleration on mobile devices. In Proceedings of the European conference on computer vision (ECCV), pages 784--800, 2018

    work page 2018

  16. [16]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021 a . URL https://openreview.net/forum?id=d7KBjmI3GmQ

    work page 2021

  17. [17]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021 b

    work page 2021

  18. [18]

    Rethinking layer relevance in large language models beyond cosine similarity

    Cristian Hinostroza, Rodrigo Toro Icarte, Christ Devia, Andres Carvallo De Ferari, Eugenio Herrera-Berg, Denis Parra, and Jorge F Silva. Rethinking layer relevance in large language models beyond cosine similarity. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=mRLnS8jQWt

    work page 2026

  19. [19]

    Distilling the Knowledge in a Neural Network

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. ArXiv preprint, abs/1503.02531, 2015. URL https://arxiv.org/abs/1503.02531

    work page internal anchor Pith review Pith/arXiv arXiv 2015

  20. [20]

    Training Compute-Optimal Large Language Models

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, DDL Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 10, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  21. [21]

    Instruction-Following Pruning for Large Language Models

    Bairu Hou, Qibin Chen, Jianyu Wang, Guoli Yin, Chong Wang, Nan Du, Ruoming Pang, Shiyu Chang, and Tao Lei. Instruction-following pruning for large language models. arXiv preprint arXiv:2501.02086, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  22. [22]

    O'Reilly Media, Inc

    Chip Huyen. Designing machine learning systems. " O'Reilly Media, Inc.", 2022

    work page 2022

  23. [23]

    Nayak, Jonathan Geuter, Marco Fumero, Francesco Locatello, and David Alvarez-Melis

    Sara Kangaslahti, Nihal V. Nayak, Jonathan Geuter, Marco Fumero, Francesco Locatello, and David Alvarez-Melis. Boomerang distillation enables zero-shot model size interpolation. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=4ZU8v4s3IR

    work page 2026

  24. [24]

    Scaling Laws for Neural Language Models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    work page internal anchor Pith review Pith/arXiv arXiv 2001

  25. [25]

    Nayak, Jack Merullo, Stephen Bach, Chen Sun, and Ellie Pavlick

    Apoorv Khandelwal, Tian Yun, Nihal V. Nayak, Jack Merullo, Stephen Bach, Chen Sun, and Ellie Pavlick. \ 100k or 100 days: Trade-offs when pre-training with academic resources. In Second Conference on Language Modeling, 2025. URL https://openreview.net/forum?id=EFxC34XbDh

    work page 2025

  26. [26]

    RACE : Large-scale R e A ding comprehension dataset from examinations

    Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. RACE : Large-scale R e A ding comprehension dataset from examinations. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel, editors, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785--794, Copenhagen, Denmark, 2017. Association for Computation...

    work page doi:10.18653/v1/d17-1082 2017

  27. [27]

    Optimal brain damage

    Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989

    work page 1989

  28. [28]

    Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Models

    Hong Liu, Sang Michael Xie, Zhiyuan Li, and Tengyu Ma. Same pre-training loss, better downstream: Implicit bias matters for language models. 2023. doi:10.48550/arXiv.2210.14199

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2210.14199 2023

  29. [29]

    Amir M. Mansourian, Rozhan Ahmadi, Masoud Ghafouri, Amir Mohammad Babaei, Elaheh Badali Golezani, Zeynab yasamani ghamchi, Vida Ramezanian, Alireza Taherian, Kimia Dinashi, Amirali Miri, and Shohreh Kasaei. A comprehensive survey on knowledge distillation. Transactions on Machine Learning Research, 2025. ISSN 2835-8856. URL https://openreview.net/forum?id...

    work page 2025

  30. [30]

    ShortGPT: Layers in Large Language Models are More Redundant Than You Expect

    Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. Shortgpt: Layers in large language models are more redundant than you expect. ArXiv preprint, abs/2403.03853, 2024. URL https://arxiv.org/abs/2403.03853

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  31. [31]

    Pointer sentinel mixture models

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings . OpenReview.net, 2017. URL https://openreview.net/forum?id=Byj72udxe

    work page 2017

  32. [32]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun ' ichi Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381--2391, Brussels, Belgiu...

    work page doi:10.18653/v1/d18-1260 2018

  33. [33]

    Compact language models via pruning and knowledge distillation

    Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov. Compact language models via pruning and knowledge distillation. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, editors,...

    work page arXiv 2024

  34. [34]

    Uniform convergence may be unable to explain generalization in deep learning

    Vaishnavh Nagarajan and J Zico Kolter. Uniform convergence may be unable to explain generalization in deep learning. Advances in neural information processing systems, 32, 2019

    work page 2019

  35. [35]

    An analysis of approximations for maximizing submodular set functions

    G L Nemhauser, L A Wolsey, and M L Fisher. An analysis of approximations for maximizing submodular set functions. 14: 0 265--294, 1978. ISSN 0025-5610,1436-4646. doi:10.1007/bf01588971

    work page doi:10.1007/bf01588971 1978

  36. [36]

    Interpreting gpt: The logit lens, 2020

    nostalgebraist. Interpreting gpt: The logit lens, 2020. URL https://www. lesswrong.com/posts/AcKRB8wDpdaN6v6ru/ interpreting-gpt-the-logit-lens

    work page 2020

  37. [37]

    Pytorch: An imperative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas K \" o pf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-...

    work page 2019

  38. [38]

    Language models are unsupervised multitask learners

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI, 2019. URL https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf

    work page 2019

  39. [39]

    Winogrande: An adversarial winograd schema challenge at scale

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Art...

    work page 2020

  40. [40]

    DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

    Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. ArXiv preprint, abs/1910.01108, 2019. URL https://arxiv.org/abs/1910.01108

    work page internal anchor Pith review Pith/arXiv arXiv 1910

  41. [41]

    Model fusion via optimal transport

    Sidak Pal Singh and Martin Jaggi. Model fusion via optimal transport. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria - Florina Balcan, and Hsuan - Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL htt...

    work page 2020

  42. [42]

    LLM Pruning and Distillation in Practice: The Minitron Approach

    Sharath Turuvekere Sreenivas, Saurav Muralidharan, Raviraj Joshi, Marcin Chochowski, Ameya Sunil Mahabaleshwarkar, Gerald Shen, Jiaqi Zeng, Zijia Chen, Yoshi Suhara, Shizhe Diao, et al. Llm pruning and distillation in practice: The minitron approach. ArXiv preprint, abs/2408.11796, 2024. URL https://arxiv.org/abs/2408.11796

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  43. [43]

    Zico Kolter

    Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. A simple and effective pruning approach for large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024. URL https://openreview.net/forum?id=PxoFut3dWW

    work page 2024

  44. [44]

    The bitter lesson

    Rich Sutton. The bitter lesson. 2019

    work page 2019

  45. [45]

    Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao, SH Cai, Yuan Cao, Y Charles, HS Che, Cheng Chen, Guanduo Chen, et al. Kimi k2. 5: Visual agentic intelligence. arXiv preprint arXiv:2602.02276, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  46. [46]

    Qwen3.5: Accelerating productivity with native multimodal agents, February 2026

    Qwen Team. Qwen3.5: Accelerating productivity with native multimodal agents, February 2026. URL https://qwen.ai/blog?id=qwen3.5

    work page 2026

  47. [47]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019. URL https://openreview.net/forum?id=rJ4km2R5t7

    work page 2019

  48. [48]

    HuggingFace's Transformers: State-of-the-art Natural Language Processing

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R \'e mi Louf, Morgan Funtowicz, and Jamie Brew. Huggingface's transformers: State-of-the-art natural language processing. ArXiv preprint, abs/1910.03771, 2019. URL https://arxiv.org/abs/1910.03771

    work page internal anchor Pith review Pith/arXiv arXiv 1910

  49. [49]

    Sheared llama: Accelerating language model pre-training via structured pruning

    Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared llama: Accelerating language model pre-training via structured pruning. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024. URL https://openreview.net/forum?id=09iOdaeOzp

    work page 2024

  50. [50]

    Qwen3 Technical Report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. ArXiv preprint, abs/2505.09388, 2025. URL https://arxiv.org/abs/2505.09388

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  51. [51]

    Model merging in llms, mllms, and beyond: Methods, theories, applications, and opportunities

    Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, Jie Zhang, and Dacheng Tao. Model merging in llms, mllms, and beyond: Methods, theories, applications, and opportunities. ACM Comput. Surv., 58 0 (8), February 2026. ISSN 0360-0300. doi:10.1145/3787849. URL https://doi.org/10.1145/3787849

    work page doi:10.1145/3787849 2026

  52. [52]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. H ella S wag: Can a machine really finish your sentence? In Anna Korhonen, David Traum, and Llu \' s M \`a rquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791--4800, Florence, Italy, 2019. Association for Computational Li...

    work page doi:10.18653/v1/p19-1472 2019

  53. [53]

    FinerCut: Finer-grained Interpretable Layer Pruning for Large Language Models

    Yang Zhang, Yawei Li, Xinpeng Wang, Qianli Shen, Barbara Plank, Bernd Bischl, Mina Rezaei, and Kenji Kawaguchi. Finercut: Finer-grained interpretable layer pruning for large language models. arXiv preprint arXiv:2405.18218, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  54. [54]

    Instruction-Following Evaluation for Large Language Models

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models, 2023. URL https://arxiv.org/abs/2311.07911

    work page internal anchor Pith review Pith/arXiv arXiv 2023