Pith. sign in

REVIEW 2 major objections 8 minor 1 cited by

Lillama: Large Language Models Compression via Low-Rank Feature Distillation

T0 review · 2 major / 8 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A one-shot local low-rank feature distillation, initialized with SVD and trained with a joint teacher-plus-student activation loss, can remove billions of parameters from large language models in minutes on a single GPU while keeping most…

desk verdict A practical one-shot compression recipe that mostly delivers; the 97% retention line overstates Table 2 and the error-propagation story is thin, but the empirical results are broad and credible. read the letter →

arxiv 2412.16719 v2 pith:R5VJ4QCK submitted 2024-12-21 cs.LG cs.AI

Yaya Sy , Christophe Cerisara , Irina Illina This is my paper · ORCID
classification cs.LGcs.AI
keywords LLMcompressionlow-rankapproximationfeaturedistillationstructuredpruningMixture-of-ExpertsMambasingularvaluedecompositionone-shot
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that the expensive continued-pretraining step in LLM compression is unnecessary. Its method, Lillama, compresses models in one pass by replacing full weight matrices with low-rank factors initialized by singular value decomposition and refining each compressed layer locally to match the original model's activations. The authors report that 20% compression retains 97% of zero-shot performance, that Mixtral-8x7B loses 10 billion parameters in minutes on a single A100 GPU, and that Phi-2 3B compressed by 40% needs only 13 million calibration tokens. If true, LLM compression becomes cheap enough to run on commodity hardware and to produce small specialized models without a full pretraining budget.

What carries the argument

The load-bearing mechanism is a local distillation loop around a low-rank factorization. Each selected weight matrix is replaced by a product of two smaller matrices initialized from the truncated SVD of the original weight, then trained layer by layer against the sum of two losses: one reproducing the preceding teacher layer's output and one reproducing the preceding student layer's output. Because updates are local, the full computation graph is never stored, which is what allows a 47B model to be compressed on a single GPU. The bottom-first variant of Algorithm 1 decides which layers to compress and lets early layers absorb the most aggressive rank reduction, so only part of the model must be loaded and forwarded during distillation.

What would settle it

Measure the teacher–student activation gap layer by layer after compressing a model by 20%, and separately compare compressing only the first layer against compressing only the last layer. If the gap grows with depth, or if compressing a single early layer destroys downstream accuracy while local losses stay low, then per-layer activation matching is not sufficient to preserve global behavior.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that one-shot low-rank feature distillation, without continued pretraining, can remove 20–40% of an LLM's parameters while keeping most of its behavior. Each compressed layer is initialized with SVD and trained with a joint loss that matches both the previous teacher layer's activations and the previous student layer's activations. With only 13 million calibration tokens the authors report 97% retention of average zero-shot accuracy at 20% compression across dense Transformers, a Mixture-of-Experts model, and Mamba state-space models; Mixtral-8x7B loses 10 billion parameters in under an hour on one A100 GPU; and a 40%-compressed Phi-2 3B matches similarly sized recent models.

Load-bearing premise

The whole method rests on the assumption that matching each compressed layer's activations to the original model's activations on a small calibration set keeps the rest of the model working, so that errors from independently compressed layers never compound.

Editorial extensions

If this is right

  • A 47B Mixtral model can lose 10 billion parameters and still fit on a single A100 GPU at short contexts, with zero-shot scores close to the original.
  • Compression no longer requires billions of tokens of continued pretraining: 13 million calibration tokens suffice for a 40% reduction of Phi-2 3B.
  • A 40%-compressed Phi-2 1.7B matches recently released models of similar size on the nine-task zero-shot benchmark, and fine-tuning on 191 million tokens improves recovery.
  • The method transfers beyond Transformers: a 20%-compressed Mamba 3B keeps 99% of its average zero-shot performance, and the same recipe compresses speech-recognition decoders by 37%.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If local activation matching is truly sufficient, the method should compose with quantization: low-rank distillation first, then quantizing the surviving weights, with the two information losses tuned jointly.
  • The bottom-first strategy contains an implicit claim that early layers tolerate more compression than later layers; a direct layer-sensitivity study could turn that into a general criterion for other pruning methods.
  • Because the recipe is architecture-agnostic and locally optimizes any module with low-rank activations, it could be pointed at embeddings and prediction heads of multilingual models, which the paper begins to explore with its Hausa experiment.
  • The one-shot nature opens a cheaper route to specialized small models: compress a large generalist, then fine-tune on a small domain dataset, rather than pretraining from scratch.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 8 minor

Summary. The paper proposes Lillama, a one-shot LLM compression method. It selects layers to compress (bottom-first, top-first, or uniform strategies; Algorithm 1), initializes their low-rank factorizations by SVD, and then optimizes each compressed module locally with a feature-distillation loss (Eq. 10) that matches teacher activations under both a teacher input and a student input; no global language-modeling loss and no continued pretraining are used. Compression targets are evaluated on nine zero-shot tasks for Phi-3 14B, Mixtral-8x7B, Phi-2 3B, Mistral-7B, Falcon-Mamba 7B, and Mamba 3B at 20% reduction (reported retentions 93-99%), with additional 40% compression results (with and without fine-tuning), a SliceGPT comparison, inference speed and memory tables, ablations of the loss choices, minimum-rank choice, and SVD initialization, plus a Whisper speech experiment and a low-resource-language embedding compression experiment.

Significance. If the results hold, the practical significance is high: with 13M calibration tokens and no continued pretraining, the method compresses a 47B Mixtral model on a single A100 GPU in under an hour, and it transfers across dense Transformer, MoE, and Mamba architectures. Credit is due for specific strengths: the joint loss in Eq. (10) has no weighting hyperparameter and is ablated (Table 8, Fig. 3); SVD initialization is isolated in Fig. 5; the released code makes the tables machine-checkable; and the comparison with SliceGPT (Table 7) and with similarly sized small models (Table 4) gives falsifiable evidence. The main reservation is statistical and quantitative: retention ratios are quoted to one percentage point from single runs, and the headline '97%' is contradicted by the table's own average of 94.7%, so the significance claim needs recalibration before these numbers can be used as stated.

major comments (2)
  1. [§6.2, Table 2, Abstract] The headline retention claims are not supported by the table's own numbers. Section 6.2 opens with 'Our method can retain 97% of the zero-shot performance' and the Table 2 caption repeats that claim, but the four 20%-compressed rows of Table 2 give retentions of 96.7% (Phi-3 14B), 95.5% (Mixtral), 93.5% (Phi-2), and 93.2% (Mistral), for an average of 94.7%. The abstract's Mixtral-specific 'retaining over 95%' therefore rests on a 95.5% figure, a margin comparable to the run-to-run noise of zero-shot evaluation; several per-task deltas are positive (e.g., Phi-3 14B TruthfulQA 57.63 to 57.88), which is consistent with roughly one-point noise. The paper reports a single run per configuration and no variance estimates, so retention ratios quoted to one percentage point are statistically unsupported. Please correct the Section 6.2 and caption wording (e.g., 'up to 97%' plus the actual range), add repeated runs or confidence intervals for the headline configurations, and soften the abstract margin claim accordingly.
  2. [§4, §5 (Algorithm 1), Eq. (10)] The 'no continued pretraining' claim rests on the sufficiency of local per-layer distillation, but the manuscript gives no evidence that the objective in Eq. (10) controls error propagation into the frozen upper layers. Under the bottom-first strategy (Algorithm 1), only early layers are distilled while the remaining layers are frozen and, at inference, consume hidden states produced by the compressed block; drift in those hidden states relative to the teacher is never measured. The L_S term of Eq. (10) conditions each compressed layer on the previous student output and thus makes the compressed block internally self-consistent, but no term constrains the distribution shift seen by the uncompressed tail. The empirical retentions are supportive but do not isolate this failure mode, since benchmark averages can mask per-layer drift that happens not to hurt the nine chosen tasks. Two concrete tests would resolve the concern: (i) report teacher-versus-student hidden-state drift (cosine similarity or CKA) at each layer boundary, including the uncompressed tail, on held-out data; (ii) on a small model such as Phi-2, compare the local objective against a variant that adds a global language-modeling loss during or after distillation. If the global variant does not improve retention, the adequacy of local matching is demonstrated; if it does, the 'local is sufficient' claim needs to be qualified.
minor comments (8)
  1. [Abstract, Appendix A.1 (Table 9)] The abstract says Mixtral-8x7B is compressed 'within minutes', but Table 9 reports 47 minutes for the bottom-first 20% compression; please say 'under an hour' or report the measured time in the abstract.
  2. [Table 7] The compression ratio for the Lillama row in Table 7 is not stated, while the SliceGPT row is at 24% reduction; please specify the Lillama ratio or align both rows at the same ratio so the comparison is well-defined.
  3. [Algorithm 1] The pseudocode does not specify whether each entry (r, W) in the stack R refers to the original pretrained matrix or to the matrix currently stored in the copy M', and because several entries for the same layer are processed consecutively (layer index increasing, rank decreasing), it is unclear whether the intermediate replacements affect the result; please clarify that only the smallest rank assigned to each layer survives or spell out the intended semantics.
  4. [§6.2, Table 5] The heading 'Compressed Models Show Good Recovery with Fine-Tuning' is not supported for Phi-2: fine-tuning the 40% compressed model improves the average by only 0.28 points (52.82 to 53.10) while decreasing several tasks (ARC-E 63.22 to 60.86, BoolQ 76.02 to 67.68); please temper the claim or discuss the task-level decreases.
  5. [Appendix A.2] The sentence 'Phi-3 14B and Phi-2 3B compressed at 30% retain 93% of their base performance' is not borne out by Table 11, which gives 92.7% and 91.8% respectively; please round consistently or state the exact values.
  6. [References (Xia et al.)] The ShearedLLaMA work appears twice, as Xia et al. (2023) and Xia et al. (2024), both citing arXiv:2310.06694 with overlapping content; please consolidate them into a single reference.
  7. [Eq. (7)] The unweighted sum of the L1 and cosine terms in Eq. (7) is advertised as a strength ('we don't introduce any hyperparameter'), but the two terms have different scales; a sentence reporting sensitivity to the relative weighting (or a justification for the fixed scales) would make the claim robust.
  8. [Table 8] For Phi-2 3B the Teacher and Teacher+Student rows report identical averages (57.38); please verify that these values were not copied, and if they are genuine, explain why the joint loss gives no gain on that particular model.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: compression targets activation matching and retention is measured on external benchmarks.

full rationale

The paper's derivation chain is empirical and self-contained. The method optimizes a per-layer activation-matching objective (Eqs. 7-10) between low-rank student modules and pretrained teacher modules, with SVD initialization. The headline claims — 95-99% zero-shot retention, single-A100 runtime, generalization to Mamba — are all measured outcomes on external benchmarks (lm-evaluation-harness), not quantities implied by the fitted low-rank matrices. No equation in the paper defines benchmark accuracy as a function of the fitted parameters, so no prediction reduces to a fit by construction. Hyperparameters such as the minimum rank k and the bottom-first strategy are empirically chosen and ablated, but the reported scores are not defined in terms of those choices. The motivating premise that activations are more low-rank than weights is attributed to prior external work (Chen et al., 2021; Yu and Wu, 2023) and independently reproduced in Appendix A.5, so it is not a self-imported uniqueness claim. The loss in Eq. 7 is adopted from Chang et al. (2022), also external. The paper contains no self-citations that are load-bearing. The identified risk that local distillation may not control error propagation through the stack is a correctness concern, not a circularity: nothing in the objective is defined in terms of the final benchmark results.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The method's free parameters are a small set of hyperparameters and data choices; there are no invented entities. The key load-bearing assumptions are empirical: activation low-rankness and the sufficiency of local distillation.

free parameters (4)
  • minimum rank k in Algorithm 1 = 1024 for most models; 1536 for Phi-3 14B; 2048 for Phi-3 14B at >20% compression
    Chosen by hand after observing that higher ranks give better accuracy (Figure 4); it controls how much lower layers are compressed and how many layers are distilled.
  • rank increment m = 256
    Used to generate candidate ranks in Algorithm 1; selected without a systematic search.
  • learning rate = 8.6e-4
    AdamW learning rate for all local distillation runs (Appendix A.10); selected by preliminary experiments.
  • calibration dataset and size = 13 million tokens from SlimOrca; 191M tokens for fine-tuning
    The authors tested RedPajama and found instruction data better; 13M tokens is a design choice that drives the low-data claim.
assumptions (5)
  • standard math SVD provides the optimal Frobenius-norm low-rank approximation (Eckart-Young theorem, used in Eq. 2).
    Invoked in Section 3.1 to initialize A and B from the pretrained weight matrix; this is standard.
  • domain assumption Transformer and Mamba layer activations are sufficiently low-rank that approximating them with rank-r factor matrices can preserve downstream accuracy after distillation.
    The paper reproduces stable-rank plots in Appendix A.5 and cites Yu and Wu 2023; it is load-bearing because local activation matching is the whole objective.
  • domain assumption Per-module activation matching with the sum of Teacher and Student losses transfers the information needed by the full model without a global objective.
    Core assumption of Section 4, Eqs. (7)-(10); no formal error-propagation bound is provided.
  • domain assumption SlimOrca instruction examples are a sufficient calibration distribution for the nine zero-shot evaluation tasks.
    Calibration data choice in Section 6.1; the model never sees the evaluation tasks during distillation except through this proxy distribution.
  • domain assumption SVD of pretrained weights is a good initialization for activation-matching low-rank factors.
    Used in Section 4 and Figure 5 to accelerate convergence; the paper shows empirically that it helps but gives no theoretical guarantee.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Lillama: Large Language Models Compression via Low-Rank Feature Distillation." pith.science (2026). https://pith.science/paper/R5VJ4QCK

@misc{pith2026241216719,
  author       = {Pith},
  title        = {Pith review of: Lillama: Large Language Models Compression via Low-Rank Feature Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/R5VJ4QCK}},
  note         = {Machine review of arXiv:2412.16719}
}
read the original abstract

Current LLM structured pruning methods typically involve two steps: (1) compression with calibration data and (2) costly continued pretraining on billions of tokens to recover lost performance. This second step is necessary as the first significantly impacts model accuracy. Prior research suggests pretrained Transformer weights aren't inherently low-rank, unlike their activations, which may explain this drop. Based on this observation, we propose Lillama, a compression method that locally distills activations with low-rank weights. Using SVD for initialization and a joint loss combining teacher and student activations, we accelerate convergence and reduce memory use with local gradient updates. Lillama compresses Mixtral-8x7B within minutes on a single A100 GPU, removing 10 billion parameters while retaining over 95% of its original performance. Phi-2 3B can be compressed by 40% with just 13 million calibration tokens, resulting in a small model that competes with recent models of similar size. The method generalizes well to non-transformer architectures, compressing Mamba-3B by 20% while maintaining 99% performance.

Figures

Figures reproduced from arXiv: 2412.16719 by the authors.

Figure 1
Figure 1. Lillama approach: STEP 1 selects layers to compress for a target compression ratio (e.g., N%) using various strategies (see Section 5). STEP 2 compresses and initializes the chosen parameters via SVD. STEP 3 distills the low-rank weights with a small calibration dataset. magnitude pruning algorithm as done in Fran￾kle and Carbin (2019), is too costly to be ap￾plied to large-scale LLMs. 2. It must achieve fast conver… view at source ↗
Figure 2
Figure 2. Accelerating convergence by learning from Teacher and Student activations through a joint loss. We propose to study the effect of three distillation strate￾gies: (a) Teacher: the input to the compressed student layer comes from the output of the previous teacher layer; (b) Student: the input to the compressed student layer comes from the output of the previous student layer; (c) Teacher+Student: the compressed stude… view at source ↗
Figure 3
Figure 3. The joint loss converges generally better. Convergence of the three losses illustrated in [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: Convergence when initializing low-rank weight randomly [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Activations are low-rank. Comparison of the stable rank of weights (*_proj) and layer activations (layer_activation). Each bar is the average stable rank across all layers. A.7 Example of generated texts We also evaluated the compressed models by ob￾serving the text th…
Figure 7
Figure 7. Figure 7: An example of PyTorch implementation of our approach with the Teacher loss (see Figure [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Illustration of 20% compression when using [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CALR: Corrective Adaptive Low-Rank Decomposition for Efficient Large Language Model Layer Compression

    cs.LG 2025-08 unverdicted novelty 4.0 of 10

    CALR adds a parallel learnable low-rank module to recover functional performance lost by SVD-based compression of LLM weight matrices.

Reference graph

Works this paper leans on

47 extracted references · 5 canonical work pages · cited by 1 Pith paper

  1. [1]

    David Ifeoluwa Adelani, Jessica Ojo, Israel Abebe Azime, Jian Yun Zhuang, Jesujoba O. Alabi, Xuanli He, Millicent Ochieng, Sara Hooker, Andiswa Bukula, En-Shiun Annie Lee, Chiamaka Chukwuneke, Happy Buzaaba, Blessing Sibanda, Godson Kalipe, Jonathan Mukiibi, Salomon Kabongo, Foutse Yuehgoh, Mmasibidi Setaka, Lolwethu Ndolela, Nkiruka Odu, Rooweither Mabuy...

    arXiv 2024

  2. [2]

    Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. 2020. https://arxiv.org/abs/2012.13255 Intrinsic dimensionality explains the effectiveness of language model fine-tuning . Preprint, arXiv:2012.13255

    arXiv 2020

  3. [3]

    Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman

    Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. 2024. http://arxiv.org/abs/2401.15024 SliceGPT : Compress Large Language Models by Deleting Rows and Columns . arXiv preprint. ArXiv:2401.15024 [cs]

    arXiv 2024

  4. [4]

    Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. https://arxiv.org/abs/2006.11477 wav2vec 2.0: A framework for self-supervised learning of speech representations . Preprint, arXiv:2006.11477

    arXiv 2020

  5. [5]

    Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence

    2020

  6. [6]

    Heng-Jui Chang, Shu wen Yang, and Hung yi Lee. 2022. https://arxiv.org/abs/2110.01900 Distilhubert: Speech representation learning by layer-wise distillation of hidden-unit bert . Preprint, arXiv:2110.01900

    arXiv 2022

  7. [7]

    Xuankai Chang, Takashi Maekaku, Pengcheng Guo, Jing Shi, Yen-Ju Lu, Aswin Shanmugam Subramanian, Tianzi Wang, Shu wen Yang, Yu Tsao, Hung yi Lee, and Shinji Watanabe. 2021. https://arxiv.org/abs/2110.04590 An exploration of self-supervised pretrained representations for end-to-end speech recognition . Preprint, arXiv:2110.04590

    work page Pith review arXiv 2021

  8. [8]

    Patrick Chen, Hsiang-Fu Yu, Inderjit Dhillon, and Cho-Jui Hsieh. 2021. https://proceedings.neurips.cc/paper_files/paper/2021/file/f56de5ef149cf0aedcc8f4797031e229-Paper.pdf Drone: Data-aware low-rank compression for large nlp models . In Advances in Neural Information Processing Systems, volume 34, pages 29321--29334. Curran Associates, Inc

    work page 2021

Show all 47 references
  1. [9]

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. https://api.semanticscholar.org/CorpusID:165163607 Boolq: Exploring the surprising difficulty of natural yes/no questions . ArXiv, abs/1905.10044

    2019 arXiv

  2. [10]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv, abs/1803.05457

    2018 arXiv

  3. [11]

    Together Computer. 2023. https://github.com/togethercomputer/RedPajama-Data Redpajama: An open source recipe to reproduce llama training dataset

    2023

  4. [12]

    Alexis Conneau, Min Ma, Simran Khanuja, Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna. 2022. https://arxiv.org/abs/2205.12446 Fleurs: Few-shot learning evaluation of universal representations of speech . Preprint, arXiv:2205.12446

    2022 arXiv

  5. [13]

    Jonathan Frankle and Michael Carbin. 2019. https://arxiv.org/abs/1803.03635 The lottery ticket hypothesis: Finding sparse, trainable neural networks . Preprint, arXiv:1803.03635

    2019 arXiv

  6. [14]

    Elias Frantar and Dan Alistarh. 2023. https://arxiv.org/abs/2301.00774 Sparsegpt: Massive language models can be accurately pruned in one-shot . Preprint, arXiv:2301.00774

    2023 arXiv

  7. [15]

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2023 doi

  8. [16]

    Albert Gu and Tri Dao. 2024. https://arxiv.org/abs/2312.00752 Mamba: Linear-time sequence modeling with selective state spaces . Preprint, arXiv:2312.00752

    2024 arXiv

  9. [17]

    Song Han, Jeff Pool, John Tran, and William J. Dally. 2015. https://arxiv.org/abs/1506.02626 Learning both weights and connections for efficient neural networks . Preprint, arXiv:1506.02626

    2015 arXiv

  10. [18]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  11. [19]

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne...

    2024 arXiv

  12. [20]

    Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020. http://arxiv.org/abs/1909.10351 TinyBERT : Distilling BERT for Natural Language Understanding . arXiv preprint. ArXiv:1909.10351 [cs]

    2020 arXiv

  13. [21]

    Jungo Kasai, Nikolaos Pappas, Hao Peng, James Cross, and Noah A. Smith. 2021. https://arxiv.org/abs/2006.10369 Deep encoder, shallow decoder: Reevaluating non-autoregressive machine translation . Preprint, arXiv:2006.10369

    2021 arXiv

  14. [22]

    Ayush Kaushal, Tejas Vaidhya, and Irina Rish. 2023. http://arxiv.org/abs/2309.14021 LORD : Low Rank Decomposition Of Monolingual Code LLMs For One - Shot Compression . arXiv preprint. ArXiv:2309.14021 [cs]

    2023 arXiv

  15. [23]

    Yann LeCun, John Denker, and Sara Solla. 1989. https://proceedings.neurips.cc/paper_files/paper/1989/file/6c9882bbac1c7093bd25041881277658-Paper.pdf Optimal brain damage . In Advances in Neural Information Processing Systems, volume 2. Morgan-Kaufmann

    1989

  16. [24]

    Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. 2018. https://arxiv.org/abs/1804.08838 Measuring the intrinsic dimension of objective landscapes . Preprint, arXiv:1804.08838

    2018 arXiv

  17. [25]

    Wing Lian, Guan Wang, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium". 2023. https://https://huggingface.co/Open-Orca/SlimOrca Slimorca: An open dataset of gpt-4 augmented flan reasoning traces, with verification

    2023

  18. [26]

    Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. https://doi.org/10.18653/v1/2022.acl-long.229 T ruthful QA : Measuring how models mimic human falsehoods . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pa...

    2022 doi

  19. [27]

    James Liu, Pragaash Ponnusamy, Tianle Cai, Han Guo, Yoon Kim, and Ben Athiwaratkun. 2024. https://arxiv.org/abs/2408.14690 Training-free activation sparsity in large language models . Preprint, arXiv:2408.14690

    2024 arXiv

  20. [28]

    Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. 2020. https://arxiv.org/abs/2007.08124 Logiqa: A challenge dataset for machine reading comprehension with logical reasoning . Preprint, arXiv:2007.08124

    2020 arXiv

  21. [29]

    Xinyin Ma, Gongfan Fang, and Xinchao Wang. 2023. https://arxiv.org/abs/2305.11627 Llm-pruner: On the structural pruning of large language models . Preprint, arXiv:2305.11627

    2023 arXiv

  22. [30]

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP

    2018

  23. [31]

    Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. 2023. https://arxiv.org/abs/2306.02707 Orca: Progressive learning from complex explanation traces of gpt-4 . Preprint, arXiv:2306.02707

    2023 arXiv

  24. [32]

    Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. https://doi.org/10.1109/ICASSP.2015.7178964 Librispeech: An asr corpus based on public domain audio books . In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), page...

    2015

  25. [33]

    Ankita Pasad, Ju-Chieh Chou, and Karen Livescu. 2022. https://arxiv.org/abs/2107.04734 Layer-wise analysis of a self-supervised speech representation model . Preprint, arXiv:2107.04734

    2022 arXiv

  26. [34]

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2022. https://arxiv.org/abs/2212.04356 Robust speech recognition via large-scale weak supervision . Preprint, arXiv:2212.04356

    2022 arXiv

  27. [35]

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Winogrande: An adversarial winograd schema challenge at scale. arXiv preprint arXiv:1907.10641

    2019 arXiv

  28. [36]

    Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019. Social iqa: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Na...

    2019

  29. [37]

    Shivalika Singh, Freddie Vargus, Daniel Dsouza, Börje F. Karlsson, Abinaya Mahendiran, Wei-Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura OMahony, Mike Zhang, Ramith Hettiarachchi, Joseph Wilson, Marina Machado, Luisa Souza Moura, Dominik Krzemiński, Hakimeh F...

    2024 arXiv

  30. [38]

    Sharath Turuvekere Sreenivas, Saurav Muralidharan, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov. 2024. https://arxiv.org/abs/2408.11796 Llm pruning and distillation in practice: The minitron approach . Pre...

    2024 arXiv

  31. [39]

    Zico Kolter

    Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. 2024. https://arxiv.org/abs/2306.11695 A simple and effective pruning approach for large language models . Preprint, arXiv:2306.11695

    2024 arXiv

  32. [40]

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan...

    2024 arXiv

  33. [41]

    Xing, and Fahad Shahbaz Khan

    Omkar Thawakar, Ashmal Vayani, Salman Khan, Hisham Cholakkal, Rao Muhammad Anwer, Michael Felsberg, Timothy Baldwin, Eric P. Xing, and Fahad Shahbaz Khan. 2024. https://arxiv.org/abs/2402.16840 Mobillama: Towards accurate and lightweight fully transparent gpt . Preprint, arXiv...

    2024 arXiv

  34. [42]

    Atnafu Lambebo Tonja, Bonaventure FP Dossou, Jessica Ojo, Jenalea Rajab, Fadel Thior, Eric Peter Wairagala, Aremu Anuoluwapo, Pelonomi Moiloa, Jade Abbott, Vukosi Marivate, et al. 2024. Inkubalm: A small language model for low-resource african languages. arXiv preprint arXiv:2...

    2024 arXiv

  35. [43]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

    2020 arXiv

  36. [45]

    Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. 2024. https://arxiv.org/abs/2310.06694 Sheared llama: Accelerating language model pre-training via structured pruning . Preprint, arXiv:2310.06694

    2024 arXiv

  37. [46]

    Hao Yu and Jianxin Wu. 2023. https://doi.org/10.1609/aaai.v37i9.26304 Compressing Transformers : Features Are Low - Rank , but Weights Are Not ! Proceedings of the AAAI Conference on Artificial Intelligence, 37(9):11007--11015

    2023 doi

  38. [47]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  39. [48]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.