DeltaLLM: Compress LLMs with Low-Rank Deltas between Shared Weights

Ayyoob Imani; Liana Mikaelyan; Mathew Salvaris; Mohsen Fayyaz; Parth Pathak

REVIEW 4 major objections 5 minor 12 references

DeltaLLM: Compress LLMs with Low-Rank Deltas between Shared Weights

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read DeltaLLM compresses LLMs by letting later layers share an anchor layer's weights plus small trained low-rank delta matrices; with roughly 30–40M tokens of delta-only training it matches similarly sized models trained from scratch and…

desk verdict A useful compression recipe with honest ablations; the headline claims about parity with from-scratch training and 90% retention outrun the evidence. read the letter →

arxiv 2501.18596 v2 pith:PC5GW6KB submitted 2025-01-30 cs.LG cs.AI

Liana Mikaelyan , Ayyoob Imani , Mathew Salvaris , Parth Pathak , Mohsen Fayyaz This is my paper · ORCID

classification cs.LGcs.AI

keywords LLMcompressionweightsharinglow-rankdeltaprogressivemodulereplacementknowledgedistillationpost-trainingtransformerredundancyparameterefficiency

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

DeltaLLM introduces a post-training compression scheme that restructures a transformer so that several later layers reuse the weights of an earlier anchor layer, with each reused layer's difference from the original captured by a small low-rank matrix called a delta. The paper claims that training only these delta modules with progressive module replacement and knowledge distillation on roughly 30 to 40 million tokens is sufficient to reach performance on par with similarly sized models trained from scratch, while a 12% parameter reduction retains about 90% of the base model's average accuracy on common reasoning benchmarks. The method is also reported to outperform JointDrop, SliceGPT, ShortGPT, and LaCo at comparable parameter savings, and it tolerates additional quantization. If correct, this gives a cheap way to shrink LLMs for storage-constrained devices without removing computations, and it suggests that much of the difference between adjacent transformer layers is low-rank.

What carries the argument

The load-bearing mechanism is the weight-decomposition identity of Eq. (1): every replaced layer $l+i$ is written as the anchor weight $W_l$ plus a delta $\tilde{\delta}_l^{l+i}$, a low-rank approximation of $W_{l+i} - W_l$ parameterized in the same two-matrix $A B$ form as LoRA. The delta modules are trained while the shared anchor weights remain frozen, under a progressive module replacement schedule in which original layers are gradually swapped for the delta layers; the loss combines cross-entropy with a KL-divergence distillation term from the original model. This configuration is what lets the whole compression run on tens of millions of tokens rather than billions, because the search space is small and the teacher provides dense signal.

What would settle it

After a DeltaLLM model is trained, set all delta matrices to zero and re-run the same benchmarks; if the zeroed-delta model scores within noise of the full model, the deltas are not doing the work the paper attributes to them, which would falsify the central mechanism.

Watch

Extended reading notes

Core claim

The central discovery is that a pretrained transformer layer can be re-expressed as an anchor layer's weights plus a low-rank delta, $W_{l+i} = W_l + \tilde{\delta}_l^{l+i}$, where $\tilde{\delta}$ is a low-rank approximation of the actual difference $W_{l+i} - W_l$. The paper shows that when several later blocks share an anchor block, storing the anchor once plus the deltas saves parameters, and that training only these deltas—using progressive module replacement and a KL-divergence distillation loss—on 37M tokens for DeltaPhi and 32M tokens for DeltaLlama yields models whose average accuracy on MMLU-Pro, WinoGrande, ARC-Challenge, HellaSwag, and PIQA stays within about 90% of the base model while parameters drop by 12%. A 24%-reduced DeltaPhi 2.9B matches a recovery fine-tuned SlicedPhi 3.3B that is roughly 400M parameters larger, despite DeltaPhi receiving no recovery fine-tuning. These properties hold for both the Phi and Llama families, with MLP-sharing variants giving lower perplexities than attention-sharing variants at the same block count.

Load-bearing premise

The method assumes that a frozen anchor layer plus a low-rank delta trained on only 30 to 40 million tokens can reproduce the replaced layer's function on the deployment distribution, without any updating of the shared weights.

Editorial extensions

If this is right

A 12% parameter reduction retains roughly 90% of the base model's average zero-shot accuracy across MMLU-Pro, WinoGrande, ARC-Challenge, HellaSwag, and PIQA.
Training only the delta modules (frozen anchors) with progressive module replacement converges faster than plain distillation, so the whole compression can be done with roughly 30–40M tokens instead of billions.
Compressing MLP layers yields lower perplexity than compressing attention layers at the same number of replaced blocks, so the method's recommended use is MLP weight sharing.
The method is orthogonal to quantization: 8-bit and 4-bit quantization of the anchors degrades performance only slightly, and keeping the small delta layers in FP16 gives a marginal gain.
DeltaPhi 2.9B with a 24% reduction and no recovery fine-tuning matches a recovery fine-tuned SlicedPhi 3.3B with a 12% reduction, despite being about 400M parameters smaller.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the low-rank delta structure is as expressive as reported, the difference between adjacent transformer layers should exhibit a small effective rank on natural pretrained models; this could be verified directly by SVD on layer weight differences, and a positive result would explain why the deltas recover performance so cheaply.
The same anchor-plus-delta recipe could plausibly be used to pretrain small models from scratch by progressively growing depth, turning the compression method into an architecture design principle; the paper only gestures at this as a future direction.
Because the deltas are small, they could be stored at higher precision than quantized anchors with only a small memory cost, a trade-off that the paper's quantization results suggest but do not optimize.
A natural testable extension is to vary the data budget downward (e.g., 5M, 10M, 20M tokens) and measure the performance curve; if performance plateaus well before 30M tokens, even cheaper compression is possible.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Worth a look if you care about post-training compression for on-device LLMs. The combination—cross-layer weight sharing with low-rank deltas, trained via progressive module replacement and distillation—is a genuine extension of BERT-of-Theseus-style PMR to decoder-only models, and the ablations on block/layer choice (Section 5.7) are the best part of the paper. The reported storage savings (12-25% fewer parameters, 90MB of deltas for the 2.9B Phi) are plausible, and the orthogonality to quantization is a nice practical touch.

The weak spots are the two headline claims. "On par with models trained from scratch" is supported only by comparisons to Qwen and Llama checkpoints with different data and training budgets; the authors never train a matched-budget from-scratch baseline. The retention numbers are averages over five tasks with no error bars or seeds. Look at Table 1 directly: DeltaLlama drops from 0.58 to 0.50 average, and ARC-Challenge goes from 0.61 to 0.35—well below 90% retention on a task that matters. The "similar results" for DeltaPhi 2.9B versus SlicedPhi holds on MMLU-Pro and WinoGrande but not on ARC or HellaSwag. So the abstract's precision ('retaining 90%', 'on par') outruns the evidence.

Also, the core expressiveness assumption—rank-100 deltas trained on ~37M tokens can re-specialize shared MLP weights across 6-9 blocks—is never stress-tested. No scaling curves in rank, token count, or number of compressed blocks, so I can't tell when the method breaks down. The paper itself flags some of this in Section 5, which I appreciate, but the abstract doesn't carry those caveats.

On the practical side, the comparisons to JointDrop, ShortGPT, SliceGPT and LaCo look fair: same compression levels, and the Delta models beat them on average accuracy. The abstract says the models are released, though I couldn't find links in the v2 text—worth checking the final version.

My recommendation: this deserves a serious referee, not a desk reject. It's a competent empirical study with a useful result, but it needs revision before acceptance: error bars or seeds, a matched from-scratch baseline (or reworded claims), and ideally code and model artifacts. If I were editing, I'd send it out with a request for those changes.

Referee Report

4 major / 5 minor

Summary. The paper proposes DeltaLLM, a post-training compression scheme that restructures selected Transformer blocks as a shared anchor weight plus a low-rank delta matrix (Eqs. 1-2), and trains only the delta modules via progressive module replacement and knowledge distillation from the original model. The authors compress Phi-3.5 and Llama-3.2 into DeltaPhi and DeltaLlama variants with 12-25% parameter reductions, and report zero-shot benchmark results in Table 1, comparisons to JointDrop/SliceGPT/ShortGPT/LaCo in Tables 2-3, quantization results in Table 4, and ablations in Tables 5-6. The central claims are that 30-40M tokens of delta-only training suffices to match models trained from scratch, and that the compressed models retain about 90% of base-model performance.

Significance. If the token-efficiency claim held, DeltaLLM would be a practically valuable contribution for on-device deployment: 30-40M tokens is orders of magnitude cheaper than pruning-plus-retraining pipelines such as MINITRON, which require billions of tokens. The paper combines known ingredients (cross-layer weight sharing, LoRA-style deltas, progressive module replacement) in a new configuration for decoder-only LLMs, and it includes useful ablations on block selection, layer type, and delta initialization. The release of DeltaPhi/DeltaLlama models is a further strength. However, the headline comparisons to from-scratch training and the '90% retention' figure rest on thin evidence, so the significance is conditional on additional controlled experiments.

major comments (4)

[Abstract and Section 4.2, Table 1] The claim that 30-40M tokens of delta-only training is 'sufficient to achieve performance on par with LLMs of comparable sizes trained from scratch' is not supported by the evidence presented. Table 1 compares DeltaPhi/DeltaLlama to Qwen 2.5 and Llama 3.2, which are public models trained on different data, with different token budgets, and at different parameter counts; no matched-architecture, matched-budget from-scratch baseline is trained by the authors. At minimum, the paper should either train a small from-scratch model on the same data and compute budget, or rephrase the claim as 'competitive with existing SLMs of similar size.'
[Table 1 and Section 4.2] The abstract's 'retaining 90% of the performance' is not reproducible from Table 1: DeltaLlama 2.52B drops from 0.58 to 0.51 average (88%) and DeltaLlama 2.41B drops to 0.50 (86%); on ARC-Challenge DeltaLlama 2.52B drops from 0.61 to 0.35 (57%). The averages have no standard errors, no confidence intervals, and no multiple-seed variation, and the benchmark set has only five tasks. Please report per-task variance or at least state the range explicitly rather than a single rounded percentage.
[Section 3.2, Eq. (2), and Section 4.1] The method relies on the empirical premise that a rank-r delta (r=100 or r=1000) trained on 32-37M tokens can re-specialize shared MLP or attention weights across several blocks, but the paper offers no analysis of when this premise fails. There is no scaling curve varying rank, number of compressed blocks, or number of training tokens; Table 1 only gives two operating points per model family. Such curves are needed to justify the choice of r and to bound compression limits, especially since the larger compression variants show substantial per-task drops.
[Section 4.6, Tables 2 and 3] The comparison to SOTA compression methods is fairer than the abstract suggests. Table 2 shows LaCo on Phi achieving higher MMLU-Pro (0.33 vs 0.32) and WinoGrande (0.71 vs 0.70) than DeltaPhi, with no training; the paper's text acknowledges this only partially. In Table 3 the baseline compression percentages vary from 18% to 25% while the DeltaLlama variants are 21% and 25%, so the 'same number of parameters removed' phrasing is imprecise. Please report the training data and token budgets used for all baselines and use matched parameter reductions or, failing that, state the mismatch explicitly.

minor comments (5)

[Section 4.1] The text says 'using the procedure outlined in Section 4', but the training procedure is described in Section 3.3; the cross-reference should be corrected.
[Eq. (3)] The loss term L_logits is not defined in Eq. (3); the surrounding text mentions KL divergence only later. Please define the distillation loss formally at first use.
[Tables 1 and 7] There is an inconsistency in the reported rank for DeltaLlama 2.41B: Table 1 lists r100/30MB, while Table 7 lists 'DELTALLAMA (9 seq MLP δ, r1000) 2.41B 25%'. Please reconcile the two entries.
[Figure 3] The y-axis label 'WinoGrade' is misspelled; it should be 'WinoGrande'.
[Section 4.4, Table 5] The PMR claim is based on comparing different epoch counts (PMR 2 epochs vs no-PMR 5 epochs on Alpaca), but on Ultrachat the no-PMR model achieves lower perplexity (7.24 vs 7.49). The text should qualify the 'faster convergence' claim and explain the Ultrachat discrepancy.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the delta reparameterization is a definition, and benchmark results are external to the fitted deltas.

full rationale

The load-bearing claim is that training low-rank delta matrices between shared transformer layers on roughly 37M tokens recovers much of the base model's performance. Equation (1) and equation (2) merely define a reparameterization: W_{l+i} = W_l + tilde-delta, with tilde-delta a low-rank approximation of W_{l+i} - W_l. That is a construction, not a derived prediction. The deltas are fit by minimizing the distillation loss in equation (3) on Alpaca/Ultrachat, while the reported accuracies on MMLU-Pro, WinoGrande, ARC-Challenge, HellaSwag, and PIQA come from an external evaluation harness and are not used to fit the deltas. The comparison with Qwen and Llama of similar size uses independently trained checkpoints, and the baselines JointDrop, SliceGPT, ShortGPT, and LaCo are independent methods. No load-bearing step reduces to a self-citation or to a cited uniqueness theorem. The paper's weakness is empirical thinness (four tasks, no error bars, no matched from-scratch baseline), which is a correctness-risk concern, not a circularity.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim rests primarily on empirical assumptions about layer redundancy, the sufficiency of low-rank corrections, and the adequacy of a small distillation set, plus a set of hyperparameters that are tuned but not fully reported. No new physical or architectural entities are introduced; delta matrices are standard low-rank adapters applied to layer differences. The paper's contribution is therefore an empirical compression recipe rather than a derivation from first principles.

free parameters (5)

delta rank r = 100 or 1000
Rank of the low-rank delta matrices; chosen per model and ablation, directly controls the compression versus performance tradeoff (Tables 1, 7).
distillation weight alpha = not reported
Weight between cross-entropy and KL distillation loss in Eq. (3); tuned via Bayesian optimization (Section 4.1).
PMR replacement probability schedule and extra epochs = not reported
Schedule parameters for progressive module replacement; tuned via hyperparameter search (Section 4.1).
LoRA scaling alpha and dropout = not reported
LoRA hyperparameters for delta layers; included in the tuned search space (Section 4.1).
learning rate and scheduler = not reported
Optimizer settings tuned via Bayesian optimization; final values are not disclosed in the paper.

assumptions (4)

domain assumption Consecutive Transformer layers are redundant enough that their weight differences are well approximated by low-rank matrices.
Central modeling premise behind Eqs. (1) and (2), motivated by cited redundancy studies (He et al., Men et al., Yang et al.) and supported only by the paper's experiments.
domain assumption Keeping the first and last two blocks unchanged preserves performance.
Stated in Section 4.5; based on prior work (Men et al., 2024; Ma et al., 2023), not derived or ablated within this paper.
domain assumption The teacher's logits provide a sufficient training signal for the deltas with only 30 to 40 million tokens.
Assumed by the distillation training in Eq. (3); the paper shows it works empirically but does not analyze the data requirement.
standard math Standard matrix factorization and the LoRA parameterization provide valid low-rank approximations.
Used in Section 3.2 to define delta matrices; standard linear algebra.

how reviews work

0 comments

Cite this review

Pith. "Pith review of DeltaLLM: Compress LLMs with Low-Rank Deltas between Shared Weights." pith.science (2026). https://pith.science/paper/PC5GW6KB

@misc{pith2026250118596,
  author       = {Pith},
  title        = {Pith review of: DeltaLLM: Compress LLMs with Low-Rank Deltas between Shared Weights},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PC5GW6KB}},
  note         = {Machine review of arXiv:2501.18596}
}

read the original abstract

We introduce DeltaLLM, a new post-training compression technique to reduce the memory footprint of LLMs. We propose an alternative way of structuring LLMs with weight sharing between layers in subsequent Transformer blocks, along with additional low-rank difference matrices between them. For training, we adopt the progressing module replacement method and show that the lightweight training of the low-rank modules with approximately 30M-40M tokens is sufficient to achieve performance on par with LLMs of comparable sizes trained from scratch. We release the resultant models, DeltaLLAMA and DeltaPHI, with a 12% parameter reduction, retaining 90% of the performance of the base Llama and Phi models on common knowledge and reasoning benchmarks. Our method also outperforms compression techniques JointDrop, LaCo, ShortGPT and SliceGPT with the same number of parameters removed. For example, DeltaPhi 2.9B with a 24% reduction achieves similar average zero-shot accuracies as recovery fine-tuned SlicedPhi 3.3B with a 12% reduction, despite being approximately 400M parameters smaller with no fine-tuning applied. This work provides new insights into LLM architecture design and compression methods when storage space is critical.

Figures

Figures reproduced from arXiv: 2501.18596 by the authors.

**Figure 1.** DELTALLM on the right, replaces some transformer layers with others, and account for the layer differences (delta) using low-rank matrices, which are trained to recover the original model’s performance. Hoffmann et al., 2022), many model families are also trained in smaller sizes due to deployment constraints. In addition, multiple approaches—such as distillation (Hinton, 2015; Gu et al., 2024), prompt and KV-cache … view at source ↗

**Figure 2.** Two ways to structure for a Delta-Model: delta-layers at each subsequent block after a base block (left) and alternating blocks with delta modules between (right). called deltas. A model weight W of layer l + i can be restructured as a function of the previous layer l and a delta between the two weights as follows: WM×N l+i = WM×N l + ˜δ l l+i (1) where ˜δ l l+i is the low-rank approximation of δ l l+i = WM×N l+i − … view at source ↗

**Figure 3.** DeltaLlama and DeltaPhi accuracies on WinoGrade and MMLU pro 4. Experiments 4.1. Experiment Settings We use Phi 3.5 (Abdin et al., 2024) and Llama 3.2 (Dubey et al., 2024) as the teacher models to obtain DELTAPHI and DELTALLAMA models respectively using the procedure outlined in Section 4. We use Alpaca (Rohan Taori & Hashimoto, 2013) and Ultrachat (Ding et al., 2023) for all training experiments with DELTAPHI and … view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 3 canonical work pages

[3]

Frankle, J., Dziugaite, G

URL https://openreview.net/forum? id=rJl-b3RcF7. Frankle, J., Dziugaite, G. K., Roy, D. M., and Carbin, M. Linear mode connectivity and the lottery ticket hy- pothesis. In Proceedings of the 37th International Con- ference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Ma- chine Learning Research, pp. 3259–326...

arXiv 2020
[9]

Oymak, S., Fabian, Z., Li, M., and Soltanolkotabi, M

URL https://openreview.net/forum? id=9U0nLnNMJ7. Oymak, S., Fabian, Z., Li, M., and Soltanolkotabi, M. Gen- eralization guarantees for neural networks via harnessing the low-rank structure of the jacobian. arXiv preprint arXiv:1906.05392, 2019. Paischer, F., Hauzenberger, L., Schmied, T., Alkin, B., Deisenroth, M. P., and Hochreiter, S. One initializa- ti...

arXiv 1906
[12]

findings-emnlp.372

URL https://aclanthology.org/2024. findings-emnlp.372. YEH, S.-Y ., Hsieh, Y .-G., Gao, Z., Yang, B. B. W., Oh, G., and Gong, Y . Navigating text-to-image customization: From lyCORIS fine-tuning to model evaluation. In The Twelfth International Conference on Learning Represen- tations, 2024. URL https://openreview.net/ forum?id=wfzXa8e783. Yu, H., Edunov,...

2024
[57]

findings-acl.57/

URL https://aclanthology.org/2024. findings-acl.57/. Rohan Taori, Ishaan Gulrajani, T. Z. Y . D. X. L. C. G. P. L. and Hashimoto, T. B. Stanford alpaca: An instruction- following llama model, 2013. URLhttps://github. com/tatsu-lab/stanford_alpaca. Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y . Winogrande: An adversarial winograd schema challen...

work page 2024
[2014]

cc/paper_files/paper/2014/file/ 2afe4567e1bf64d32a5527244d104cea-Paper

URL https://proceedings.neurips. cc/paper_files/paper/2014/file/ 2afe4567e1bf64d32a5527244d104cea-Paper. pdf. Dettmers, T., Lewis, M., Belkada, Y ., and Zettlemoyer, L. Gpt3. int8 (): 8-bit matrix multiplication for transformers 9 DELTALLM: Compress LLMs with Low-Rank Deltas between Shared Weights at scale. Advances in Neural Information Processing System...

arXiv 2014
[2017]

Shoaib Ahmed Siddiqui, Xin Dong, G

URL https://openreview.net/forum? id=B1ckMDqlg. Shoaib Ahmed Siddiqui, Xin Dong, G. H. T. B. J. K. D. K. P. M. A deeper look at depth pruning of llms. arXiv preprint arXiv:2407.16286, 2024. Singh, S. P. and Alistarh, D. Woodfisher: Efficient second-order approximation for neural network com- pression. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M...

arXiv 2024
[2018]

Mishra, A

URL https://openreview.net/forum? id=S1lN69AT-. Mishra, A. K., Latorre, J. A., Pool, J., Stosic, D., Stosic, D., Venkatesh, G., Yu, C., and Micikevicius, P. Accelerating sparse deep neural networks. CoRR, abs/2104.08378, 2021. URL https://arxiv.org/ abs/2104.08378. Muralidharan, S., Sreenivas, S. T., Joshi, R. B., Chochowski, M., Patwary, M., Shoeybi, M.,...

arXiv 2021
[2019]

Denton, E

URL https://openreview.net/forum? id=HyzdRiR9Y7. Denton, E. L., Zaremba, W., Bruna, J., LeCun, Y ., and Fergus, R. Exploiting linear structure within convolu- tional networks for efficient evaluation. In Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N., and Wein- berger, K. (eds.), Advances in Neural Information Processing Systems, volume 27. Curran ...

work page

Show all 12 references

[2020]

Leviathan, Y ., Kalman, M., and Matias, Y

URL https://openreview.net/forum? id=H1eA7AEtvS. Leviathan, Y ., Kalman, M., and Matias, Y . Fast infer- ence from transformers via speculative decoding. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), International Conference on Mac...

2023
[2022]

URL https: //doi.org/10.48550/arXiv.2210.17323

doi: 10.48550/ARXIV .2210.17323. URL https: //doi.org/10.48550/arXiv.2210.17323. Gale, T., Elsen, E., and Hooker, S. The state of sparsity in deep neural networks. CoRR, abs/1902.09574, 2019. URL http://arxiv.org/abs/1902.09574. Gao, L., Tow, J., Abbasi, B., Biderman, S., Blac...

1902 doi arXiv
[2023]

Li, Y ., Liang, Y ., and Risteski, A

URL https://proceedings.mlr.press/ v202/leviathan23a.html. Li, Y ., Liang, Y ., and Risteski, A. Recovery guarantee of weighted low-rank approximation via alternating mini- mization. In International Conference on Machine Learn- ing, pp. 2358–2367. PMLR, 2016. Li, Y ., Ma, T.,...

2016 arXiv
[2024]

Han, S., Mao, H., and Dally, W

URL https://openreview.net/forum? id=5h0qf7IBZZ. Han, S., Mao, H., and Dally, W. J. Deep compres- sion: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149, 2015a. Han, S., Pool, J., Tran, J., and Dally, W. J....

arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.

[1] [3]

Frankle, J., Dziugaite, G

URL https://openreview.net/forum? id=rJl-b3RcF7. Frankle, J., Dziugaite, G. K., Roy, D. M., and Carbin, M. Linear mode connectivity and the lottery ticket hy- pothesis. In Proceedings of the 37th International Con- ference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Ma- chine Learning Research, pp. 3259–326...

arXiv 2020

[2] [9]

Oymak, S., Fabian, Z., Li, M., and Soltanolkotabi, M

URL https://openreview.net/forum? id=9U0nLnNMJ7. Oymak, S., Fabian, Z., Li, M., and Soltanolkotabi, M. Gen- eralization guarantees for neural networks via harnessing the low-rank structure of the jacobian. arXiv preprint arXiv:1906.05392, 2019. Paischer, F., Hauzenberger, L., Schmied, T., Alkin, B., Deisenroth, M. P., and Hochreiter, S. One initializa- ti...

arXiv 1906

[3] [12]

findings-emnlp.372

URL https://aclanthology.org/2024. findings-emnlp.372. YEH, S.-Y ., Hsieh, Y .-G., Gao, Z., Yang, B. B. W., Oh, G., and Gong, Y . Navigating text-to-image customization: From lyCORIS fine-tuning to model evaluation. In The Twelfth International Conference on Learning Represen- tations, 2024. URL https://openreview.net/ forum?id=wfzXa8e783. Yu, H., Edunov,...

2024

[4] [57]

findings-acl.57/

URL https://aclanthology.org/2024. findings-acl.57/. Rohan Taori, Ishaan Gulrajani, T. Z. Y . D. X. L. C. G. P. L. and Hashimoto, T. B. Stanford alpaca: An instruction- following llama model, 2013. URLhttps://github. com/tatsu-lab/stanford_alpaca. Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y . Winogrande: An adversarial winograd schema challen...

work page 2024

[5] [2014]

cc/paper_files/paper/2014/file/ 2afe4567e1bf64d32a5527244d104cea-Paper

URL https://proceedings.neurips. cc/paper_files/paper/2014/file/ 2afe4567e1bf64d32a5527244d104cea-Paper. pdf. Dettmers, T., Lewis, M., Belkada, Y ., and Zettlemoyer, L. Gpt3. int8 (): 8-bit matrix multiplication for transformers 9 DELTALLM: Compress LLMs with Low-Rank Deltas between Shared Weights at scale. Advances in Neural Information Processing System...

arXiv 2014

[6] [2017]

Shoaib Ahmed Siddiqui, Xin Dong, G

URL https://openreview.net/forum? id=B1ckMDqlg. Shoaib Ahmed Siddiqui, Xin Dong, G. H. T. B. J. K. D. K. P. M. A deeper look at depth pruning of llms. arXiv preprint arXiv:2407.16286, 2024. Singh, S. P. and Alistarh, D. Woodfisher: Efficient second-order approximation for neural network com- pression. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M...

arXiv 2024

[7] [2018]

Mishra, A

URL https://openreview.net/forum? id=S1lN69AT-. Mishra, A. K., Latorre, J. A., Pool, J., Stosic, D., Stosic, D., Venkatesh, G., Yu, C., and Micikevicius, P. Accelerating sparse deep neural networks. CoRR, abs/2104.08378, 2021. URL https://arxiv.org/ abs/2104.08378. Muralidharan, S., Sreenivas, S. T., Joshi, R. B., Chochowski, M., Patwary, M., Shoeybi, M.,...

arXiv 2021

[8] [2019]

Denton, E

URL https://openreview.net/forum? id=HyzdRiR9Y7. Denton, E. L., Zaremba, W., Bruna, J., LeCun, Y ., and Fergus, R. Exploiting linear structure within convolu- tional networks for efficient evaluation. In Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N., and Wein- berger, K. (eds.), Advances in Neural Information Processing Systems, volume 27. Curran ...

work page