Pith. sign in

REVIEW 4 major objections 5 minor 66 references

$\mu$-MoE: Test-Time Pruning as Micro-Grained Mixture-of-Experts

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Pruning a language model's weights with each prompt's own activations matches or beats static offline pruning, while cutting inference compute roughly in proportion to the active weight fraction.

desk verdict Running Wanda's score on the test prompt's own activations is a simple, plausible trick with real reported gains, but the evidence needs matched baselines and a stability check before I'd trust it. read the letter →

arxiv 2505.18451 v1 pith:JTECTJH2 submitted 2025-05-24 cs.LG cs.AIcs.CL

Toshiaki Koike-Akino , Jing Liu , Ye Wang This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords test-timepruningactivation-awareWandamixtureofexpertsdynamicsparsityLLMcompressiondomainshiftprompt-dependent
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that a language model can prune its own weights at the moment of inference, using the current prompt itself as the calibration data. It applies Wanda's activation-aware score, the product of weight magnitude and the prompt-activation norm, to select a sparse mask per prompt, and calls the resulting collection of active weights a mixture of micro-experts (μ-MoE). The claim is that this online dynamic pruning matches or beats static offline pruning on perplexity and reasoning benchmarks, while cutting inference FLOPs by roughly the active-weight fraction ρ. If the claim is correct, test-time adaptation can replace calibration-data-dependent compression, removing the domain-shift problem of static pruning.

What carries the argument

The central object is the Wanda score, S'_{i,j}=|W_{i,j}|\cdot\|X_{j,:}\|_2, an activation-aware pruning metric whose product of weight magnitude and input-activation ℓ2-norm requires only a diagonal approximation to the Hessian used by SparseGPT. At test time, the paper replaces the offline calibration tokens with the current prompt's tokens X, applies the score per layer, and keeps the top-ρ fraction of weights per row via a kth-value threshold. The machinery that makes this feasible is Wanda's quadratic complexity O[3dd'+dT+ρdd'T], which gives a complexity ratio relative to full inference of ρ+3/T+1/d' ≈ ρ for long sequences, plus the use of torch.kthvalue to avoid full sorting. This is what turns per-prompt pruning from a costly operation into a near-free one.

What would settle it

Measure the layer-by-layer feedback effect: compute Wanda masks online using the actual pruned activations versus using unpruned reference activations passed only for calibration. If the online-pruned-choice perplexity diverges from the reference as depth grows, the claim that per-prompt Wanda scores are safe under sequential pruning is falsified. A simpler check is to find any prompt where the per-row top-ρ mask according to |W|·‖X‖_2 differs from the oracle-optimal mask so much that online pruning collapses the output.

Watch

Extended reading notes

Core claim

The paper's central claim is that applying Wanda's score S'_{i,j}=|W_{i,j}|\cdot\|X_{j,:}\|_2, computed on the activations of the current prompt rather than on fixed offline calibration tokens, selects a per-prompt sparse mask that preserves accuracy as well as or better than static pruning. Over OPT models from 125M to 13B parameters, μ-MoE obtains lower perplexity than offline Wanda and magnitude pruning across WikiText-2, PTB, and C4; on LLaVA-7B it improves accuracy over Wanda and SparseGPT on ScienceQA and TextVQA, with the largest gains at the lowest active-weight fraction tested (40%). The authors interpret this as evidence that each weight can serve as a micro-expert activated per prompt, realizing a task-agnostic mixture of experts without retraining.

Load-bearing premise

The assumption that the prompt's own activation magnitudes reliably mark which weights can be zeroed for that prompt, with no retraining or error correction, is the load-bearing premise; the paper does not analyze how pruning errors compound through layers when later masks are computed from already-pruned activations.

Editorial extensions

If this is right

  • Inference cost can scale with the active-weight fraction ρ: at 40% active weights the reported OPT-17B MACs drop from 1.64T to 671B, near-linear in ρ.
  • Because each prompt supplies its own calibration tokens, accuracy no longer depends on how well a fixed calibration set matches the deployment domain.
  • The method requires no weight updates or retraining; it only zeroes weights temporarily, so a single pretrained model can serve many tasks with different sparse masks.
  • The reported gains grow as ρ shrinks, suggesting μ-MoE is most useful at aggressive compression ratios (40% active weights and below).
  • The same per-prompt score could be applied to any transformer with linear layers, including vision-language models, as demonstrated on LLaVA-7B.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the Wanda score is cheap enough to compute, one could cache or predict masks for recurring prompt types (for example, few-shot templates) and amortize the calibration cost across many queries.
  • The sequential-pruning feedback loop, where later masks are computed from already-pruned activations, is an implicit dynamical system; analyzing it might reveal whether pruning order or per-layer ρ schedules can further improve the accuracy-cost trade-off.
  • A natural extension is to fine-tune with the μ-MoE mask in the loop: if the masks are prompt-dependent, low-rank or LoRA adjustments per active weight could replace simple zeroing.
  • The same per-prompt calibration idea could be applied to quantization scales or rank reduction, not just binary masks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes µ-MoE, a test-time pruning method that applies Wanda's activation-aware weight pruning to each prompt individually, using the prompt's own activations as online calibration. The authors frame the selection of active weights as a mixture of single-parameter micro-experts, and derive a complexity ratio showing that the overhead of online scoring is negligible for sufficiently long sequences. Experiments on OPT models (perplexity on WikiText-2, PTB, C4) and LLaVA-7B (accuracy on ScienceQA and TextVQA) report consistent improvements over static Wanda and SparseGPT baselines, especially at low active-weight ratios.

Significance. If the reported results hold, the paper provides a simple, training-free mechanism for input-adaptive compression that could mitigate calibration domain shift at inference. The use of Wanda's score is well-grounded, the complexity derivation is mostly correct, and the experiments cover a useful range of model sizes and modalities. A notable strength is the empirical demonstration that per-prompt masks can improve over even matched-calibration static pruning in the OPT perplexity tables. However, the central mechanism—sequential pruning with masks computed from already-pruned activations—is not analyzed, and the LLaVA comparisons lack a same-domain static baseline, so the significance is currently conditional on additional validation.

major comments (4)
  1. [Section 2, 'Instant Wanda Pruning as µ-MoE'] The method computes each layer's mask from activations that have themselves been produced by previously pruned layers. The citation of Williams and Aletras (2023) supports robustness to a single calibration sample for offline static pruning, where calibration activations come from the unpruned model; it does not cover the online sequential setting. The paper provides no layerwise analysis, no comparison of masks chosen with pruned versus unpruned activations, and no error bars, so the observed aggregate gains could be fragile. Please add an ablation that isolates the feedback loop (e.g., compare online sequential pruning against Wanda masks computed from unpruned activations at each layer, and report layerwise score correlation or perplexity as a function of depth).
  2. [Section 3, Tables 2 and 3] The LLaVA experiments compare µ-MoE against SparseGPT and Wanda calibrated on a different dataset (TextVQA for ScienceQA, ScienceQA for TextVQA). This confounds the effect of test-time adaptation with the effect of calibration mismatch. The claim that µ-MoE 'tackles domain shift' requires a same-domain static baseline, e.g., Wanda calibrated on the training split of the test benchmark. Without such a baseline, part of the reported gains may simply reflect the cost of mismatched calibration rather than the benefit of online adaptation.
  3. [Section 3, Tables 1–3] No error bars or statistical significance tests are reported. Some improvements are small (e.g., Table 1, OPT-125M at 60% active weights: µ-MoE average 34.1 vs. Wanda (WT2 Calib) 35.1, but individual cells are closer), while the 40% active-weight differences in Table 2 are large. The absence of variance estimates makes it difficult to assess whether the consistent but modest gains in Table 1 are meaningful. Please report standard errors over prompts or repeated calibration runs.
  4. [Section 2, complexity analysis, and Table 4] The derived complexity ratio ρ + 3/T + 1/d' implies that for T=128 and d'≈4096 the overhead is about 2.4% of full inference. However, Table 4 shows that at 80% active weights the FLOPs are 3.21T, whereas 80% of the full 3.29T would be 2.63T, leaving 0.58T (≈18%) of overhead. This discrepancy is not explained. Please clarify whether scores are computed once per prompt or per generated token, and specify exactly which operations calflops counts; as written, Table 4 appears inconsistent with the paper's own complexity formula.
minor comments (5)
  1. [Section 3 heading] The heading 'Expriments' should be spelled 'Experiments'.
  2. [Table 2 caption] The caption contains a typo: 'SpargeGPT' should be 'SparseGPT'.
  3. [Section 2, Remark 2.1] The remark states that torch.kthvalue has linear theoretical complexity, but the subsequent discussion in Section B notes that torch.topk has O[d′d log(kc)] and torch.kthvalue O[d′d] on average; consider making the distinction between worst-case and average-case explicit in the main text.
  4. [Related Work, Section A] Dynamic pruning and contextual sparsity methods such as DejaVu (Liu et al., 2023c) are cited but not discussed in the context of the proposed approach; a brief comparison of how test-time weight pruning differs from activation sparsity would help position the contribution.
  5. [General] The first page includes the note 'Preliminary work.'; if this is intended for a journal submission, that line should be removed or replaced with the appropriate submission status.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the method applies the external Wanda score at test time and validates against benchmarks; the µ-MoE label is post-hoc framing rather than a fitted prediction.

full rationale

The paper's central mechanism is Eq. (3), the Wanda activation-aware score from Sun et al. (2023), applied to test-time tokens instead of offline calibration tokens. This is an empirical proposal, not a derivation whose output equals its input: the compression target rho is externally chosen, and the reported improvements are measured perplexity and accuracy values on OPT and LLaVA benchmarks compared with static pruning baselines. The single-sample robustness citation (Williams & Aletras, 2023) is external and merely motivational. The "micro-MoE" vocabulary is a post-hoc interpretation of per-weight dynamic pruning; no load-bearing conclusion depends on the naming, and the empirical comparisons retain independent content. The only self-citations appear in the Appendix related-work enumeration of PEFT methods and are not load-bearing. The feedback-loop concern about later-layer masks being computed from already-pruned activations is a genuine unvalidated assumption and a correctness risk, but it is not a case of the conclusion being equivalent to the input. No circular step meets the required quote-and-reduction bar.

Assumptions & free parameters 2 free parameters · 4 assumptions · 1 invented entities

No parameters are fitted to data; the target sparsity is set by the experiment. The method rests on Wanda's heuristic score, the sufficiency of per-prompt calibration, and the assumption that layer-wise online masks remain useful when upstream activations are themselves pruned. The only invented entity is the 'micro-expert' label.

free parameters (2)
  • active weight ratio rho = 0.60, 0.50, 0.40 (and 0.20 to 0.80 in figures)
    Target sparsity level selected for each experiment; it controls the accuracy-complexity trade-off. It is an experimental condition, not a fitted parameter, but the reported gains are specific to these ratios.
  • token length T = 128
    Used in the Table 4 complexity analysis; the overhead ratio 3/T is negligible only for large T, and the paper's complexity argument depends on this choice.
assumptions (4)
  • domain assumption Wanda's activation-aware score ranks weight importance accurately enough for pruning.
    Adopted from Sun et al. (2023) and Williams and Aletras (2023); the paper does not re-derive or validate the score for the online setting.
  • domain assumption Per-prompt activations are a sufficient calibration signal.
    Section 2 justifies online use by citing Wanda's robustness to a single calibration sample, but no direct evidence for per-prompt masks is given.
  • ad hoc to paper Sequential layer-wise pruning with masks computed from pruned upstream activations remains near-optimal.
    Unstated in Section 2; the paper applies online Wanda at every layer without analyzing the feedback from earlier pruning decisions.
  • standard math Quickselect-based kthvalue has average-case linear complexity, making pruning overhead O[dd'] and the ratio approximately rho for large T.
    Used in Section 2 and Appendix B to argue negligible overhead; this is a theoretical complexity claim and is hardware-dependent in practice.
invented entities (1)
  • micro-expert (single-parameter weight multiplier)
    purpose: Labels each individual weight as an expert that can be activated or deactivated per prompt; supplies the mixture-of-experts framing.
    No measurable prediction separates a micro-expert from a Wanda-selected weight; it is a conceptual relabeling, not a new mechanism.

how reviews work

0 comments
Cite this review

Pith. "Pith review of $\mu$-MoE: Test-Time Pruning as Micro-Grained Mixture-of-Experts." pith.science (2026). https://pith.science/paper/JTECTJH2

@misc{pith2026250518451,
  author       = {Pith},
  title        = {Pith review of: $\mu$-MoE: Test-Time Pruning as Micro-Grained Mixture-of-Experts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JTECTJH2}},
  note         = {Machine review of arXiv:2505.18451}
}
abstract

To tackle the huge computational demand of large foundation models, activation-aware compression techniques without retraining have been introduced. However, since these rely on calibration data, domain shift may arise for unknown downstream tasks. With a computationally efficient calibration, activation-aware pruning can be executed for every prompt adaptively, yet achieving reduced complexity at inference. We formulate it as a mixture of micro-experts, called $\mu$-MoE. Several experiments demonstrate that $\mu$-MoE can dynamically adapt to task/prompt-dependent structured sparsity on the fly.

Figures

Figures reproduced from arXiv: 2505.18451 by the authors.

Figure 1
Figure 1. Coarse to micro-grained MoE. Pretrained LLM (a) Offline Static Pruning Task-Specific Sparse LLM Task A Dataset Task B Dataset Calibration Data Static Compression Good Accuracy Poor Accuracy (b) Online Dynamic Pruning Pretrained LLM Task-Agnostic Sparse LLM Task A Dataset Task B Dataset Test-Time Compression Good Accuracy Good Accuracy [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Offline vs online pruning: dynamic pruning finds prompt￾dependent sparse structure at test time, preventing domain shift. to minimize the approximation loss: L = EX [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Wanda pruning complexity based on torch.sort/topk/kthvalue on CPU and GPU at ρ = 0.25, 0.50, 0.75. C. LLM Models The Open Pre-trained Transformers (OPT) (Zhang et al., 2022) is a suite of decoder-only pre-trained transformers ranging from 125M to 175B parameters. It was claimed that OPT-175B is comparable to GPT-3, while requiring only 1/7th the carbon footprint to develop [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Perplexity results averaged over WT2, PTB, and C4 datasets for compressed OPT models. E. Datasets Wikitext-2 (WT2) The WikiText language modeling dataset (Merity et al., 2016) is a collection of over 100 million tokens extracted from the set of verified good and featur…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

66 extracted references · 25 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. GPT -4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [3]

    and Frey, B

    Ba, J. and Frey, B. Adaptive dropout for training deep neural networks. Advances in neural information processing systems, 26, 2013

    work page 2013

  4. [4]

    Beyond efficiency: A systematic survey of resource-efficient large language models

    Bai, G., Chai, Z., Ling, C., Wang, S., Lu, J., Zhang, N., Shi, T., Yu, Z., Zhu, M., Zhang, Y., et al. Beyond efficiency: A systematic survey of resource-efficient large language models. arXiv preprint arXiv:2401.00625, 2024 a

    arXiv 2024

  5. [5]

    SparseLLM : Towards global pruning for pre-trained language models

    Bai, G., Li, Y., Ling, C., Kim, K., and Zhao, L. SparseLLM : Towards global pruning for pre-trained language models. arXiv preprint arXiv:2402.17946, 2024 b

    arXiv 2024

  6. [6]

    Rethinking the role of scale for in-context learning: An interpretability-based case study at 66 billion scale

    Bansal, H., Gopalakrishnan, K., Dingliwal, S., Bodapati, S., Kirchhoff, K., and Roth, D. Rethinking the role of scale for in-context learning: An interpretability-based case study at 66 billion scale. arXiv preprint arXiv:2212.09095, 2022

    arXiv 2022

  7. [7]

    LoTR : Low tensor rank weight adaptation

    Bershatsky, D., Cherniuk, D., Daulbaev, T., Mikhalev, A., and Oseledets, I. LoTR : Low tensor rank weight adaptation. arXiv preprint arXiv:2402.01376, 2024

    arXiv 2024

  8. [8]

    J., Frankle, J., and Guttag, J

    Blalock, D., Gonzalez Ortiz, J. J., Frankle, J., and Guttag, J. What is the state of neural network pruning? Proceedings of machine learning and systems, 2: 0 129--146, 2020

    2020

Show all 66 references
  1. [9]

    T., Li, Y., Lundberg, S., et al

    Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. Sparks of artificial general intelligence: Early experiments with GPT -4. arXiv preprint arXiv:2303.12712, 2023

    2023 arXiv

  2. [10]

    Buehler, E. L. and Buehler, M. J. X-LoRA : Mixture of low-rank adapter experts, a flexible framework for large language models with applications in protein mechanics and molecular design. APL Machine Learning, 2 0 (2), 2024

    2024

  3. [11]

    Self-adaptive network pruning

    Chen, J., Zhu, Z., Li, C., and Zhao, Y. Self-adaptive network pruning. In Neural Information Processing: 26th International Conference, ICONIP 2019, Sydney, NSW, Australia, December 12--15, 2019, Proceedings, Part I 26, pp.\ 175--186. Springer, 2019

    2019

  4. [12]

    SuperLoRA : Parameter-efficient unified adaptation for large vision models

    Chen, X., Liu, J., Wang, Y., Wang, P., Brand, M., Wang, G., and Koike-Akino, T. SuperLoRA : Parameter-efficient unified adaptation for large vision models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 8050--8055, 2024 a

    2024

  5. [13]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling

    Chen, Z., Wang, W., Cao, Y., Liu, Y., Gao, Z., Cui, E., Zhu, J., Ye, S., Tian, H., Liu, Z., et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024 b

    2024 arXiv

  6. [14]

    DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., Zhang, X., Yu, X., Wu, Y., Wu, Z. F., Gou, Z., Shao, Z., Li, Z., Gao, Z., Liu, A., Xue, B., Wang, B., Wu, B., Feng, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., D...

    2025 arXiv

  7. [15]

    Learning to prune deep neural networks via layer-wise optimal brain surgeon

    Dong, X., Chen, S., and Pan, S. Learning to prune deep neural networks via layer-wise optimal brain surgeon. Advances in neural information processing systems, 30, 2017

    2017

  8. [16]

    P., Clark, J

    Edalati, A., Tahaei, M., Kobyzev, I., Nia, V. P., Clark, J. J., and Rezagholizadeh, M. KronA : Parameter efficient tuning with kronecker adapter. arXiv preprint arXiv:2212.10650, 2022

    2022 arXiv

  9. [17]

    and Carbin, M

    Frankle, J. and Carbin, M. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635, 2018

    2018 arXiv

  10. [18]

    and Alistarh, D

    Frantar, E. and Alistarh, D. SparseGPT : Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pp.\ 10323--10337. PMLR, 2023

    2023

  11. [19]

    GPTQ : Accurate post-training quantization for generative pre-trained transformers

    Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. GPTQ : Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323, 2022

    2022 arXiv

  12. [20]

    Dynamic channel pruning: Feature boosting and suppression

    Gao, X., Zhao, Y., Dudziak, ., Mullins, R., and Xu, C.-z. Dynamic channel pruning: Feature boosting and suppression. arXiv preprint arXiv:1810.05331, 2018

    2018 arXiv

  13. [21]

    Han, S., Mao, H., and Dally, W. J. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149, 2015

    2015 arXiv

  14. [22]

    Optimal brain surgeon: Extensions and performance comparisons

    Hassibi, B., Stork, D., and Wolff, G. Optimal brain surgeon: Extensions and performance comparisons. Advances in neural information processing systems, 6, 1993

    1993

  15. [23]

    Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes

    Hsieh, C.-Y., Li, C.-L., Yeh, C.-K., Nakhost, H., Fujii, Y., Ratner, A., Krishna, R., Lee, C.-Y., and Pfister, T. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. arXiv preprint arXiv:2305.02301, 2023

    2023 arXiv

  16. [24]

    J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al

    Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al. LoRA : Low-rank adaptation of large language models. ICLR, 1 0 (2): 0 3, 2022

    2022

  17. [25]

    M., Zhang, Z., and Suh, G

    Hua, W., Zhou, Y., De Sa, C. M., Zhang, Z., and Suh, G. E. Channel gating neural networks. Advances in neural information processing systems, 32, 2019

    2019

  18. [26]

    PC-LoRA : Low-rank adaptation for progressive model compression with knowledge distillation

    Hwang, I., Park, H., Lee, Y., Yang, J., and Maeng, S. PC-LoRA : Low-rank adaptation for progressive model compression with knowledge distillation. arXiv preprint arXiv:2406.09117, 2024

    2024 arXiv

  19. [27]

    Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. d. l., Hanna, E. B., Bressand, F., et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024

    2024 arXiv

  20. [28]

    M., Bommarito, M

    Katz, D. M., Bommarito, M. J., Gao, S., and Arredondo, P. GPT -4 passes the bar exam. Philosophical Transactions of the Royal Society A, 382 0 (2270): 0 20230254, 2024

    2024

  21. [29]

    Z., Candogan, L

    Koike-Akino, T., Tonin, F., Wu, Y., Wu, F. Z., Candogan, L. N., and Cevher, V. Quantum-PEFT : Ultra parameter-efficient fine-tuning. arXiv preprint arXiv:2503.05431, 2025

    2025 arXiv

  22. [30]

    Scaling laws for fine-grained mixture of experts

    Krajewski, J., Ludziejewski, J., Adamczewski, K., Pi \'o ro, M., Krutul, M., Antoniak, S., Ciebiera, K., Kr \'o l, K., Odrzyg \'o \'z d \'z , T., Sankowski, P., et al. Scaling laws for fine-grained mixture of experts. arXiv preprint arXiv:2402.07871, 2024

    2024 arXiv

  23. [31]

    Optimal brain damage

    LeCun, Y., Denker, J., and Solla, S. Optimal brain damage. Advances in neural information processing systems, 2, 1989

    1989

  24. [32]

    MoE-LlaVa : Mixture of experts for large vision-language models

    Lin, B., Tang, Z., Ye, Y., Cui, J., Zhu, B., Jin, P., Zhang, J., Ning, M., and Yuan, L. MoE-LlaVa : Mixture of experts for large vision-language models. arXiv preprint arXiv:2401.15947, 2024 a

    2024 arXiv

  25. [33]

    Runtime neural pruning

    Lin, J., Rao, Y., Lu, J., and Zhou, J. Runtime neural pruning. Advances in neural information processing systems, 30, 2017

    2017

  26. [34]

    AWQ : Activation-aware weight quantization for on-device LLM compression and acceleration

    Lin, J., Tang, J., Tang, H., Yang, S., Chen, W.-M., Wang, W.-C., Xiao, G., Dang, X., Gan, C., and Han, S. AWQ : Activation-aware weight quantization for on-device LLM compression and acceleration. Proceedings of Machine Learning and Systems, 6: 0 87--100, 2024 b

    2024

  27. [35]

    DeepSeek -v3 technical report

    Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. DeepSeek -v3 technical report. arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  28. [36]

    Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning, 2023 a

    2023

  29. [37]

    LoDA : Low-dimensional adaptation of large language models

    Liu, J., Koike-Akino, T., Wang, P., Brand, M., Wang, Y., and Parsons, K. LoDA : Low-dimensional adaptation of large language models. In NeurIPS’23 Workshop on on Efficient Natural Language and Speech Processing, 2023 b

    2023

  30. [38]

    and Deng, J

    Liu, L. and Deng, J. Dynamic deep neural networks: Optimizing accuracy-efficiency trade-offs by selective execution. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018

    2018

  31. [39]

    Deja vu: Contextual sparsity for efficient LLMs at inference time

    Liu, Z., Wang, J., Dao, T., Zhou, T., Yuan, B., Song, Z., Shrivastava, A., Zhang, C., Tian, Y., Re, C., et al. Deja vu: Contextual sparsity for efficient LLMs at inference time. In International Conference on Machine Learning, pp.\ 22137--22176. PMLR, 2023 c

    2023

  32. [40]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems (NeurIPS), 2022

    2022

  33. [41]

    LLM-Pruner : On the structural pruning of large language models

    Ma, X., Fang, G., and Wang, X. LLM-Pruner : On the structural pruning of large language models. Advances in neural information processing systems, 36: 0 21702--21720, 2023

    2023

  34. [42]

    A., MacIntyre, R., Bies, A., Ferguson, M., Katz, K., and Schasberger, B

    Marcus, M., Kim, G., Marcinkiewicz, M. A., MacIntyre, R., Bies, A., Ferguson, M., Katz, K., and Schasberger, B. The penn treebank: Annotating predicate argument structure. In Human Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994, 1994

    1994

  35. [43]

    Pointer sentinel mixture models

    Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016

    2016 arXiv

  36. [44]

    L., Fei-Fei, L., Hajishirzi, H., Zettlemoyer, L., Liang, P., Cand \`e s, E., and Hashimoto, T

    Muennighoff, N., Yang, Z., Shi, W., Li, X. L., Fei-Fei, L., Hajishirzi, H., Zettlemoyer, L., Liang, P., Cand \`e s, E., and Hashimoto, T. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393, 2025

    2025 arXiv

  37. [45]

    Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 0 (140): 0 1--67, 2020

    2020

  38. [46]

    Compressing large language models using low rank and low precision decomposition

    Saha, R., Sagan, N., Srivastava, V., Goldsmith, A., and Pilanci, M. Compressing large language models using low rank and low precision decomposition. Advances in Neural Information Processing Systems, 37: 0 88981--89018, 2024

    2024

  39. [47]

    Eigen attention: Attention in low-rank space for KV cache compression

    Saxena, U., Saha, G., Choudhary, S., and Roy, K. Eigen attention: Attention in low-rank space for KV cache compression. arXiv preprint arXiv:2408.05646, 2024

    2024 arXiv

  40. [48]

    A., and Etzioni, O

    Schwartz, R., Dodge, J., Smith, N. A., and Etzioni, O. Green AI . Communications of the ACM, 63 0 (12): 0 54--63, 2020

    2020

  41. [49]

    Towards VQA models that can read

    Singh, A., Natarjan, V., Shah, M., Jiang, Y., Chen, X., Parikh, D., and Rohrbach, M. Towards VQA models that can read. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.\ 8317--8326, 2019

    2019

  42. [50]

    Sun, M., Liu, Z., Bair, A., and Kolter, J. Z. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695, 2023

    2023 arXiv

  43. [51]

    Llama 2: Open foundation and fine-tuned chat models

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  44. [52]

    Neurons in large language models: Dead, n-gram, positional

    Voita, E., Ferrando, J., and Nalmpantis, C. Neurons in large language models: Dead, n-gram, positional. arXiv preprint arXiv:2309.04827, 2023

    2023 arXiv

  45. [53]

    Q-VLM : Post-training quantization for large vision-language models

    Wang, C., Wang, Z., Xu, X., Tang, Y., Zhou, J., and Lu, J. Q-VLM : Post-training quantization for large vision-language models. arXiv preprint arXiv:2410.08119, 2024

    2024 arXiv

  46. [54]

    H., and Gao, J

    Wang, Y., Agarwal, S., Mukherjee, S., Liu, X., Gao, J., Awadallah, A. H., and Gao, J. AdaMix : Mixture-of-adaptations for parameter-efficient model tuning. arXiv preprint arXiv:2205.12410, 2022

    2022 arXiv

  47. [55]

    Emergent abilities of large language models

    Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682, 2022

    2022 arXiv

  48. [56]

    and Aletras, N

    Williams, M. and Aletras, N. On the impact of calibration data in post-training quantization and pruning. arXiv preprint arXiv:2311.09755, 2023

    2023 arXiv

  49. [57]

    Mixture of LoRA experts

    Wu, X., Huang, S., and Wei, F. Mixture of LoRA experts. arXiv preprint arXiv:2404.13628, 2024

    2024 arXiv

  50. [58]

    Automated fine-grained mixture-of-experts quantization

    Xie, Z., Ma, Y., Zheng, X., Chao, F., and Ji, R. Automated fine-grained mixture-of-experts quantization

  51. [59]

    and McAuley, J

    Xu, C. and McAuley, J. A survey on model compression and acceleration for pretrained language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp.\ 10566--10575, 2023

    2023

  52. [60]

    and Chen, H.-H

    Yang, Y.-C. and Chen, H.-H. Dynamic DropConnect : Enhancing neural network robustness through adaptive edge dropping strategies. arXiv preprint arXiv:2502.19948, 2025

    2025 arXiv

  53. [61]

    B., Oh, G., and Gong, Y

    Yeh, S.-Y., Hsieh, Y.-G., Gao, Z., Yang, B. B., Oh, G., and Gong, Y. Navigating text-to-image customization: From lycoris fine-tuning to model evaluation. In The Twelfth International Conference on Learning Representations, 2023

    2023

  54. [62]

    ASVD : Activation-aware singular value decomposition for compressing large language models

    Yuan, Z., Shang, Y., Song, Y., Wu, Q., Yan, Y., and Sun, G. ASVD : Activation-aware singular value decomposition for compressing large language models. arXiv preprint arXiv:2312.05821, 2023

    2023 arXiv

  55. [63]

    J., et al

    Yuan, Z., Shang, Y., Zhou, Y., Dong, Z., Zhou, Z., Xue, C., Wu, B., Li, Z., Gu, Q., Lee, Y. J., et al. LLM inference unveiled: Survey and roofline model insights. arXiv preprint arXiv:2402.16363, 2024

    2024 arXiv

  56. [64]

    MiLoRA : Efficient mixture of low-rank adaptation for large language models fine-tuning

    Zhang, J., Zhao, Y., Chen, D., Tian, X., Zheng, H., and Zhu, W. MiLoRA : Efficient mixture of low-rank adaptation for large language models fine-tuning. arXiv preprint arXiv:2410.18035, 2024

    2024 arXiv

  57. [65]

    V., et al

    Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. OPT : Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022

    2022 arXiv

  58. [66]

    A survey on model compression for large language models

    Zhu, X., Li, J., Liu, Y., Ma, C., and Wang, W. A survey on model compression for large language models. Transactions of the Association for Computational Linguistics, 12: 0 1556--1577, 2024

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.