Shape Mutating Expert Compression:LorExperts and BTExperts

Ashish Sirasao; Bowen Bao; Inesh Chakrabarti; Sourjya Roy; Spandan Tiwari; Thiago Crepaldi

REVIEW 2 major objections 4 minor 14 references

Shape Mutating Expert Compression:LorExperts and BTExperts

T0 review · 2 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Local expert anchors preserve MoE accuracy at 50% expert compression without retraining the router.

desk verdict Local-anchor low-rank expert compression with strong empirical results, but the central low-rank premise is never directly measured. read the letter →

arxiv 2608.07814 v1 pith:ZNJLEU3F submitted 2026-08-07 cs.LG cs.AI

Inesh Chakrabarti , Sourjya Roy , Bowen Bao , Thiago Crepaldi , Spandan Tiwari , Ashish Sirasao This is my paper · ORCID

classification cs.LGcs.AI

keywords Mixture-of-Expertsexpertcompressionlow-rankdecompositionrouter-preservingco-activationclusteringneuronpermutationalignmentMoEinferenceBTExperts

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to solve a deployment problem: Mixture-of-Experts language models are cheap per token only if the expert weights fit in memory, and expert weights are growing faster than the memory budget. The authors argue that the usual fixes—pruning experts, merging similar ones, or representing every expert as one shared component plus a small delta—all break down, because expert weights are nearly orthogonal to one another yet still form functional co-activation communities. LorExperts instead clusters experts, keeps one full-precision 'dominant' per cluster, and writes every other member as a low-rank correction to its local dominant after permuting the member's neurons to align with the dominant. At roughly 50% expert compression on Qwen3-30B-A3B and Gemma-4-26B-A4B, it preserves downstream accuracy and perplexity better than pruning, merging, or single-shared-component decomposition on most tasks, and its margin over the single-anchor method grows with the number of experts. If right, this means large MoE models can be served at half the expert memory with no router retraining and no lost experts.

What carries the argument

The load-bearing object is the per-cluster dominant with permutation-aligned low-rank residuals. For each cluster, a full-precision dominant expert $W_d$ is kept; for every other member $W_m$, the method solves for a permutation $P$ of the member's neurons that minimizes the total squared distance between concatenated gate/up/down columns (Eq. 1–2, exact via Jonker-Volgenant), then writes the aligned member as $W_m^P \approx W_d + B_m A_m$ with $B_m, A_m$ from a rank-$r$ truncated SVD of the residual (Eq. 3–4). The insight that lets this work is the co-activation structure of the router: experts cluster into functional communities that are largely decoupled from weight similarity, so clustering by co-activation groups experts whose residual against a local dominant is actually small, and a frequency-seeded k-medoids step chooses dominants the router actually uses.

What would settle it

Record the singular-value spectrum of the aligned residual $W_m^P - W_d$ for a sample of member-dominant pairs. If the fraction of energy retained by a rank-64 truncation of these residuals is close to the flat-spectrum baseline (about 26% on these expert shapes, per the paper's own Table 2), then alignment is not making the residuals low-rank, and LorExperts is merely re-encoding near-full-rank differences. A complementary check: on a held-out domain that the calibration set underrepresents, such as math or code, if LorExperts at ~50% compression no longer beats D2-MoE at matched compression, the routing-aware clustering claim is domain-limited.

Watch

Extended reading notes

Core claim

LorExperts claims that MoE experts, despite being near-orthogonal in weight space, can be compressed at scale by replacing one global shared component with many local ones. The paper partitions experts into clusters using either weight distance or routing co-activation, keeps the highest-firing expert in each cluster as a full-precision dominant, aligns each remaining member's neurons to the dominant by solving a minimum-cost assignment over concatenated gate/up/down columns, and stores the member as the dominant plus a rank-64 SVD correction. Because every expert and the original router remain addressable, no router retraining is needed. The measured consequence is that at ~50% expert compression on Qwen3-30B-A3B and Gemma-4-26B-A4B, LorExperts preserves perplexity and knowledge/reasoning accuracy better than the baselines on most tasks, and its advantage over single-shared-component decomposition (D2-MoE) widens as the number of experts grows. The paper also introduces BTExperts, a binary-tree organization of dominants and corrections that does not improve quality but can amortize shared computation when the router co-selects experts in the same cluster.

Load-bearing premise

The load-bearing premise is that after permuting a member expert's neurons to match its dominant, the leftover difference $W_m^P - W_d$ can be captured by a small set of 64 correction patterns; if the aligned residuals stay nearly as complex as the original experts, LorExperts stops being real compression and becomes a storage scheme that moves the same information into different files.

Editorial extensions

If this is right

At ~50% expert compression on 128-expert models, knowledge- and reasoning-heavy metrics (MMLU, ARC-C, HellaSwag) hold up markedly better than with pruning or merging, so memory-bound deployment of large MoE models becomes feasible without router retraining.
Because the advantage over a single shared component grows with the expert count, the method targets the regime MoE designs are moving toward, E=128 and beyond.
Routing-aware clustering (co-activation NPMI or soft consensus) beats weight-similarity clustering, implying grouping should use functional routing statistics rather than raw weights.
The compression ratio is a tunable knob with a smooth quality frontier, so a practitioner can choose an operating point to hit a quality target instead of accepting a fixed degradation.
BTExperts can cut per-token FLOPs and memory traffic by roughly 1.9x when co-selected experts share a dominant, because the dominant is computed once and broadcast to the group.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

This raises a testable extension: check whether the same co-activation communities persist after compression; if they do, routing statistics alone could guide per-layer compression ratios, skipping the weight-space analysis.
The permutation-alignment step could be lifted into other compression schemes: quantizing experts in a shared aligned basis may create redundancy that reduces quantization error, a direction the paper does not explore.
The paper's ongoing Hessian-weighted anchor suggests a stronger general result: the optimal shared component is a traffic-weighted combination of the cluster rather than any existing expert, which would carry a guarantee that using an anchor never hurts; this remains outside the paper's reported experiments.
BTExperts' speedup depends on the router co-selecting experts from the same cluster; a routing regularizer that nudges co-selection could increase amortization, at the cost of altering the learned routing.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague, here's my take on arXiv:2608.07814.

The new idea is a local-anchor variant of delta decomposition: instead of one shared component for all experts, LorExperts clusters experts by co-activation, keeps a full-precision dominant per cluster, and encodes each remaining member as a permutation-aligned low-rank correction to that dominant. The router and all experts stay untouched. The empirical results are genuinely good: on Qwen3-30B-A3B, the best LorExperts variant reaches PPL ~12 vs D2-MoE's 23.6 at similar compression, and MMLU 54.9 vs 45.7. The paper also deserves credit for what it does not claim: BTExperts is honestly labeled as quality-neutral, and the co-activation analysis in Sec 2 is a useful measurement.

The main soft spot is that the paper never directly measures the thing its compression depends on: the spectral decay of the aligned residual R = W_m^P - W_dominant. Table 2 reports spectra for raw expert matrices only, where rank 64 retains just 26% of Frobenius energy. If the aligned residual has similar flatness, the method's stated mechanism cannot deliver the claimed 50% compression; the storage saving would still be real, but the quality would rely entirely on the reconstruction fine-tuning, which is a different story. This is easy to fix with one table: report the aligned-residual energy fraction or reconstruction error. Without it, the central claim rests on an unverified assumption.

Secondary issues: Tables 4-5 have no error bars or seeds; the best variant (Msoft) is reported at 40.6% and 41.4% compression in the headlines while baselines are at ~50%, so the headline margin is somewhat unfair (though 51% variants are also shown and still beat D2-MoE). And the 'margin grows with E' claim is not backed by a controlled experiment varying E. These are minor, but the first is load-bearing.

All in all, I'd send this to a serious referee. The method is well-motivated, the results are strong enough to matter, and the missing analysis is a gap a referee can close. I'd ask for the aligned-residual spectra and error bars before acceptance. This is worth a reading group slot, too, because the gap between mechanism and evidence is instructive.

Referee Report

2 major / 4 minor

Summary. This paper proposes LorExperts, a router-preserving expert compression method for MoE LLMs. It clusters experts per layer (using co-activation or weight-based distances), keeps one full-precision dominant expert per cluster, and represents each other member as W_dominant + B_m A_m, where the residual is formed after aligning the member's neurons to the dominant via the Jonker-Volgenant algorithm. It also introduces BTExperts, a tree arrangement of dominants and corrections for inference-time amortization, together with a reconstruction fine-tuning procedure. The main empirical claim is that at roughly 50% expert compression on Qwen3-30B-A3B and Gemma-4-26B-A4B, LorExperts preserves perplexity and downstream accuracy better than pruning (REAP, Freq-Prune), merging (M-SMoE, HC-SMoE), and single-shared-component delta decomposition (D2-MoE), while keeping all experts and the original router intact.

Significance. If the empirical claim holds, the paper makes a useful contribution: it identifies a scalable alternative to single-anchor delta decomposition, provides a mechanistic explanation in terms of co-activation communities that are decoupled from weight similarity, and uses the concentration of routing mass to justify keeping full-precision dominants. The paper is also honest in its limitations, explicitly noting that the BTExperts tree is quality-neutral and that its amortization benefit is routing-dependent, and it includes a controlled tree-vs-flat ablation with seeds. However, the central low-rank premise is not directly measured, and the best variant is evaluated at an unmatched compression ratio in the headline tables; these gaps must be closed before the central claim can be accepted.

major comments (2)

[Sec. 3, Stage 3(i)-(ii), Eq. (3)] The load-bearing premise of the method is that after permutation alignment, the residual R = W_m^P - W_dominant is well approximated at rank r=64, but no evidence for this is reported. Section 3, Stage 3(i)-(ii) and Eq. (3) construct the entire member representation from this residual, while Table 2 reports spectra only for raw expert matrices (rank-64 retains only 26% of Frobenius energy). Please report, for at least the same layers/experts as Table 2, the reconstruction error or energy fraction of the aligned residual at r=64 (and a sweep over r), and add an ablation of downstream PPL with and without the permutation-alignment step. If the aligned residual is not distinctly lower-rank than the raw expert, the stated mechanism cannot deliver the claimed roughly 50% compression.
[Tables 4-5 and Sec. 6] The headline '~50% expert compression' comparison is not made at matched compression ratios for the best variant. In Tables 4 and 5, the strongest variant (Msoft) is evaluated at 40.6%/41.4%, while all baselines are at ~50%; the matched Msoft rows at ~51%/51.6% show a large degradation on Qwen (PPL 15.37 vs 11.99, MMLU 46.0 vs 54.9). This makes the abstract's claim that LorExperts 'at ~50%' beats baselines rest on the WS-Frob/Coact-NPMI rows rather than on the best variant. Please present the main comparison at matched compression for all methods, either by evaluating baselines at ~40% or by using a ~50% operating point of the strongest variant.

minor comments (4)

[Tables 4-5] Tables 4 and 5 do not report variances or the number of runs; some margins between variants are small (e.g., Qwen PPL 12.36 vs 12.20 for WS-Frob vs Coact-NPMI), so error bars or seeds are needed to support ranking claims.
[Abstract and Sec. 8] The statement that the margin over D2-MoE 'grows with expert count E' is not directly tested, because no controlled sweep over E on a single architecture is presented; the two model families differ in more than E. This claim should either be backed by such a sweep or softened.
[Sec. 6.2 and Limitations] Section 6.2 claims a guaranteed ~100x speedup from a closed-form factor fit, but absolute wall-clock times are omitted and the Limitations section states the closed-form solve is 'not yet the default'. Please clarify which fitting procedure produced Tables 4-5 and whether the speedup claim refers to the current implementation or a future one.
[Throughout] There are a few presentation issues: 'V olgenant' should be 'Volgenant' (Sec. 3, Stage 3(i)); the reference 'Sec. 6, Gate H' in the Limitations section appears to be a stale cross-reference; and Fig. 5 would be easier to read if the operating points of Table 4 were marked on the curves.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: LorExperts' compression construction and accuracy claims rest on explicit equations and external benchmarks; the unverified aligned-residual low-rank premise is a correctness risk, not a circular reduction.

full rationale

Walking the derivation chain: Sec. 2 supplies independent measurements (near-orthogonal weights via Table 1, slow spectral decay via Table 2, routing concentration via Table 3); Sec. 3 defines the representation Wm ≈ Wdominant + BmAm with explicit equations (1)–(4), including the Jonker-Volgenant alignment and the rank-r SVD of the residual. No parameter is fitted to the downstream benchmarks: Stage 4 fits only the low-rank factors to cached calibration outputs, and Tables 4–5 evaluate on Wikitext-103, MMLU, ARC-C, HellaSwag, WinoGrande, and PIQA, which are external to the fitted values. The claim that the margin over D2-MoE grows with expert count E is an empirical observation from the table comparisons, not a consequence of the method's definitions. The only self-reference, the Figure 4 caption 'Adapted from the prior draft's FLOP analysis,' is not load-bearing. The paper's own Limitations section notes calibration cost and bounded amortization but does not assert any circular dependency. The reviewer-identified gap—absence of spectra or reconstruction error for the permutation-aligned residuals R = Wm^P − Wdominant—is a missing measurement that undermines confidence in the low-rank premise, but it is not a case of the derivation reducing to its own inputs. The co-activation structure is measured on the same model and calibration set used to build the compression, yet the downstream evaluation is independent of that measurement, so no fitted input is renamed as a prediction. Overall, the paper is self-contained against external benchmarks and shows no significant circularity.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The method's central claim rests on hand-set hyperparameters (r, k, protection set size, operating compression point) and several empirical domain assumptions about expert structure and calibration representativeness. No new physical entities are introduced; BTExperts is a data structure, not a postulated entity.

free parameters (4)

low_rank_rank_r = 64
Hand-set correction rank used for all reported results; directly controls compression ratio and approximation fidelity.
cluster_count_k = approximately E/2
Number of full-precision dominants per layer, chosen to reach ~50% expert compression; exact per-layer k is not reported.
protection_set_size = 8 (protK8 variant)
Number of full-precision protected experts per layer in WS-Frob+protK8; selected by hand, not by a stated criterion.
operating_compression_point = 40.6%-51.6% depending on variant
Msoft is reported at multiple compression levels; best MMLU points are at lower compression than the ~50% baselines, so the comparison point is a tunable choice.

assumptions (6)

domain assumption MoE expert weight matrices are near-orthogonal
Section 2 opens with this claim and uses it to argue single-anchor decomposition fails; Table 1 shows weak weight-co-activation correlations but no direct pairwise orthogonality measurements.
domain assumption Neuron permutation alignment by Jonker-Volgenant on concatenated gate/up/down columns yields a functionally correct matching with low-rank residual
Section 3 Stage 3(i)-(ii) and Eq. (3) rely on this; no aligned-residual spectra or alignment quality metrics are reported.
domain assumption Co-activation NPMI communities are stable across datasets and representative for clustering at every layer
Section 2.1 asserts this from 13 datasets and supports it with figures; stability is presented visually, not quantified per layer.
domain assumption Small calibration set (C4, 64 sequences of length 2048) provides representative firing counts and routing concentration
Stage 1 uses this set and Section 2.3 measures routing concentration on WikiText-2; the paper itself notes the busy-expert set is domain-dependent.
standard math Activation-weighted low-rank regression has a closed-form rank-r optimum given by whitening and generalized SVD
Sections 6.2 and 7 assert that ~500 gradient steps can be replaced by one whitened SVD at the same optimum, with no proof or wall-clock measurement.
domain assumption The top-k router remains a valid routing policy when expert outputs are approximated
The method keeps every expert and the gate, so routing is unchanged, but the implicit assumption that router logits remain well-calibrated after approximation is untested.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Shape Mutating Expert Compression:LorExperts and BTExperts." pith.science (2026). https://pith.science/paper/ZNJLEU3F

@misc{pith2026260807814,
  author       = {Pith},
  title        = {Pith review of: Shape Mutating Expert Compression:LorExperts and BTExperts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZNJLEU3F}},
  note         = {Machine review of arXiv:2608.07814}
}

read the original abstract

Mixture-of-Experts (MoE) language models deliver high capacity at low per-token compute, but deploying them cheaply requires compressing their many expert weight matrices. Expert pruning (e.g., REAP) and merging reduce cost but sacrifice accuracy and require retraining the router; low-rank delta decomposition of experts (e.g., D^2-MoE) preserves all experts and the router, but degrades sharply as the expert count grows because a single shared component cannot approximate many near-orthogonal experts. Because MoE expert weights are near-orthogonal, a single shared component (as in prior delta decomposition) scales poorly with the expert count; we show that experts nonetheless organize into functional co-activation communities that are decoupled from weight similarity. Building on this, we introduce LorExperts, a router-preserving compression method that clusters experts, keeps one full-precision dominant per cluster, and represents the remaining members as low-rank corrections to their local dominant. LorExperts retains all experts and the original router (no router retraining). At ~50% expert compression on Qwen3-30B-A3B and Gemma-4-26B-A4B, LorExperts preserves downstream accuracy and perplexity better than the baselines on most of the tasks; the margin over D^2-MoE grows with expert count E. We further give a reconstruction fine-tuning procedure for LorExperts, and BTExperts, a tree organization of dominants and corrections that enables inference-time amortization of shared computation.

Figures

Figures reproduced from arXiv: 2608.07814 by the authors.

**Figure 1.** LorExperts overview. Experts are clustered (by co-activation or weight similarity); each cluster keeps its highest-firing expert as a full-precision dominant Wd and represents every other member as Wd + BmAm with a rank-r correction. Unlike single-shared-component decomposition, which uses one global root, LorExperts uses k local dominants, so approximation quality does not collapse as the expert count E grows. The … view at source ↗

**Figure 2.** Expert co-activation structure at Qwen3-30B-A3B layer 23 as a [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** BTExperts tree (E=8). Shared components near the root are computed once and amortized across experts that share a path; leaves are experts. The highlighted red path is path(e3). The tree is an inference-time organization, not a compression-quality mechanism. Two caveats keep it honest: the benefit is routing-dependent (it applies only when the router co-selects experts from the same cluster), and it is shared with a… view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: Per-token forward pass (gate_proj, top-2 routing). [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

**Figure 5.** Figure 5: Compression–quality frontier for LorExperts [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]

**Figure 6.** Figure 6: Union-mean NPMI co-activation matrix at Qwen3-30B-A3B layer 47 (over 13 datasets), experts reordered by community: co-firing experts form clear diagonal blocks even though their weights are nearorthogonal. NPMI heatmap at a late layer, and [PITH_FULL_IMAGE:figures/f…

**Figure 7.** Figure 7: Per-layer counts of dense expert cliques by [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 8 canonical work pages

[1]

Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa

Samuel K. Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. 2023. Git re-basin: Merging models modulo permutation symmetries. In International Conference on Learning Representations (ICLR)

work page 2023
[2]

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

work page 2020
[3]

I-Chun Chen, Hsu-Shen Liu, Wei-Fang Sun, Chen-Hao Chao, Yen-Chang Hsu, and Chun-Yi Lee. 2025. Retraining-free merging of sparse MoE via hierarchical clustering. In International Conference on Machine Learning (ICML), volume 267, pages 8594--8620

work page 2025
[4]

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradb...

work page 2023
[5]

Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y

Damai Dai, Chengqi Deng, Chenggang Zhao, R.X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y.K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. 2024. DeepSeekMoE : Towards ultimate expert specialization in mixture-of-experts language models. In Proceedings of the 62nd Annual Meeting of the Ass...

work page 2024
[6]

Artyom Eliseev and Denis Mazur. 2023. Fast inference of mixture-of-experts language models with offloading. arXiv preprint arXiv:2312.17238

arXiv 2023
[7]

Lee, Shengjie Sun, Wei Xue, and Yike Guo

Hao Gu, Wei Li, Lujun Li, Qiyuan Zhu, Mark G. Lee, Shengjie Sun, Wei Xue, and Yike Guo. 2025. Delta decompression for MoE -based LLM s compression. In International Conference on Machine Learning (ICML), volume 267, pages 20497--20514

work page 2025
[8]

Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA : Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR)

2022

Show all 14 references

[9]

Dick, Yuan Cheng, Fan Yang, Tun Lu, Chun Zhang, and Li Shang

Ruijun Huang, Fang Dong, Xin Zhang, Hengjie Cao, Zhendong Huang, Anrui Chen, Jixian Zhou, Mengyi Chen, Yifeng Yang, Mingzhi Dong, Yujiang Wang, Jinlong Hou, Qin Lv, Robert P. Dick, Yuan Cheng, Fan Yang, Tun Lu, Chun Zhang, and Li Shang. 2026. SD-MoE : Spectral decomposition fo...

2026
[10]

Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, L \'e lio Renard Lavaud, Lucile Saulnier, Marie-...

2024 arXiv
[11]

Mike Lasby, Ivan Lazarevich, Nish Sinnadurai, Sean Lie, Yani Ioannou, and Vithursan Thangarasa. 2026. REAP the experts: Why pruning prevails for one-shot MoE compression. In International Conference on Learning Representations (ICLR)

2026
[12]

Pingzhi Li, Zhenyu Zhang, Prateek Yadav, Yi-Lin Sung, Yu Cheng, Mohit Bansal, and Tianlong Chen. 2024. Merge, then compress: Demystify efficient SMoE with hints from its routing policy. In International Conference on Learning Representations (ICLR)

2024
[13]

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA : Open and efficient foun...

2023 arXiv
[14]

An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jia...

2025 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa

Samuel K. Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. 2023. Git re-basin: Merging models modulo permutation symmetries. In International Conference on Learning Representations (ICLR)

work page 2023

[2] [2]

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

work page 2020

[3] [3]

I-Chun Chen, Hsu-Shen Liu, Wei-Fang Sun, Chen-Hao Chao, Yen-Chang Hsu, and Chun-Yi Lee. 2025. Retraining-free merging of sparse MoE via hierarchical clustering. In International Conference on Machine Learning (ICML), volume 267, pages 8594--8620

work page 2025

[4] [4]

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradb...

work page 2023

[5] [5]

Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y

Damai Dai, Chengqi Deng, Chenggang Zhao, R.X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y.K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. 2024. DeepSeekMoE : Towards ultimate expert specialization in mixture-of-experts language models. In Proceedings of the 62nd Annual Meeting of the Ass...

work page 2024

[6] [6]

Artyom Eliseev and Denis Mazur. 2023. Fast inference of mixture-of-experts language models with offloading. arXiv preprint arXiv:2312.17238

arXiv 2023

[7] [7]

Lee, Shengjie Sun, Wei Xue, and Yike Guo

Hao Gu, Wei Li, Lujun Li, Qiyuan Zhu, Mark G. Lee, Shengjie Sun, Wei Xue, and Yike Guo. 2025. Delta decompression for MoE -based LLM s compression. In International Conference on Machine Learning (ICML), volume 267, pages 20497--20514

work page 2025

[8] [8]

Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA : Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR)

2022