Pith. sign in

REVIEW 3 major objections 3 minor 2 cited by

Monet: Mixture of Monosemantic Experts for Transformers

T0 review · 3 major / 3 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read With 262,144 experts per layer, deleting the few tuned to a domain, language, or toxicity erases that knowledge while general performance holds.

desk verdict Monet's expert-composition architecture is a real contribution, but the knowledge-manipulation results rest on in-sample expert selection and need a held-out evaluation before they can be trusted. read the letter →

arxiv 2412.04139 v4 pith:R3KTTOAC submitted 2024-12-05 cs.AI

Jungwoo Park , Young Jin Ahn , Kee-Eung Kim , Jaewoo Kang This is my paper · ORCID
classification cs.AI
keywords mixture-of-expertsmonosemanticitymechanisticinterpretabilitysparsedictionarylearningproductkeycompositionknowledgeunlearningtoxicitymitigationparameter-efficientscaling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a transformer can be made interpretable by construction: decompose each feedforward expert into two shared parameter pools so that 262,144 experts exist per layer while total parameters grow only as the square root of the expert count. At that granularity, the paper argues, individual experts become monosemantic — each one fires for a single concept, such as chemical compounds, U.S. states, or the stem 'Bay' in either a geographic or a Bayesian sense. The payoff is claimed to be surgical knowledge manipulation: removing the experts assigned to a domain, a programming language, or toxicity drops performance on exactly that target while leaving everything else nearly intact. If correct, this would collapse mechanistic interpretability and model editing into the pretraining objective itself, as a built-in alternative to post-hoc sparse autoencoders, which the paper argues degrade model quality.

What carries the argument

The central object is product key composition through expert decomposition. Instead of storing $N$ expert MLPs, the model stores two pools of $\sqrt{N}$ parameter blocks each — bottom versus top layers in the horizontal variant, left versus right segments in the vertical variant — and treats their Cartesian product as the expert set. Routing scores factorize as $g_{hij} = g^1_{hi}g^2_{hj}$, so retrieval never enumerates the $N$ composed experts; batch-normalization quantile estimates take the place of top-$k$ sorting, and two auxiliary losses (a uniformity loss over routing distributions and an ambiguity loss against low-confidence routing) push the model toward uniform-but-specialized expert usage. The load-bearing identity is the rearranged summation of Equation 12, which turns the composed-expert output into per-pool precomputable operations and is what makes $O(\sqrt{N})$ parameter scaling compatible with $O(\sqrt{N})$ retrieval time.

What would settle it

Split each MMLU domain's questions into two disjoint halves, compute expert routing skewness on half A, delete the skewed experts, and measure accuracy on half B; if the target-domain drop shrinks toward the cross-domain drop under this held-out protocol, the reported surgical unlearning (for example Biology at -4.66 points, other domains at -0.42) is an artifact of selecting experts on the evaluation set itself.

Watch

Extended reading notes

Core claim

MONET replaces each standalone expert network with a composed one built from two parameter pools. In the horizontal variant the model stores $\sqrt{N}$ bottom projections $\{U_i\}_{i=1}^{\sqrt{N}}$ and $\sqrt{N}$ top projections $\{V_j\}_{j=1}^{\sqrt{N}}$; the $(i,j)$-th expert is $E_{ij}(x) = V_j\,\sigma(U_i x + b^1_i) + b^2_j$, so $N$ distinct experts cost only $O(\sqrt{N})$ parameters. Routing follows the product-key scheme: the hidden state is split in half, independent top-$k$ selections are made on each half, and the composed expert's routing score is the product of its two components' scores; a reordering of the double summation precomputes each pool's contribution, keeping retrieval cost at $O(\sqrt{N}Hd)$. At 262,144 experts per layer, the paper claims, routing becomes strongly skewed — each token commits to a single expert — and this yields mutually exclusive knowledge: distinct experts fire for 'Bay Area' versus 'Bayesian', and deleting the experts skewed toward a target MMLU domain drops that domain by roughly 1 to 5 accuracy points while other domains shift by a few tenths of a point.

Load-bearing premise

Everything claimed about unlearning rests on the assumption that an expert's tendency to fire on a benchmark's questions reveals where that knowledge actually lives, even though the same questions are used both to choose which experts to delete and to score how well the deletion worked.

Editorial extensions

If this is right

  • Interpretability can be scaled cheaply: because total parameters grow as the square root of the expert count, the 262,144-expert regime is not a one-off but a range in which finer-grained routing is available at modest memory cost.
  • Knowledge control reduces to a delete operation: purging routing-skewed experts drops the target MMLU domain by up to about 5.5 accuracy points, the target language's pass@100 by up to 31 points, and expected maximum toxicity from 0.795 toward 0.55, while other-domain accuracy moves by well under a point and average helpfulness stays nearly flat.
  • Post-hoc sparse autoencoders become unnecessary for feature attribution, because the expert dictionary is learned end-to-end with the language-modeling loss and there is no reconstruction error to cascade through the network.
  • The architecture carries over to code modeling and to vision-language fine-tuning, with the paper reporting monosemantic specialization preserved across those settings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The factorization principle is general: any decomposition of expert weights into two or more shared pools — not just bottom/top or left/right splits — converts the parameter cost of $N$ experts from linear to roughly $N^{1/k}$, so tensor or multilinear expert decompositions are a natural next step the paper does not explore.
  • The manipulation numbers are upper bounds: experts are identified by routing skewness on the same MMLU or MultiPL-E items used to evaluate the deletion, so a held-out version of the protocol would likely show smaller target-domain drops; the paper itself concedes its expert-selection criteria are basic and minimal.
  • A scaling law for monosemanticity is directly testable: train MONET variants with, say, 4K, 16K, 64K, and 256K experts under matched budgets and measure whether routing entropy falls and expert exclusivity rises with expert count, as the paper's claims imply.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper proposes Monet, a Mixture-of-Experts transformer variant in which each of N composed experts is formed as a product-key combination of shared bottom/top (horizontal decomposition) or left/right (vertical decomposition) weight matrices, allowing N = 262,144 experts per layer with expert-parameter growth O(√N). The authors pretrain models at 850M, 1.4B, and 4.1B scale plus a code model and a chat model, evaluate general-domain benchmarks, present qualitative routing examples, and report domain, language, and toxicity manipulation by deleting experts identified through routing statistics. The parameter-complexity derivations (Eq. 12, Eqs. 30-31) are self-consistent, and the paper releases source code and pretrained checkpoints.

Significance. If the manipulation claims survive held-out evaluation, Monet would be a meaningful contribution to interpretable-by-construction MoE architectures: the √N expert-parameter scaling is a clean and useful alternative to PEER-like linear scaling, and the qualitative specialization examples (e.g., Figure 2) are striking. The architectural core is credible and well positioned against existing MoE and SAE baselines. However, the quantitative evidence for precise knowledge manipulation currently rests on expert identification and evaluation on the same benchmark instances, and the monosemanticity claim is not quantified. These issues are fixable but are load-bearing for the paper's headline interpretability and controllability claims.

major comments (3)
  1. [§5.1, §5.2, §5.3; Appendices D.1-D.3] The manipulation experiments identify the experts to delete on the same benchmark on which the deletion effect is measured. For domain masking, Appendix D.1 assigns experts to MMLU domains using routing probabilities computed on the reorganized MMLU questions, and the same MMLU benchmark is then used to measure target-domain accuracy; the reported ΔTarget in Figure 3 and Tables 11-14 therefore conflates in-sample selection with genuine unlearning. An expert is selected precisely because it activates on the evaluation examples, so removing it lowers their accuracy even if it is not a general domain-knowledge store. The same protocol is used in §5.2/D.2 for MULTIPL-E and in §5.3/D.3 for RealToxicityPrompts/ToxiGen, where Pearson correlations are computed on the same prompts as the evaluation. The out-of-sample helpfulness results (Table 16) show only mild degradation, but the central claim of precise, disentangled manipulation depends on target-domain drops that are not demonstrated out of sample. Please re-run the experiments with a strict held-out split—for example, identifying experts on one half of the benchmark and evaluating on the other half, with folds—and report whether the ΔTarget/ΔOthers contrast survives.
  2. [§3, Eqs. (8)-(12); §5] The paper does not specify how an individual composed expert (i,j) is 'deleted'. Because E_ij is built from shared U_i and V_j, zeroing a router component (g1_i or g2_j) removes an entire row or column of the Cartesian product, while masking a single (i,j) pair cannot be done inside the factorized computation of Eq. (12) without computing that pair's contribution explicitly. With reported masking ratios as high as 14.4% of experts (Table 4), this distinction matters both for the semantics of 'expert removal' and for the claimed O(√N) computation benefit. Please state the exact masking operation (component-level vs. pair-level) and, if pair-level, provide the implementation and its complexity.
  3. [§4.3, Fig. 2; §6 Limitations] The claim of 'mutual exclusivity of knowledge across experts' is supported only by qualitative routing examples and by the manipulation experiments, whose in-sample status is noted above. The paper's own Limitations section acknowledges that the self-explained expert descriptions are 'demonstrated only qualitatively' and that quantitative evaluation of automated interpretability remains open. Please add a quantitative selectivity or exclusivity metric—for example, overlap of top-activated token sets across experts, or routing-score skewness on a held-out corpus—and report it for the full expert population rather than for hand-picked examples.
minor comments (3)
  1. [Abstract; §1] The statements that 'total parameters scale proportionally to the square root of the number of experts' are imprecise; Eqs. (30)-(31) establish O(√N md) for expert parameters only. Please rephrase to 'expert parameters' or 'the expert parameter contribution' to avoid overclaiming.
  2. [Figure 3; Tables 11-14] The heatmaps and the dense 14×15 numeric tables are difficult to read at print size; consider reporting only ΔTarget and ΔOthers with a clear color scale and confidence intervals.
  3. [Appendix D.3] The toxicity identification procedure uses Pearson correlation between routing scores and toxicity scores, but the paper does not state whether the same prompts are used for correlation, threshold selection, and evaluation; please make this explicit (this is related to Major Comment 1).

Circularity Check

3 steps flagged · score 6.0 of 10

In-sample expert identification inflates the domain, language, and toxicity manipulation claims, although the O(sqrt N) scaling derivation is self-contained.

  1. fitted input called prediction [Section 5.1, Appendix D.1]
    "Using the MMLU Pro (Wang et al., 2024) benchmark taxonomy, which divides question-answer sets into 14 distinct domains, we investigated the effects of domain-specific knowledge unlearning on MMLU (Hendrycks et al., 2021). For each expert, if the routing probability for a particular domain was at least twice as high as for the second most activated domain, we labeled that expert as specialized in that domain. After assigning experts to domains, we selectively deleted the experts and evaluated the impact of knowledge unlearning across all 14 domains."

    The same MMLU benchmark is used for both expert identification and evaluation. Experts are labeled as domain-specialized precisely because they show skewed routing probability on the MMLU questions of that domain, and the reported target-domain accuracy drop is then measured on those same MMLU questions. No held-out identification/evaluation split is described. The reported delta is therefore an in-sample selection effect rather than an out-of-sample test of where domain knowledge is stored.

  2. fitted input called prediction [Section 5.2, Appendix D.2]
    "In line with our approach for domain masking, we identified language-specific experts (see Table 10) by examining the skewness in routing probabilities. Based on this, we masked experts associated with each language and re-evaluated the code generation benchmark to estimate the model’s capability to unlearn programming languages."

    Language-specialized experts are selected from routing statistics computed on the MULTIPL-E benchmark, and the pass@100 drop after deletion is evaluated on the same MULTIPL-E benchmark. The experts are chosen because they activate on those language problems, so the large target-language drops in Table 3 are not independent predictions of language knowledge localization. The reported delta conflates selection and evaluation on the same examples.

1 more flagged steps
  1. fitted input called prediction [Section 5.3, Appendix D.3]
    "To identify toxic knowledge within the model, we collected expert routing scores alongside toxicity scores, and computed Pearson correlations. A higher correlation indicates a greater likelihood of an expert being selected when toxic content is generated. Based on predefined thresholds, we removed experts with high toxicity correlations."

    Toxic experts are fitted by correlating routing scores with toxicity scores on the same RealToxicityPrompts-style prompts whose toxicity is later measured after purging in Table 4. The expert set is determined by the very labels used as the evaluation outcome, so the observed drop in expected maximum toxicity and toxicity probability is an in-sample consequence of the fitting procedure. The paper does not describe a held-out prompt split for identification versus evaluation.

full rationale

The architecture claim is not circular: the O(sqrt N) parameter scaling in Section 3 and Appendix A.2 is a direct count of the factored expert weights, independent of the manipulation experiments. The qualitative monosemanticity examples on C4 and StarCoder are also separate evidence. However, the paper's central manipulation claims for domains, languages, and toxicity all follow the same protocol: identify experts using routing statistics or correlations computed on a benchmark, delete those experts, and report the effect on that same benchmark. Because no held-out identification/evaluation split is described, the large target drops in Figures 3 and Tables 3-5 are partially forced by in-sample selection. The ToxiGen results and the 8-benchmark helpfulness table provide some external signal, and the authors candidly note that automated expert interpretation remains qualitative, but those do not rescue the specific target-domain and target-language unlearning claims from being in-sample. This is partial circularity rather than complete equivalence, so the score is 6 rather than higher.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

No new physical entities are introduced; the architecture components are methods, not entities. The central claim rests on design hyperparameters and domain assumptions about routing as a proxy for knowledge storage.

free parameters (4)
  • Auxiliary loss weight lambda = 1e-3
    Added to the language modeling loss in Eq. 18; chosen via ablation in Appendix C.1. Affects routing specialization and all manipulation results.
  • Domain specialization skewness threshold = 2x routing probability
    Used in Section 5.1 to label experts as domain-specialized before deletion. This hand-chosen criterion determines which experts are purged in the MMLU domain masking experiments.
  • Toxicity correlation threshold = 0.2, 0.1, 0.05
    In Section 5.3, experts with Pearson correlation above the chosen threshold between routing score and toxicity score are purged. The thresholds determine the masking ratio and the reported toxicity reduction.
  • Routing group size = 4 layers
    Router probabilities are reused every 4 layers (Appendix B.1), reducing routing parameters and FLOPs. Ablation in Table 8 shows the performance trade-off.
assumptions (5)
  • domain assumption Sparse dictionary learning via end-to-end MoE pretraining yields monosemantic expert specialization.
    The paper assumes that increasing expert count and adding uniformity/ambiguity losses produce experts that each capture a single concept. This is the interpretability premise, illustrated qualitatively but not proven.
  • domain assumption Routing-score skewness identifies where knowledge is stored.
    Domain and language masking assume that an expert having high routing probability for a domain implies that expert parametrically encodes that domain's knowledge, as used in Section 5.1 and Appendix D.1.
  • ad hoc to paper Deleting identified experts removes the associated knowledge without large collateral damage.
    The unlearning experiments rely on this causal assumption. The authors note the selection criteria are basic and minimal in the Limitations, but the evaluation does not independently validate the causal link.
  • domain assumption Batch Normalization statistics provide reliable quantile estimates for top-k routing.
    Adaptive routing with batch normalization (Section 3) replaces exact top-k sorting; this is an engineering assumption tested only through downstream performance.
  • standard math Linear algebra identities used to reorder summations in Eq. 12 and Eqs. 22-29 are valid.
    The efficient computation of decomposed experts relies on standard matrix multiplication and summation reordering; these identities are elementary and correct.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Monet: Mixture of Monosemantic Experts for Transformers." pith.science (2026). https://pith.science/paper/R3KTTOAC

@misc{pith2026241204139,
  author       = {Pith},
  title        = {Pith review of: Monet: Mixture of Monosemantic Experts for Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/R3KTTOAC}},
  note         = {Machine review of arXiv:2412.04139}
}
read the original abstract

Understanding the internal computations of large language models (LLMs) is crucial for aligning them with human values and preventing undesirable behaviors like toxic content generation. However, mechanistic interpretability is hindered by polysemanticity -- where individual neurons respond to multiple, unrelated concepts. While Sparse Autoencoders (SAEs) have attempted to disentangle these features through sparse dictionary learning, they have compromised LLM performance due to reliance on post-hoc reconstruction loss. To address this issue, we introduce Mixture of Monosemantic Experts for Transformers (Monet) architecture, which incorporates sparse dictionary learning directly into end-to-end Mixture-of-Experts pretraining. Our novel expert decomposition method enables scaling the expert count to 262,144 per layer while total parameters scale proportionally to the square root of the number of experts. Our analyses demonstrate mutual exclusivity of knowledge across experts and showcase the parametric knowledge encapsulated within individual experts. Moreover, Monet allows knowledge manipulation over domains, languages, and toxicity mitigation without degrading general performance. Our pursuit of transparent LLMs highlights the potential of scaling expert counts to enhance mechanistic interpretability and directly resect the internal knowledge to fundamentally adjust model behavior. The source code and pretrained checkpoints are available at https://github.com/dmis-lab/Monet.

Figures

Figures reproduced from arXiv: 2412.04139 by the authors.

Figure 1
Figure 1. Architectural comparison of expert scaling approaches in large language models. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Activated tokens for experts in LLMs (MONET-1.4B, MONET-4.1B) on C4 validation dataset. CODEMONET-1.4B’s examples were collected from the StarCoder dataset. Tokens are sorted according to the expert’s routing score (or ghij in Eq. 7), notated in parenthesis. Descriptions in bottom rows are self-explained experts, generated from the automated interpretation framework. Parametric Knowledge In MONET, feedforward MLP in… view at source ↗
Figure 3
Figure 3. Knowledge unlearning and accuracy perturbation across 14 MMLU domains. Rows rep [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: CODEMONET’s generation capability on Python problems in HumanEval dataset before and after purging Python experts. Expert pruning follows the schemes mentioned in D.1. Docstrings are the prompts that are given to the model for code completion task. For each of these la…
Figure 5
Figure 5. Figure 5: Detection of toxic experts through token activations and toxicity scores. The top row lists [PITH_FULL_IMAGE:figures/full_fig_p026_5.png]
Figure 6
Figure 6. Figure 6: Detoxified text completion examples based on prompts of R [PITH_FULL_IMAGE:figures/full_fig_p027_6.png]
Figure 7
Figure 7. Figure 7: List of qualitative examples according to the domains. [PITH_FULL_IMAGE:figures/full_fig_p031_7.png]
Figure 8
Figure 8. Figure 8: List of qualitative examples according to the programming languages. [PITH_FULL_IMAGE:figures/full_fig_p032_8.png]
Figure 9
Figure 9. Figure 9: List of image and text activation examples of vision-language model V [PITH_FULL_IMAGE:figures/full_fig_p033_9.png]
Figure 10
Figure 10. Figure 10: List of image and text activation examples of vision-language model V [PITH_FULL_IMAGE:figures/full_fig_p034_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Interpretability in Parameter Space: Minimizing Mechanistic Description Length with Attribution-based Parameter Decomposition

    cs.LG 2025-01 conditional novelty 7.0 of 10

    Attribution-based Parameter Decomposition splits a network's parameters into faithful, minimal, and simple components and recovers ground-truth mechanisms in toy models of superposition and compressed computation.

  2. Studying Cross-cluster Modularity in Neural Networks

    cs.LG 2025-02 conditional novelty 6.0 of 10

    A clusterability regularizer creates strongly separated clusters in neural networks, shrinking effective circuit size by up to 90% on CIFAR-10, but it does not create task-specialized modules.

Reference graph

Works this paper leans on

39 extracted references · 20 canonical work pages · cited by 2 Pith papers

  1. [1]

    Nemotron-4 340B Technical Report.arXiv preprint arXiv:2406.11704,

    Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H Anh, Pallab Bhattacharya, Annika Brundyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan Cohen, et al. Nemotron-4 340B Technical Report.arXiv preprint arXiv:2406.11704,

    arXiv

  2. [2]

    b", nn.initializers.zeros, b_shape) 11 12def __call__(self, x, g1, g2): 13x = nn.relu(self.u(x)) ** 2 14x = jnp.einsum(

    A.2 COMPLEXITYCALCULATIONS We present detailed derivations of computational complexity (expert retrieval time) and memory requirements for different expert architectures to demonstrate the efficiency of MONET. SMoEThe conventional SMoE architecture requires computing similarity scores between input vectors and all expert embeddings. For an inputx∈R d andN...

    work page 2019

  3. [4]

    Haozhe Chen, Carl V ondrick, and Chengzhi Mao

    doi: 10.1109/TSE.2023.3267446. Haozhe Chen, Carl V ondrick, and Chengzhi Mao. SelfIE: Self-Interpretation of Large Language Model Embeddings.arXiv preprint arXiv:2403.10949,

    arXiv 2023

  4. [6]

    F**kyou!F**k (...)* (16.68%)(...)Snakesonamotherf*ckingplane

    25 Published as a conference paper at ICLR 2025 Idiot – MONET-1.4B / Group 4 / Expert 3,400 id (65.68%)(...)Lt.Governorarebothidiots,butthat (...)id (59.73%)(...)’scharacterisacompleteidiotwhodoesthingsa (...)id (59.20%)(...)hehadhischaractersdowhateveridioticormund (...)id (58.20%)(...)timesintelligentandattimesidiotic,thedialog (...)id (58.14%)(...)gene...

    work page 2025

  5. [7]

    RealToxi- cityPrompts: Evaluating Neural Toxic Degeneration in Language Models

    13 Published as a conference paper at ICLR 2025 Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. RealToxi- cityPrompts: Evaluating Neural Toxic Degeneration in Language Models. InFindings of the Association for Computational Linguistics: EMNLP 2020, November

    work page 2025

  6. [8]

    Transformer Feed-Forward Layers Are Key-Value Memories

    Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer Feed-Forward Layers Are Key-Value Memories. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, November

    work page 2021

  7. [10]

    Mixture of a million experts.arXiv preprint arXiv:2407.04153,

    Xu Owen He. Mixture of a million experts.arXiv preprint arXiv:2407.04153,

    arXiv

  8. [11]

    An Overview of Catastrophic AI Risks

    Dan Hendrycks, Mantas Mazeika, and Thomas Woodside. An Overview of Catastrophic AI Risks. arXiv preprint arXiv:2306.12001,

    arXiv

Show all 39 references
  1. [12]

    AI Alignment: A Comprehensive Survey.arXiv preprint arXiv:2310.19852,

    Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, et al. AI Alignment: A Comprehensive Survey.arXiv preprint arXiv:2310.19852,

    arXiv

  2. [13]

    Mixtral of Experts.arXiv preprint arXiv:2401.04088,

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bam- ford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of Experts.arXiv preprint arXiv:2401.04088,

    arXiv

  3. [14]

    Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Mu ˜noz Ferran- dis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, et al

    URLhttps://www.alignmentforum.org/posts/ 8ev6coxChSWcxCDy8. Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Mu ˜noz Ferran- dis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, et al. The Stack: 3 TB of permissively licensed source code....

    arXiv

  4. [15]

    Theory on Mixture-of- Experts in Continual Learning.arXiv preprint arXiv:2406.16437,

    14 Published as a conference paper at ICLR 2025 Hongbo Li, Sen Lin, Lingjie Duan, Yingbin Liang, and Ness B Shroff. Theory on Mixture-of- Experts in Continual Learning.arXiv preprint arXiv:2406.16437,

    2025 arXiv

  5. [16]

    StarCoder: may the source be with you!arXiv preprint arXiv:2305.06161,

    Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. StarCoder: may the source be with you!arXiv preprint arXiv:2305.06161,

    arXiv

  6. [17]

    Scaling Laws for Fine-Grained Mixture of Experts

    Jan Ludziejewski, Jakub Krajewski, Kamil Adamczewski, Maciej Pi ´oro, Michał Krutul, Szymon Antoniak, Kamil Ciebiera, Krystian Kr ´ol, Tomasz Odrzyg´o´zd´z, Piotr Sankowski, Marek Cygan, and Sebastian Jaszczur. Scaling Laws for Fine-Grained Mixture of Experts. InICLR 2024 Work...

    2024

  7. [18]

    Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Mod- els.arXiv preprint arXiv:2403.19647,

    Samuel Marks, Can Rager, Eric J Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Mod- els.arXiv preprint arXiv:2403.19647,

    arXiv

  8. [19]

    Jesse Mu and Jacob Andreas

    URL https://proceedings.neurips.cc/paper_files/paper/2022/file/ 6f1d43d5a82a37e89b0665b33bf3a182-Paper-Conference.pdf. Jesse Mu and Jacob Andreas. Compositional Explanations of Neurons. InAdvances in Neural Information Processing Systems, volume 33, pp. 17153–17163,

    2022

  9. [20]

    OLMoE: Open Mixture-of-Experts Language Models.arXiv preprint arXiv:2409.02060,

    Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, et al. OLMoE: Open Mixture-of-Experts Language Models.arXiv preprint arXiv:2409.02060,

    arXiv

  10. [21]

    Dense Training, Sparse Inference: Rethinking Training of Mixture-of- Experts Language Models.arXiv preprint arXiv:2404.05567,

    Bowen Pan, Yikang Shen, Haokun Liu, Mayank Mishra, Gaoyuan Zhang, Aude Oliva, Colin Raffel, and Rameswar Panda. Dense Training, Sparse Inference: Rethinking Training of Mixture-of- Experts Language Models.arXiv preprint arXiv:2404.05567,

    arXiv

  11. [22]

    The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale.arXiv preprint arXiv:2406.17557,

    Guilherme Penedo, Hynek Kydl ´ıˇcek, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro V on Werra, Thomas Wolf, et al. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale.arXiv preprint arXiv:2406.17557,

    arXiv

  12. [23]

    Taking features out of superposition with sparse autoencoders

    15 Published as a conference paper at ICLR 2025 Lee Sharkey, Dan Braun, and Beren Millidge. Taking features out of superposition with sparse autoencoders

    2025

  13. [24]

    JetMoE: Reaching Llama2 Performance with 0.1M Dollars.arXiv preprint arXiv:2404.07413,

    Yikang Shen, Zhen Guo, Tianle Cai, and Zengyi Qin. JetMoE: Reaching Llama2 Performance with 0.1M Dollars.arXiv preprint arXiv:2404.07413,

    arXiv

  14. [25]

    Codebook Features: Sparse and Discrete Interpretability for Neural Networks.arXiv preprint arXiv:2310.17230,

    Alex Tamkin, Mohammad Taufeeque, and Noah D Goodman. Codebook Features: Sparse and Discrete Interpretability for Neural Networks.arXiv preprint arXiv:2310.17230,

    arXiv

  15. [26]

    Gemma 2: Improving Open Language Models at a Practical Size.arXiv preprint arXiv:2408.00118,

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu- patiraju, L´eonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram ´e, et al. Gemma 2: Improving Open Language Models at a Practical Size.arXiv preprint arXiv:2408.00118,

    arXiv

  16. [27]

    ReLU 2 Wins: Discovering Efficient Activation Func- tions for Sparse LLMs.arXiv preprint arXiv:2402.03804,

    Zhengyan Zhang, Yixin Song, Guanghui Yu, Xu Han, Yankai Lin, Chaojun Xiao, Chenyang Song, Zhiyuan Liu, Zeyu Mi, and Maosong Sun. ReLU 2 Wins: Discovering Efficient Activation Func- tions for Sparse LLMs.arXiv preprint arXiv:2402.03804,

    arXiv

  17. [28]

    CONTENTS A Method Descriptions 18 A.1 Expansion of Vertical Decomposition

    16 Published as a conference paper at ICLR 2025 Appendix Content Warning: This section contains examples of harmful language. CONTENTS A Method Descriptions 18 A.1 Expansion of Vertical Decomposition . . . . . . . . . . . . . . . . . . . . . . . . 18 A.2 Complexity Calculation...

    2025

  18. [29]

    Moreover, the multi-head expert routing probabilities are consoli- dated into single routing coefficients PH h=1 ˆg1 hi and PH h=1 ˆg2 hj, reducing redundant aggregations

    These computations can be simplified as follows: X11 = HX h=1 √ NX i=1 √ NX j=1 ˆg1 hiˆg2 hjV 11 i σ(U 1 i x+b 11 i ) = √ NX i=1 HX h=1   √ NX j=1 ˆg2 hj   ˆg1 hiV 11 i σ(U 1 i x+b 11 i )(22) = √ NX i=1 HX h=1 ˆg1 hi ! V 11 i σ(U 1 i x+b 11 i ),(23) X22 = HX h=1 √ NX i=1 √...

    2025

  19. [31]

    implementation of a MONET-HD layer. 1class MonetMoVDE(nn.Module): 2dim: int = 2048 3moe_dim: int = 16 4moe_experts: int = 512 5 6def setup(self): 7self.u1 = nn.DenseGeneral((self.moe_experts, self.moe_dim // 2)) 8self.u2 = nn.DenseGeneral((self.moe_experts, self.moe_dim // 2))...

  20. [32]

    b1", nn.initializers.zeros, b_shape) 16self.b2 = self.param(

    15self.b1 = self.param("b1", nn.initializers.zeros, b_shape) 16self.b2 = self.param("b2", nn.initializers.zeros, b_shape) 17 18def __call__(self, x, g1, g2): 19x1, x2 = nn.relu(self.u1(x)) ** 2, nn.relu(self.u2(x)) ** 2 20 21x11 = self.v11(jnp.einsum("btim,bthi->btim", x1, g1)...

    2025

  21. [33]

    To manage computational resources effectively, we adopt a group routing strategy wherein the routing probabilities are reused every 4 layers

    as the activation function. To manage computational resources effectively, we adopt a group routing strategy wherein the routing probabilities are reused every 4 layers. This approach reduces the overhead associated with the expert routing parameters. The weight of the auxilia...

    2023

  22. [34]

    and encompasses approxi- mately 86 programming languages. B.2 INSTRUCTIONTUNING To enhance the conversational and instructional capabilities of our models, we perform instruction tuning on the MONET1.4B model following the instruction tuning recipe (Tunstall et al.) used by SM...

    2024

  23. [35]

    The ta- ble reports the number of experts assigned to each programming language across all routing groups

    We re- port parameter size, FLOPs (TFLOPs) for forward computation over 2M tokens, and the 5-shot 22 Published as a conference paper at ICLR 2025 Language Group 1 Group 2 Group 3 Group 4 Group 5 Group 6 Total Python 7,813 9,616 8,844 7,580 10,791 12,518 57,162 C++ 7,144 11,436...

    2025

  24. [36]

    "" 12#!/usr/bin/env bash 13 14echo

    23 Published as a conference paper at ICLR 2025 CODEMONET-1.4B / Python / HumanEval-0 Original 1from typing import List 2 3 4def has_close_elements(numbers: List[float], threshold: float) -> bool: 5""" Check if in given list of numbers, are any two numbers closer to each other...

    2025

  25. [37]

    Based on this, we masked experts associated with each language and re-evaluated the code generation benchmark to estimate the model’s capa- bility to unlearn programming languages

    by examining the skewness in routing probabilities. Based on this, we masked experts associated with each language and re-evaluated the code generation benchmark to estimate the model’s capa- bility to unlearn programming languages. 24 Published as a conference paper at ICLR 2...

    2025

  26. [2018]

    JULYIV (...)rew (59.50%)(...)TheembroideryreadsinHebrew:

    dataset, based on the routing score of a multimodal expert. 33 Published as a conference paper at ICLR 2025 Dogs – VISIONMONET-1.4B / Group 4 / Expert 100,768 agle (85.75%)(...)pherdmaltesebeaglerottweilerd (...)og (85.33%)(...)ahuapugbulldoggermanshepherd (...)iler (82.13%)(....

    2025

  27. [2020]

    BatchTopK: A Simple Improvement for TopK- SAEs.AI Alignment F orum,

    12 Published as a conference paper at ICLR 2025 Bart Bussmann, Patrick Leask, and Neel Nanda. BatchTopK: A Simple Improvement for TopK- SAEs.AI Alignment F orum,

    2025

  28. [2021]

    Patchscope: A Unifying Framework For Inspecting Hidden Representations of Language Models.arXiv preprint arXiv:2401.06102,

    Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, and Mor Geva. Patchscope: A Unifying Framework For Inspecting Hidden Representations of Language Models.arXiv preprint arXiv:2401.06102,

    arXiv

  29. [2022]

    William Fedus, Jeff Dean, and Barret Zoph

    URLhttps://transformer-circuits.pub/ 2022/toy_model/index.html. William Fedus, Jeff Dean, and Barret Zoph. A Review of Sparse Expert Models in Deep Learning. arXiv preprint arXiv:2209.01667, 2022a. William Fedus, Barret Zoph, and Noam Shazeer. Switch Transformers: Scaling to T...

    2022 arXiv

  30. [2023]

    URLhttps://transformer-circuits.pub/ 2023/monosemantic-features/index.html. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-V oss, Gretchen...

    2023

  31. [2024]

    Mem- ory Augmented Language Models through Mixture of Word Experts

    Cicero dos Santos, James Lee-Thorp, Isaac Noble, Chung-Ching Chang, and David C Uthus. Mem- ory Augmented Language Models through Mixture of Word Experts. InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human...

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.