Pith. sign in

REVIEW 4 major objections 5 minor 68 references

On the Role of Discrete Representation in Sparse Mixture of Experts

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Replacing the learned MoE router with vector-quantized discrete codes improves training consistency, mitigates representation collapse, and cuts fine-tuning cost.

desk verdict A novel and plausible router-free MoE design with honest but small-scale empirical work, undermined by a theoretical section that does not hold up. read the letter →

arxiv 2411.19402 v2 pith:2654HVOD submitted 2024-11-28 cs.LG

Giang Do , Kha Pham , Hung Le , Truyen Tran This is my paper · ORCID
classification cs.LG MSC 68T07
keywords sparsemixtureofexpertsvectorquantizationdiscreterepresentationexpertroutingcollapselanguagemodelingvisiontransformers
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper seeks to establish that the learned router in sparse Mixture of Experts (SMoE) — a linear layer that scores tokens against expert embeddings — is not merely improvable but is itself the cause of two training failures: inconsistent expert selection and representation collapse. The proposed architecture, VQMoE, removes the router and instead selects experts through discrete codes learned by vector quantization, using a deterministic codebook-to-expert mapping. The authors prove that a standard router is inconsistent in the sense of Definition 4.1 when the attention output converges later than the expert embeddings, and that assigning each input cluster to its own expert is optimal. They also show that VQMoE's Jacobian spans a strictly larger subspace than SMoE's, which they argue mitigates collapse by design. Empirically, VQMoE matches or outperforms SMoE, StableMoE, XMoE, and SMoE-Dropout across language and vision pre-training and fine-tuning, and fine-tunes with 28% lower FLOPs.

What carries the argument

The mechanism that carries the argument is the vector-quantized codebook used as a router-free indexer. A vector quantizer assigns each input representation to one of K learned codebook vectors, and a modulo hash, i_exp = i_cb mod N, deterministically maps each codebook index to one of N expert networks, so no learned router scores are needed. During pre-training, VQMoE runs a standard continuous SMoE path in parallel with the discrete quantized path, combining them through a two-dimensional learned gate, and trains with the task loss plus the standard vector-quantization $\ell^2$ and commitment losses. During fine-tuning, only the discrete path is active, which is the source of the reported 28% reduction in fine-tuning FLOPs.

What would settle it

A direct test is to train VQMoE and a router-based SMoE on data whose MoE-layer inputs are deliberately non-clustered, for example tokens drawn uniformly from a high-dimensional sphere with no N distinct modes; if VQMoE still wins, the clustering-based optimality argument is not the operative mechanism. A complementary check is to measure on real checkpoints whether the expert embeddings do converge much earlier than the MHA output, since Theorem 4.3 hinges on t_m >> t_e.

Watch

Extended reading notes

Core claim

The central discovery, on the paper's own terms, is that discrete representation learning and expert selection are the same operation. Given a clustering of MoE inputs into N clusters, the optimal routing is to assign each cluster to one expert; VQMoE realizes this by quantizing each token to a codebook vector and hashing the codebook index to an expert index. Under the assumption that MHA outputs converge at step t_m much later than the expert embeddings converge at step t_e, the paper proves that the linear router of a conventional SMoE is inconsistent, because the converging embeddings force the router to select experts by distances that vanish. The Jacobian analysis adds a structural claim: the SMoE output projection spans a subspace of dimension N (the number of experts), while VQMoE's spans N+K+2, where K is the codebook size, so the discrete pathway widens the representational subspace and thereby avoids collapse. The empirical section then argues that this design is not just principled but practical, with consistent wins over learned-router baselines in language modeling, vision classification, and parameter-efficient fine-tuning.

Load-bearing premise

The load-bearing assumption is that MoE inputs are generated from exactly N well-separated clusters, one per expert, so that assigning each cluster to its own expert is guaranteed optimal; if real token representations are not that cleanly clustered, the consistency and optimality proofs lose their grip.

Editorial extensions

If this is right

  • If the router is truly unnecessary, future MoE designs can drop the learned router and its auxiliary balancing losses, replacing them with a codebook and a hash, which simplifies the architecture.
  • VQMoE's reported pre-training gains on enwik8, text8, WikiText-103, and lm1b imply that discrete routing scales with dataset size, since the gap over baselines widens from character-level to word-level benchmarks.
  • The fine-tuning result implies that a discrete expert-selection pathway can serve as a parameter-efficient adapter: freezing the codebook and updating only experts yields a 28% FLOP reduction while improving downstream accuracy.
  • The vision results suggest the discrete-routing benefit is not language-specific; it transfers to image classification and even to a segmentation task on ADE20K reported in the appendix.
  • The training-free 7B-model experiment implies that a pre-trained MoE's router can be replaced at inference time by codebook assignment, improving MTEB scores by 4.6% relative without any training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The consistency framework suggests a practical diagnostic: tracking the router consistency score during training could serve as an early-warning signal for collapse, letting practitioners switch to VQ assignment before the router degrades.
  • The 28% fine-tuning saving is architectural — the discrete path is cheaper than the combined path — so the same VQ module could be retrofitted onto already-trained router-based MoEs, which is exactly what the appendix's OLMoE experiment explores.
  • The ablation showing best performance when the codebook size equals the expert count implies the modulo hash is a bridge for engineering convenience; in the aligned case the mapping becomes the identity, so the method reduces to direct cluster-to-expert assignment.
  • One testable extension is to replace the fixed modulo hash with a learned or data-dependent permutation from codebooks to experts, which could relax the requirement that codebook size match expert number while preserving the theoretical guarantees.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes VQMoE, a sparse mixture-of-experts architecture that replaces the learned router with a vector-quantized discrete representation: inputs are quantized to codebook entries and then mapped to experts via a modulo-based hash, with a gated combination of a continuous SMoE path and a discrete expert path during pre-training and a discrete-only path during fine-tuning. The authors claim theoretical results (Theorem 4.3, Proposition 4.4, and a Jacobian analysis in Section 4.2) showing that conventional routers are inconsistent and that VQMoE intrinsically mitigates representation collapse. They report experiments on language modeling and vision datasets, claiming better or comparable performance to SMoE, StableMoE, XMoE, SMoE-Dropout, and a 28% reduction in fine-tuning FLOPs.

Significance. If the theoretical claims were correct, the paper would make a substantive contribution by showing that learned routing is unnecessary and that discrete representation learning offers a principled alternative for expert selection. The paper is commendable for proposing a concrete architecture and for assembling a broad set of empirical comparisons, including parameter-efficiency robustness plots, a small-scale multi-run variance check, and a training-free evaluation on the 7B OLMoE across MTEB tasks. However, the theoretical core is not established: the proofs in Appendix A.1 contain gaps and circular reasoning, and the Jacobian analysis in Section 4.2 rests on a count-of-terms fallacy. Because the abstract and Section 6 present VQMoE as 'theoretically-grounded' and claim that it overcomes representation collapse 'by design,' the unsupported theoretical results are load-bearing for the paper's primary contribution. The empirical gains over SMoE are thin in most comparisons, and several key claims (e.g., the 28% figure) are mis-stated in the abstract.

major comments (4)
  1. [Section 4.2, Eqs. (10)-(11), Appendix A.1.1] The Jacobian argument is based on a rank/count fallacy. For SMoE, Eq. (10) writes J_SMoE = S_k J_FFN + sum_{j=1}^N c_j e_j^T and then asserts that because the second term lies in an N-dimensional subspace, the output space is restricted from R^d to R^N. This is false: adding a low-rank matrix to S_k J_FFN does not restrict the column space of the sum unless the first term also lies in that subspace, which is not shown (and is not true in general since J_FFN usually has rank up to d). For VQMoE, Eq. (11) obtains J_VQMoE = J1 + sum_{j=1}^{N+K+2} o_j e_j^T and concludes that N+K+2 >> N 'implies VQMoE is better than SMoE in solving the representation collapse issue.' Counting summands does not control rank or subspace dimension: the K extra outer products could all be parallel to one another, contributing rank one, and even if independent they occupy a subspace of dimension at most min(d, N+K+2). No argument is given that the codebook-related Jacobians are diverse or orthogonal to the collapsed expert embeddings. The central claim that VQMoE mitigates representation collapse 'by design' is therefore not derived from the Jacobian analysis.
  2. [Appendix A.1.2, proof of Theorem 4.3] The proof of Theorem 4.3 is not justified. It considers three MHA outputs x1, x2, x3 and collapsed expert embeddings e1, e2, e3, then writes dist(x2, u2) ≤ min(dist(x1,e1), dist(x2,e2), dist(x3,e3)) ≤ dist(e1,e3). The first inequality is unexplained and the second is not generally true; most importantly, convergence of the expert embeddings (dist(e1,e3) → 0) does not imply convergence of the MHA output x2 to the expert embedding u2, so the claimed limit lim_{te→tm} dist(x2,u2)=0 does not follow. The proof then concludes that the MHA output y converges at step te, contradicting tm >> te, but this inference is exactly what the inequality chain was supposed to establish. In addition, the premise tm >> te is supported only qualitatively by the 2D convergence plots in Appendix A.3, with no quantitative threshold or statistical comparison. Thus Theorem 4.3 does not demonstrate that 'existing router methods are inconsistent.'
  3. [Appendix A.1.3, proof of Proposition 4.4] The proof of Proposition 4.4 is circular. It assumes that after tm-1 training steps the diagonal assignment satisfies L(Ej(xj)) = min_c L(Ec(xj)), and then, under the contrary assumption that some non-diagonal pair (xj, Ei) with i≠j has lower loss, it asserts that 'by definition of the loss minimization process' the inequality L(Ej(xj)) ≤ L(Ei(xj)) must hold. This is precisely the statement being proved, so the contradiction is manufactured rather than derived. Moreover, the proposition is stated with the index range i ∈ [1,k] although there are N clusters and N experts, and the proof never connects the VQ mechanism (Eqs. (3) and (6)) to the assumed cluster-to-expert assignment. The proposition therefore does not establish that VQMoE performs 'optimal expert selection.'
  4. [Section 4.1 and Theorem 4.3] The entire consistency framework rests on the assumption that the MoE input is generated from N distinct clusters with a one-to-one correspondence between clusters and experts. Definitions 4.1 and 4.2 are stated in terms of cluster centers u_i, but Theorem 4.3 is stated in terms of expert embeddings e_i; the paper does not show that these coincide or that actual MHA outputs satisfy the cluster assumption. The empirical plots in Appendix A.3 show trajectory divergences, not cluster quality, so there is no evidence that real transformer hidden states are cleanly clusterable. Since Proposition 4.4's optimality claim and the inconsistency claim both depend on this assumption, the theoretical framework has no demonstrated applicability to the models evaluated experimentally.
minor comments (5)
  1. [Equation (4)] Equation (4) contains LaTeX artifacts ('/bracehtipupleft', '/bracehtipdownright') that must be removed; the displayed formula as written is not readable.
  2. [Section 5.3] The fine-tuning description states that 'we freeze the router' following Chen et al. (2023a), but VQMoE has no router in the discrete-only fine-tuning path; please clarify which parameters (codebook, expert weights, gates) are frozen and which are updated.
  3. [Abstract and Section 6] The abstract says VQMoE 'achieves a 28% improvement in robustness,' while Section 5.3 and Section 6 describe a 28% reduction in computational resources in fine-tuning; these are different claims and should be worded consistently.
  4. [Equation (8)] In Equation (8), the L2 loss and commitment loss are written with subscript norms (e.g., \|sg[zv(x)]-v\|_2) rather than squared norms; the squared form is standard in VQVAE and appears implied but not written.
  5. [Appendix A.3 and Figure 4a] The 'consistency score' in Figure 4a and the 'coverage rate' mentioned in Section 5.5 are never formally defined; please provide the exact computation used to produce the score.

Circularity Check

1 steps flagged · score 6.0 of 10

Partial circularity: Prop. 4.4's optimal-expert-selection proof assumes its conclusion; the collapse-mitigation claim rests on an invalid Jacobian count rather than a self-referential reduction.

  1. self definitional [Section 4.1, Proposition 4.4; proof in Appendix A.1.3]
    "From step t0 to tm−1, each pair (xj,Ej), where j∈[1,N], is updated using the following gradient descent equation: W^{t_{l+1}}_{E_j} = W^{t_l}_{E_j}−ηJ(xj)... After tm−1 training steps, the following condition holds: L(Ej(xj)) = min_{c∈[1,N]} L(Ec(xj)). Under the assumption of contradiction, there exists a set of pairs, where xj is assigned to an expert Ei ... L(Ei(xj))≤L(Ej(xj)). However, by definition of the loss minimization process, the inequality L(Ej(xj))≤L(Ei(xj)) must hold."

    The proof begins by stipulating that each cluster input xj has been trained against its matching expert Ej, so that Ej is the loss-minimizing expert for xj. That is exactly the optimal assignment Proposition 4.4 claims to establish. The contradiction step only pits a loss-minimizing pairing against a non-minimizing one, which is tautological. It never shows that VQMoE's codebook-plus-hash assignment (Eqs. 3-6) realizes this loss-minimizing cluster-to-expert map. Thus the proposition's conclusion is equivalent to its premise, and the paper's inference that 'learning a discrete representation ... yields an optimal solution' inherits the circularity.

full rationale

The clearest circularity is Proposition 4.4: its proof in Appendix A.1.3 assumes the very pairing it sets out to establish. From step t0 to tm−1, each pair (xj,Ej) is updated by gradient descent, and after training L(Ej(xj)) = min_c L(Ec(xj)). The contradiction then shows only that a loss-minimizing assignment cannot be improved, which is tautological. No argument connects VQMoE's codebook/hash assignment to this loss-minimizing cluster-to-expert map, so the paper's claim that discrete representation gives an optimal expert selection reduces to its premise. The other theoretical pillar, Section 4.2, is not circular but is a non-sequitur. Eq. 11 is an algebraic identity for J_VQMoE, and the statement 'N + K + 2 >> N, implying that VQMoE is better than SMoE in solving the representation collapse issue' does not follow from counting rank-one summands; the earlier SMoE subspace argument also ignores the S_k J_FFN term. This is a validity flaw, not a self-referential reduction, so it is noted here but not counted as a circular step. Likewise, the proof of Theorem 4.3 in Appendix A.1.2 has an unsupported step: it assumes three expert embeddings collapse and treats convergence of embeddings as collapse; this is missing support rather than circularity. The empirical comparisons (Tables 1-3, 7, 8) are independent of these proofs and provide some standalone evidence for VQMoE's practical effectiveness. The paper also honestly acknowledges in Appendix A.2 Q6 that a discrete-only variant does not scale, which tempers but does not circularize the central claim. No load-bearing self-citation chain was found: Do et al. (2023, 2024) and Pham et al. (2024) appear only in related work, not as the basis of the proofs. Overall, the paper's headline theoretical claims—'intrinsically mitigates representation collapse' and 'optimal expert selection'—rest on the circular Proposition 4.4 and the invalid Jacobian inference, while the empirical content remains largely self-contained; hence partial circularity rather than full equivalence.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central theoretical claims rest on three unproven or partially supported premises: the cluster structure of MoE inputs, the convergence-time separation t_m >> t_e, and the Jacobian rank proxy for collapse. The two hand-tuned hyperparameters (alpha and codebook size K) are not fixed by the theory.

free parameters (2)
  • VQ loss weight alpha = not reported; recommended range (0.05, 0.15)
    Chosen by hand based on Figure 6c ablation; controls trade-off between task loss and discrete-representation losses.
  • Codebook size K = not reported per experiment; ablation says best when K equals number of experts
    Key architectural hyperparameter; main-table values are not given, and performance depends on it.
assumptions (3)
  • domain assumption MoE input data is generated from N distinct clusters with centers u_i
    Stated in Section 4.1 before Proposition 4.4; the optimality of one-to-one cluster-to-expert assignment rests on this.
  • domain assumption The MHA module converges at step t_m and expert embeddings converge at step t_e with t_m >> t_e
    Premise of Theorem 4.3; supported only by a qualitative convergence plot in Appendix A.3, not by a proof.
  • domain assumption Representation collapse is quantified by the rank of the Jacobian's outer-product summation term
    Section 4.2 uses the number of summands (N vs N+K+2) as a proxy for collapse; no theorem connects this count to expert representation diversity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On the Role of Discrete Representation in Sparse Mixture of Experts." pith.science (2026). https://pith.science/paper/2654HVOD

@misc{pith2026241119402,
  author       = {Pith},
  title        = {Pith review of: On the Role of Discrete Representation in Sparse Mixture of Experts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2654HVOD}},
  note         = {Machine review of arXiv:2411.19402}
}
read the original abstract

Sparse mixture of experts (SMoE) is an effective solution for scaling up model capacity without increasing the computational costs. A crucial component of SMoE is the router, responsible for directing the input to relevant experts; however, it also presents a major weakness, leading to routing inconsistencies and representation collapse issues. Instead of fixing the router like previous works, we propose an alternative that assigns experts to input via indirection, which employs the discrete representation of input that points to the expert. The discrete representations are learnt via vector quantization, resulting in a new architecture dubbed Vector-Quantized Mixture of Experts (VQMoE). We provide theoretical support and empirical evidence demonstrating the VQMoE's ability to overcome the challenges present in traditional routers. Through extensive evaluations on both large language models and vision tasks for pre-training and fine-tuning, we show that VQMoE achieves a 28% improvement in robustness compared to other SMoE routing methods, while maintaining strong performance in fine-tuning tasks.

Figures

Figures reproduced from arXiv: 2411.19402 by the authors.

Figure 1
Figure 1. Illustration of the proposed VQMoE architecture for Pre-training and fine-tuning. (a) At the [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Perplexity (PPL) over training steps for the Transformer-XL base model on two datasets: (a) [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Illustration of the proposed Robust VQMoE architecture for Pre-training on Enwik8 and Text8 [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Analysis Inconsistent Expert Selection and Representation Collapse issues when training SMoE. [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: Comparison of Token Representation and Expert Representation across Training Iteration. [PITH_FULL_IMAGE:figures/full_fig_p022_5.png]
Figure 6
Figure 6. Figure 6: Pre-training small Transformer-XL on WikiText-103 across different hyperparameters. [PITH_FULL_IMAGE:figures/full_fig_p022_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

68 extracted references · 29 canonical work pages

  1. [1]

    Efficient large scale language modeling with mixtures of experts, 2022

    Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victoria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasunuru, Giri Anantharaman, Xian Li, Shuohui Chen, Halil Akin, Mandeep Baines, Louis Martin, Xing Zhou, Punit Singh Koura, Brian O'Horo, Jeff Wang, Luke Zettlemoyer, Mona Diab, Zornitsa Kozareva, and Ves Stoyanov. Ef...

    work page 2022

  2. [2]

    Efficient intent detection with dual sentence encoders

    I \ n igo Casanueva, Tadas Tem c inas, Daniela Gerz, Matthew Henderson, and Ivan Vuli \'c . Efficient intent detection with dual sentence encoders. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pp.\ 38--45, Online, July 2020. Association for Computational Linguistics. doi:10.18653/v1/2020.nlp4convai-1.5. URL http...

    doi:10.18653/v1/2020.nlp4convai-1.5 2020

  3. [3]

    One billion word benchmark for measuring progress in statistical language modeling, 2014

    Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling, 2014. URL https://arxiv.org/abs/1312.3005

    arXiv 2014

  4. [4]

    Sparse moe as the new dropout: Scaling dense and self-slimmable transformers, 2023 a

    Tianlong Chen, Zhenyu Zhang, Ajay Jaiswal, Shiwei Liu, and Zhangyang Wang. Sparse moe as the new dropout: Scaling dense and self-slimmable transformers, 2023 a

    work page 2023

  5. [5]

    Learned-Miller, and Chuang Gan

    Zitian Chen, Yikang Shen, Mingyu Ding, Zhenfang Chen, Hengshuang Zhao, Erik G. Learned-Miller, and Chuang Gan. Mod-squad: Designing mixtures of experts as modular multi-task learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 11828--11837, June 2023 b

    work page 2023

  6. [6]

    Towards understanding the mixture-of-experts layer in deep learning

    Zixiang Chen, Yihe Deng, Yue Wu, Quanquan Gu, and Yuanzhi Li. Towards understanding the mixture-of-experts layer in deep learning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp.\ 23049--23062. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/pa...

    work page 2022

  7. [7]

    On the representation collapse of sparse mixture of experts, 2022

    Zewen Chi, Li Dong, Shaohan Huang, Damai Dai, Shuming Ma, Barun Patra, Saksham Singhal, Payal Bajaj, Xia Song, Xian-Ling Mao, Heyan Huang, and Furu Wei. On the representation collapse of sparse mixture of experts, 2022

    work page 2022

  8. [8]

    Generating long sequences with sparse transformers, 2019

    Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers, 2019. URL https://arxiv.org/abs/1904.10509

    arXiv 2019

Show all 68 references
  1. [9]

    An Analysis of Single Layer Networks in Unsupervised Feature Learning

    Adam Coates, Andrew Ng, and Honglak Lee. An Analysis of Single Layer Networks in Unsupervised Feature Learning . In AISTATS, 2011. https://cs.stanford.edu/ acoates/papers/coatesleeng_aistats_2011.pdf

    2011

  2. [10]

    Stablemoe: Stable routing strategy for mixture of experts, 2022

    Damai Dai, Li Dong, Shuming Ma, Bo Zheng, Zhifang Sui, Baobao Chang, and Furu Wei. Stablemoe: Stable routing strategy for mixture of experts, 2022

    2022

  3. [11]

    Transformer- XL : Attentive language models beyond a fixed-length context

    Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. Transformer- XL : Attentive language models beyond a fixed-length context. In Anna Korhonen, David Traum, and Llu \' s M \`a rquez (eds.), Proceedings of the 57th Annual Meeting of the As...

    2019 doi

  4. [12]

    Le, and Ruslan Salakhutdinov

    Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V. Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context, 2019 b

    2019

  5. [13]

    Le, and Ruslan Salakhutdinov

    Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V. Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context, 2019 c . URL https://arxiv.org/abs/1901.02860

    2019 arXiv

  6. [14]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp.\ 248--255. Ieee, 2009

    2009

  7. [15]

    On the benefits of learning to route in mixture-of-experts models

    Nishanth Dikkala, Nikhil Ghosh, Raghu Meka, Rina Panigrahy, Nikhil Vyas, and Xin Wang. On the benefits of learning to route in mixture-of-experts models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Lan...

    2023 doi

  8. [16]

    Nguyen, Chenghao Liu, Savitha Ramasamy, Xiaoli Li, and Steven Hoi

    Giang Do, Khiem Le, Quang Pham, TrungTin Nguyen, Thanh-Nam Doan, Bint T. Nguyen, Chenghao Liu, Savitha Ramasamy, Xiaoli Li, and Steven Hoi. Hyperrouter: Towards efficient training and inference of sparse mixture of experts, 2023

    2023

  9. [17]

    Simsmoe: Solving representational collapse via similarity measure, 2024

    Giang Do, Hung Le, and Truyen Tran. Simsmoe: Solving representational collapse via similarity measure, 2024. URL https://arxiv.org/abs/2406.15883

    2024 arXiv

  10. [18]

    An image is worth 16x16 words: Transformers for image recognition at scale, 2021

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at...

    2021 arXiv

  11. [19]

    Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen Meier-Hell...

    2022

  12. [20]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2022

    William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2022

    2022

  13. [21]

    Transformer feed-forward layers are key-value memories

    Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Pro...

    2021 doi

  14. [22]

    Kwok, and Yu Zhang

    Yunhao Gou, Zhili Liu, Kai Chen, Lanqing Hong, Hang Xu, Aoxue Li, Dit-Yan Yeung, James T. Kwok, and Yu Zhang. Mixture of cluster-conditional lora experts for vision-language instruction tuning, 2024. URL https://arxiv.org/abs/2312.12379

    2024 arXiv

  15. [23]

    David Ha, Andrew Dai, and Quoc V. Le. Hypernetworks, 2016

    2016

  16. [24]

    Deep residual learning for image recognition, 2015

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015. URL https://arxiv.org/abs/1512.03385

    2015 arXiv

  17. [25]

    Learning discrete representations via information maximizing self-augmented training

    Weihua Hu, Takeru Miyato, Seiya Tokui, Eiichi Matsumoto, and Masashi Sugiyama. Learning discrete representations via information maximizing self-augmented training. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, v...

    2017

  18. [26]

    Tutel: Adaptive mixture-of-experts at scale, 2023

    Changho Hwang, Wei Cui, Yifan Xiong, Ziyue Yang, Ze Liu, Han Hu, Zilong Wang, Rafael Salas, Jithin Jose, Prabhat Ram, Joe Chau, Peng Cheng, Fan Yang, Mao Yang, and Yongqiang Xiong. Tutel: Adaptive mixture-of-experts at scale, 2023

    2023

  19. [28]

    Jacobs, Michael I

    Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts. Neural Computation, 3 0 (1): 0 79--87, 1991 b . doi:10.1162/neco.1991.3.1.79

    1991 doi

  20. [29]

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne...

    2024

  21. [30]

    Hierarchical mixtures of experts and the

    Michael Jordan and Robert Jacobs. Hierarchical mixtures of experts and the. Neural computation, 6: 0 181--, 01 1994

    1994

  22. [31]

    Challenges and applications of large language models, 2023

    Jean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert McHardy. Challenges and applications of large language models, 2023

    2023

  23. [32]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017

    2017

  24. [33]

    Auto-encoding variational bayes, 2022

    Diederik P Kingma and Max Welling. Auto-encoding variational bayes, 2022. URL https://arxiv.org/abs/1312.6114

    2022 arXiv

  25. [34]

    Scaling laws for fine-grained mixture of experts, 2024

    Jakub Krajewski, Jan Ludziejewski, Kamil Adamczewski, Maciej Pióro, Michał Krutul, Szymon Antoniak, Kamil Ciebiera, Krystian Król, Tomasz Odrzygóźdź, Piotr Sankowski, Marek Cygan, and Sebastian Jaszczur. Scaling laws for fine-grained mixture of experts, 2024

    2024

  26. [35]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, UoT, 2009

    2009

  27. [36]

    Mole : Mixture of language experts for multi-lingual automatic speech recognition

    Yoohwan Kwon and Soo-Whan Chung. Mole : Mixture of language experts for multi-lingual automatic speech recognition. In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.\ 1--5, 2023. doi:10.1109/ICASSP49357.2023.10096227

    2023

  28. [37]

    Smith, and Luke Zettlemoyer

    Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A. Smith, and Luke Zettlemoyer. Branch-train-merge: Embarrassingly parallel training of expert language models, 2022

    2022

  29. [38]

    Moe-llava: Mixture of experts for large vision-language models, 2024

    Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Jinfa Huang, Junwu Zhang, Munan Ning, and Li Yuan. Moe-llava: Mixture of experts for large vision-language models, 2024

    2024

  30. [39]

    Sgdr: Stochastic gradient descent with warm restarts, 2017

    Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts, 2017. URL https://arxiv.org/abs/1608.03983

    2017 arXiv

  31. [40]

    Maas, Raymond E

    Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning Word Vectors for Sentiment Analysis . In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics : Human Language Technologies , pp.\ 142--1...

    2011

  32. [41]

    Large text compression benchmark, 2011

    Matt Mahoney. Large text compression benchmark, 2011. URL http://www.mattmahoney.net/dc/text.html

    2011

  33. [42]

    Discrete representations strengthen vision transformer robustness, 2022

    Chengzhi Mao, Lu Jiang, Mostafa Dehghani, Carl Vondrick, Rahul Sukthankar, and Irfan Essa. Discrete representations strengthen vision transformer robustness, 2022. URL https://arxiv.org/abs/2111.10493

    2022 arXiv

  34. [43]

    Finite scalar quantization: Vq-vae made simple, 2023

    Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantization: Vq-vae made simple, 2023. URL https://arxiv.org/abs/2309.15505

    2023 arXiv

  35. [44]

    Pointer sentinel mixture models, 2016

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016. URL https://arxiv.org/abs/1609.07843

    2016 arXiv

  36. [45]

    Mteb: Massive text embedding benchmark

    Niklas Muennighoff, Nouamane Tazi, Lo \" c Magne, and Nils Reimers. Mteb: Massive text embedding benchmark. arXiv preprint arXiv:2210.07316, 2022. doi:10.48550/ARXIV.2210.07316. URL https://arxiv.org/abs/2210.07316

    2022 doi arXiv

  37. [46]

    Smith, Pang Wei Koh, Amanpreet Singh, and Hannaneh Hajishirzi

    Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, Yuling Gu, Shane Arora, Akshita Bhagia, Dustin Schwenk, David Wadden, Alexander Wettig, Binyuan Hui, Tim Dettmers, Douwe Kiela, Ali F...

    2025 arXiv

  38. [47]

    Reading digits in natural images with unsupervised feature learning

    Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. NIPS Workshop, 2011

    2011

  39. [48]

    Nguyen, Savitha Ramasamy, Xiaoli Li, Steven Hoi, and Nhat Ho

    Quang Pham, Giang Do, Huy Nguyen, TrungTin Nguyen, Chenghao Liu, Mina Sartipi, Binh T. Nguyen, Savitha Ramasamy, Xiaoli Li, Steven Hoi, and Nhat Ho. Competesmoe -- effective training of sparse mixture of experts via competition, 2024

    2024

  40. [49]

    From sparse to soft mixtures of experts, 2024

    Joan Puigcerver, Carlos Riquelme, Basil Mustafa, and Neil Houlsby. From sparse to soft mixtures of experts, 2024. URL https://arxiv.org/abs/2308.00951

    2024 arXiv

  41. [50]

    Scaling vision with sparse mixture of experts, 2021 a

    Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts, 2021 a . URL https://arxiv.org/abs/2106.05974

    2021 arXiv

  42. [51]

    Scaling vision with sparse mixture of experts

    Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, Andr\' e Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Adva...

    2021

  43. [52]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer, 2017

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer, 2017

    2017

  44. [53]

    Mixture-of-experts meets instruction tuning:a winning combination for large language models, 2023 a

    Sheng Shen, Le Hou, Yanqi Zhou, Nan Du, Shayne Longpre, Jason Wei, Hyung Won Chung, Barret Zoph, William Fedus, Xinyun Chen, Tu Vu, Yuexin Wu, Wuyang Chen, Albert Webson, Yunxuan Li, Vincent Zhao, Hongkun Yu, Kurt Keutzer, Trevor Darrell, and Denny Zhou. Mixture-of-experts mee...

    2023

  45. [54]

    Scaling vision-language models with sparse mixture of experts

    Sheng Shen, Zhewei Yao, Chunyuan Li, Trevor Darrell, Kurt Keutzer, and Yuxiong He. Scaling vision-language models with sparse mixture of experts. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp.\ 11...

    2023 doi

  46. [55]

    Manning, Andrew Ng, and Christopher Potts

    Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank . In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Proce...

    2013

  47. [56]

    Segmenter: Transformer for semantic segmentation, 2021

    Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation, 2021. URL https://arxiv.org/abs/2105.05633

    2021 arXiv

  48. [57]

    Neural discrete representation learning

    Aaron van den Oord, Oriol Vinyals, and koray kavukcuoglu. Neural discrete representation learning. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associ...

    2017

  49. [58]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurip...

    2017

  50. [59]

    Language-routing mixture of experts for multilingual and code-switching speech recognition, 2023

    Wenxuan Wang, Guodong Ma, Yuke Li, and Binbin Du. Language-routing mixture of experts for multilingual and code-switching speech recognition, 2023

    2023

  51. [60]

    Openmoe: An early effort on open mixture-of-experts language models, 2024

    Fuzhao Xue, Zian Zheng, Yao Fu, Jinjie Ni, Zangwei Zheng, Wangchunshu Zhou, and Yang You. Openmoe: An early effort on open mixture-of-experts language models, 2024

    2024

  52. [61]

    Hifi-codec: Group-residual vector quantization for high fidelity audio codec, 2023

    Dongchao Yang, Songxiang Liu, Rongjie Huang, Jinchuan Tian, Chao Weng, and Yuexian Zou. Hifi-codec: Group-residual vector quantization for high fidelity audio codec, 2023. URL https://arxiv.org/abs/2305.02765

    2023 arXiv

  53. [62]

    Taskexpert: Dynamically assembling multi-task representations with memorial mixture-of-experts

    Hanrong Ye and Dan Xu. Taskexpert: Dynamically assembling multi-task representations with memorial mixture-of-experts. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp.\ 21828--21837, October 2023

    2023

  54. [63]

    Vector-quantized image modeling with improved vqgan, 2022

    Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vqgan, 2022. URL https://arxiv.org/abs/2110.04627

    2022 arXiv

  55. [64]

    Hauptmann, Ming-Hsuan Yang, Yuan Hao, Irfan Essa, and Lu Jiang

    Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G. Hauptmann, Ming-Hsuan Yang, Yuan Hao, Irfan Essa, and Lu Jiang. Magvit: Masked generative video transformer, 2023. URL https://arxiv.org/abs/2212.05199

    2023 arXiv

  56. [65]

    Semantic understanding of scenes through the ade20k dataset, 2018

    Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset, 2018. URL https://arxiv.org/abs/1608.05442

    2018 arXiv

  57. [66]

    Mixture-of-experts with expert choice routing

    Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, zhifeng Chen, Quoc V Le, and James Laudon. Mixture-of-experts with expert choice routing. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Inform...

    2022

  58. [67]

    Brainformers: Trading simplicity for efficiency, 2024

    Yanqi Zhou, Nan Du, Yanping Huang, Daiyi Peng, Chang Lan, Da Huang, Siamak Shakeri, David So, Andrew Dai, Yifeng Lu, Zhifeng Chen, Quoc Le, Claire Cui, James Laudon, and Jeff Dean. Brainformers: Trading simplicity for efficiency, 2024

    2024

  59. [68]

    St-moe: Designing stable and transferable sparse expert models, 2022

    Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. St-moe: Designing stable and transferable sparse expert models, 2022

    2022

  60. [69]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.