REVIEW 2 major objections 5 minor 12 references
The power of fine-grained experts: Granularity boosts expressivity in Mixture of Experts
T0 review · 2 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read Increasing the granularity of a mixture-of-experts layer — the number of experts active on each input — exponentially increases what the layer can represent, even when the number of active parameters is held fixed.
desk verdict The constant-activation separation for MoE granularity is solid and interesting, but the linear-activation theorem rests on a false covariance decomposition and needs a corrected proof before it can be trusted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the randomized routing construction (Lemma 3.2): Gaussian routing vectors in dimension $d \ge C k (\log m)^2$ partition a rotationally invariant input distribution into at least $(1/9)\binom{m}{k}$ regions $U_S$, each with probability at least $1/(2\binom{m}{k})$, proved by a first- and second-moment argument on the random vectors. On top of this partition, the paper places separated local functions: random expert vectors for the constant case, random low-rank matrices for the linear case, and random sparse matrices for the ReLU case, all engineered so that different active-expert sums are far apart. The final inapproximability argument is a linear program (or hypergraph matching, for ReLU) that shows any approximating MoE with too few regions must leave a constant fraction of the input mass badly approximated; a key supporting lemma states that high-probability subsets of a Gaussian or uniform-ball distribution have high-rank conditional covariance.
What would settle it
Compute, by Monte Carlo or exact integration, the second moment $\mathbb{E}[\mu(U_S)^2]$ for i.i.d. Gaussian routing vectors $r_1,\dots,r_m$ with $d = C k (\log m)^2$; Lemma A.7's proof requires it to be at most $3/\binom{m}{k}^2$ for standard Gaussian $\mu$. If the observed value exceeds that, the routing construction that all three separation theorems depend on is false. Alternatively, train an $(m',k')$-MoE with $\binom{m'}{k'}$ just below $c\binom{m}{k}^{0.99}$ against the hard teacher $f$ and check whether the $L^2$ error falls below the theorem's constant; the theorem predicts it cannot.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is the identification of the configuration count $\binom{m}{k}$ — the number of possible sets of $k$ active experts among $m$ — as the quantity that controls MoE expressivity. The Informal Theorem 1.1 states that if $m \ge C k$ and $\binom{m'}{k'} < c \binom{m}{k}^{0.99}$, then a carefully constructed $(m,k)$-MoE cannot be approximated by any $(m',k')$-MoE with the same number of active parameters, in the sense that the $L^2$ error is at least a constant times the energy of the target function. Because $\binom{m}{k}$ grows like $\Theta(m^k)$ when $k$ is fixed, this yields an exponential separation in granularity. The same style of separation is proven for constant, linear, and ReLU activations (Theorems 3.1, 3.4, and 3.6), with the technical condition $w \gtrsim \log m$ on expert width for the nontrivial activations.
Load-bearing premise
The separation relies on Lemma 3.2's claim that random Gaussian routing vectors in dimension $d \ge C k (\log m)^2$ split any rotationally invariant input distribution into at least one ninth of all $\binom{m}{k}$ possible active-expert regions, each carrying at least $1/(2\binom{m}{k})$ of the probability; if that randomized construction fails, the linear-programming lower bounds in Theorems 3.1, 3.4, and 3.6 collapse.
Editorial extensions
If this is right
- A coarse MoE cannot be rescued simply by scaling total parameters: with the same active parameter count, the fine-grained model expresses functions the coarse model misses by a constant relative $L^2$ error.
- For fixed granularity $k$, the separation grows exponentially in $k$, so the benefit of moving from 1 to 8 active experts, as in Llama-4 versus DeepSeek-V3, is much larger than a constant-factor gain.
- The heuristic bound for DeepSeek-V3's configuration count, $\binom{256}{8} \ge 4\times 10^{14}$, suggests a granularity-1 MoE would need on the order of $10^{14}$ experts to match it at equal active parameters.
- Experiments on random teacher-student MoEs show that a student must match the teacher's granularity to learn it; a lower-granularity student fails even with 16 times more total parameters.
- The proven advantage is for expressivity and ignores routing or communication costs, so the practical recommendation to increase granularity is conditional on hardware and routing schemes that keep those costs low.
Reading between the lines
- The configuration count $\binom{m}{k}$ looks like the right general currency for conditional computation: any architecture that reuses parameters across more input regions should inherit a similar exponential expressivity advantage, so analogous separations may hold for learned routing, softmax routing, or branchy layers.
- The theorem's 0.99 exponent is slack; a natural sharpening would replace it by the full condition $\binom{m'}{k'} \le c \binom{m}{k}$, or even a polynomial gap, and the paper's techniques seem pointed in that direction.
- The high-rank conditional-covariance lemma is a transferable tool: any low-dimensional function class approximating a target over many large-volume cells of a high-dimensional isotropic distribution should suffer the same separation, which could give lower bounds for other sparse or structured models.
- A direct empirical test outside the paper's setup: in real language-model pretraining, the loss gap between a granularity-$k$ and granularity-1 model at fixed active parameters should persist as total parameters grow, not close with scale.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies the expressivity of linearly-routed Mixture-of-Experts (MoE) layers as a function of granularity k (the number of active experts), comparing architectures with equal active parameter counts. It states three separation theorems for constant, linear, and ReLU activations: under dimension/width conditions and the combinatorial dominance of binom(m,k) over binom(m',k'), a randomly constructed (m,k)-MoE cannot be approximated in L2 by any (m',k')-MoE with substantially fewer expert configurations. The proof strategy combines a randomized routing construction (balanced top-k regions via a second-moment argument), random expert constructions satisfying boundedness and separation properties, and LP/hypergraph matching lower bounds that abstract away the adversary's routing. Section 4 reports experiments where student MoEs trained to fit teacher MoEs succeed only when their granularity matches or exceeds that of the teacher.
Significance. The central claim—that fine-grained experts provide an exponential expressivity advantage at equal active parameter count—is interesting, plausible, and directly relevant to current MoE architecture debates. The paper contributes a reusable technical framework: a routing lemma with a second-moment analysis, random packing constructions of expert functions, and a linear-programming/matching lower-bound technique that targets the number of routing regions rather than the details of the competitor's routing. The paper is also candid about its limitations, explicitly flagging the kw <= 0.99d regime and the non-explicit constants in the DeepSeek heuristic. If all three separation theorems are fully established, this would be a notable theoretical result. However, two load-bearing proof gaps, detailed in the major comments, mean that the linear and ReLU separations are not yet proven as written; the constant-activation separation and the routing construction appear sound. The experimental section is supportive but does not by itself compensate for the proof gaps.
major comments (2)
- [Appendix B.3, Lemma B.10 (Lemma 3.5)] The proof of Lemma B.10 asserts, after invoking Lemma B.5, that Sigma_U = (c/d)I + V with rank(V) <= kappa and ||V|| <= c/d. Lemma B.5 only lower-bounds the d-kappa+1 largest eigenvalues of Sigma_U and does not imply a rank-kappa decomposition of the deviation from (c/d)I. For a full-measure set U in the Gaussian case, Sigma_U = I/d, so the required V = (1-c/d)I has rank d, which contradicts rank(V) <= kappa for kappa < d. Consequently, the subsequent Von Neumann trace bound, which relies on rank(V) <= kappa, is not justified. Since Lemma 3.5 is the mechanism that gives expert separation in the linear case, Theorem 3.4 is not proven as written. The gap appears repairable by splitting V into its negative part (rank at most kappa, norm at most c/d, handled by Von Neumann) and its positive part (handled by positivity of the trace), but this argument must be supplied.
- [Appendix C.3, Claim C.10] The greedy construction in Claim C.10 does not account for previously used vertices, and its final arithmetic is incorrect. With R = 10^6, the displayed inequality 'TR >= 0.0001Rk = 1000k' uses 0.0001 x 10^6 = 100, not 1000, so the greedy argument only guarantees TR >= 100k, not the required 750k. Even after correcting R to 10^7, the step 'by the pigeonhole principle there is j_{s+1}' requires the number of unused vertices to exceed the instantaneous bad-set size, i.e. l - s > binom(750k,k)binom(m, floor(0.0001k)) for each s; this is not implied by the stated assumption l > binom(750k,k)binom(m, floor(0.0001k)). Since Lemma C.9's maximal fractional matching bound, and hence Theorem 3.6, depends on Claim C.10, the ReLU separation is not proven as written. The gap can likely be fixed by taking R to be a sufficiently large universal constant and adding R to the non-saturation threshold, which is dominated by binom(m,k) in the intended regime, but the correction should be made explicit.
minor comments (5)
- [Section 3, Theorems 3.1 and 3.4] The informal Theorem 1.1 and the abstract frame the result as a comparison at equal active parameter counts, but the formal statements of Theorems 3.1 and 3.4 do not impose k'w' = kw (only Theorem 3.6 does). Please clarify whether the formal results are intentionally stronger (they appear to hold for all (m',k',w',d')) or whether the equal-parameter condition is missing.
- [Appendix A.1, proof of Lemma A.5] The final probability bound contains the expression '1/m - Omega(sqrt(C))k + exp(-C'd)', which as written can be negative; this is presumably a typo for a sum of positive failure probabilities and should be corrected.
- [Section 4, Figure 2 caption] The caption says 'test loss of a teacher MoE trained to learn a student MoE', while the text says students are trained to learn teacher models; the roles should be made consistent.
- [Appendix C.2, Lemma C.6] The statement of (C.5) uses sigma_i^2((sum_{l in S_j} M_l) Pi^T) where Pi is a projection; the proof and Lemma C.3 require the complementary projection Pi^perp. Please correct the projection notation so that the statement matches the proof.
- [Appendix C, Theorem C.1] The assumptions list 'm >= Ck' and later 'm >= C'k' redundantly; since C and C' are used elsewhere as different universal constants, this duplication can confuse. Please consolidate the lower bound on m.
Circularity Check
No circularity: the separation theorems are derived from explicit randomized constructions and concentration inequalities, with no fitted parameters or self-citation chain.
full rationale
The central claims (Theorems 3.1, 3.4, 3.6) are proved by constructing a target (m,k)-MoE f using random Gaussian routing vectors (Lemma 3.2/A.7) and random expert matrices (Lemmas 3.3/A.8, B.9, C.6), then lower-bounding the L2 error against any (m',k')-MoE using a linear-programming/matching argument that counts routing regions. The quantity binom(m,k) enters as the number of top-k configurations of the constructed routing vectors, which is a structural fact about the architecture, not a fitted parameter renamed as a prediction. The expert constructions are probabilistic packing arguments with universal constants; no data are used to set parameters, and the heuristic DeepSeek extrapolation is explicitly flagged as heuristic in footnote 1. The experimental section is illustrative and is not used in the proofs. The paper contains no load-bearing self-citations: the cited related work on granularity scaling laws is motivation, not a premise of the theorems. A possible proof gap in Lemma 3.5/B.10 (the rank-k decomposition of the conditional covariance) would be a correctness issue for the linear-activation theorem, but it is not circularity: the lemma's conclusion is not identical to its assumptions, and the surrounding lower bound does not reduce to the statement being proved. Overall, the derivation chain is self-contained in the sense required here.
Assumptions & free parameters
assumptions (6)
- standard math Standard concentration and tail bounds for chi-squared, Gaussian, and binomial random variables (Laurent-Massart, Hanson-Wright, McDiarmid, Vershynin spectral norm).
- standard math For a rotationally invariant probability measure mu, the probability of a cone defined by random Gaussian routing vectors depends only on the uniform directional measure, so mu can be handled through Gaussian or ball representatives.
- domain assumption The MoE architecture under study, with linear top-k routing and equal weights and two-layer fully connected experts, is a faithful enough model that conclusions about expressivity transfer to practical MoE design.
- domain assumption Input distribution is standard Gaussian or uniform on the unit ball for linear and ReLU theorems; for the constant-activation theorem any rotationally invariant distribution.
- standard math Probabilistic method: if a construction succeeds with positive probability, the desired object exists.
- ad hoc to paper The regime kw <= 0.99d and d >= C k (log m)^2 is assumed; the paper notes this is beyond current techniques for kw = 2d.
Cite this review
Pith. "Pith review of The power of fine-grained experts: Granularity boosts expressivity in Mixture of Experts." pith.science (2026). https://pith.science/paper/LC42PP7X
@misc{pith2026250506839,
author = {Pith},
title = {Pith review of: The power of fine-grained experts: Granularity boosts expressivity in Mixture of Experts},
year = {2026},
howpublished = {\url{https://pith.science/paper/LC42PP7X}},
note = {Machine review of arXiv:2505.06839}
}
read the original abstract
Mixture-of-Experts (MoE) layers are increasingly central to frontier model architectures. By selectively activating parameters, they reduce computational cost while scaling total parameter count. This paper investigates the impact of the number of active experts, termed granularity, comparing architectures with many (e.g., 8 per layer in DeepSeek) to those with fewer (e.g., 1 per layer in Llama-4 models). We prove an exponential separation in network expressivity based on this design parameter, suggesting that models benefit from higher granularity. Experimental results corroborate our theoretical findings and illustrate this separation.
Figures
Reference graph
Works this paper leans on
-
[1]
[ASB+25] Samira Abnar, Harshay Shah, Dan Busbridge, Alaaeldin Mohamed Elnouby Ali, Josh Susskind, and Vimal Thilak. Parameters vs flops: Scaling laws for optimal sparsity for mixture-of-experts language models.arXiv preprint arXiv:2501.12370,
-
[3]
Learning factored representa- tions in a deep mixture of experts.arXiv preprint arXiv:1312.4314,
[ERS13] David Eigen, Marc’Aurelio Ranzato, and Ilya Sutskever. Learning factored representa- tions in a deep mixture of experts.arXiv preprint arXiv:1312.4314,
-
[4]
[He24] Xu Owen He. Mixture of a million experts. arXiv preprint arXiv:2407.04153,
-
[8]
Scaling laws for neural language models.arXiv preprint arXiv:2001.08361,
[KMH+20] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361,
arXiv 2001
-
[1937]
Toward inference- optimal mixture-of-expert large language models.arXiv preprint arXiv:2404.02852,
[YZF+24] Longfei Yun, Yonghao Zhuang, Yao Fu, Eric P Xing, and Hao Zhang. Toward inference- optimal mixture-of-expert large language models.arXiv preprint arXiv:2404.02852,
-
[1991]
Mixture of parrots: Experts improve memorization more than reasoning
[JMB+24] Samy Jelassi, Clara Mohri, David Brandfonbrener, Alex Gu, Nikhil Vyas, Nikhil Anand, David Alvarez-Melis, Yuanzhi Li, Sham M Kakade, and Eran Malach. Mixture of parrots: Experts improve memorization more than reasoning. arXiv preprint arXiv:2410.19034,
-
[2006]
Outrageously large neural networks: The sparsely-gated mixture-of-experts layer
[SMM+17] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538,
-
[2020]
Towards A Unified View of Sparse Feed-Forward Network in Pretraining Large Language Model
[LDL+23] Zeyu Leo Liu, Tim Dettmers, Xi Victoria Lin, Veselin Stoyanov, and Xian Li. Towards a unified view of sparse feed-forward network in pretraining large language model. arXiv preprint arXiv:2305.13999,
Show all 12 references
-
[2022]
Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models.arXiv preprint arXiv:2401.06066,
[DDZ+24] Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Yu Wu, et al. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models.arXiv preprint arXiv:2401.06066,
-
[2023]
Deepseek-v3 technical report
[LFX+24] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437,
-
[2024]
Mixtral of experts.arXiv preprint arXiv:2401.04088,
[JSR+24] Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts.arXiv preprint arXiv:2401.04088,
-
[2025]
Scaling laws for fine-grained mixture of experts.arXiv preprint arXiv:2402.07871,
36 [KLA+24] Jakub Krajewski, Jan Ludziejewski, Kamil Adamczewski, Maciej Pioro, Michal Kru- tul, Szymon Antoniak, Kamil Ciebiera, Krystian Krol, Tomasz Odrzygozdz, Piotr Sankowski, et al. Scaling laws for fine-grained mixture of experts.arXiv preprint arXiv:2402.07871,
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.