{"id":"d389fcdb-8eaa-4671-8f52-14baa9ffadc2","arxiv_id":"2411.19402","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"VQMoE is a routing-free SMoE variant using vector-quantized discrete codes for expert selection; its empirical gains are small and its theoretical proofs are circular or unsupported.","lead":"This paper replaces the learned router in sparse mixture-of-experts (SMoE) models with a vector-quantization codebook that assigns inputs to experts, calling the result VQMoE. The reported gains are modest in pre-training perplexity and accuracy, while fine-tuning uses about 28% less compute, although the abstract mislabels this as a robustness improvement.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The core theoretical argument for VQMoE's collapse mitigation is not established: Eq. 11 infers reduced collapse from a larger number of Jacobian summands, which is a rank/count fallacy; the central claim therefore lacks support.","rationale":"The reader's verdict of REJECT is supported. I focus on a different load-bearing weakness than the one named as the weakest assumption: even if the cluster assumption and the convergence-time assumption in Theorem 4.3 are granted, the Jacobian argument in Section 4.2 is not a valid derivation. The paper itself flags the importance of this argument in Section 6, where it says VQMoE 'intrinsically mitigates' collapse and is 'theoretically-grounded.' The flaw is concrete: a sum of rank-one terms is not made less collapsible by adding more terms, and the rank of the non-J1 part does not control the rank of the full Jacobian. This concern is testable with the numerical rank checks described above. The reader's weakest_assumption focused on the cluster property and the convergence times; those are also legitimate concerns, and the reader's rationale already notes that the Jacobian 'counts summands' without derivation. I therefore mark agreement as partial: we agree the paper should be rejected and that the theory is unsound, but the single most load-bearing failure is the collapse-mitigation argument, not the cluster assumption. No further verdict adjustment is needed.","tokens_in":20190,"tokens_out":6261,"duration_ms":56071,"concrete_test":"Independently re-derive Section 4.2 from the actual VQMoE definition and test the rank claim numerically on a trained checkpoint. Concretely: (1) take a trained small VQMoE (e.g., the Transformer-XL base from Table 1), compute the Jacobians J_VQMoE and J_SMoE with respect to a batch of inputs; (2) compute the effective rank (participation ratio) of the additive terms Q_S = sum_{j=1}^N c_j e_j^T and Q_V = sum_{j=1}^{N+K+2} o_j e_j^T after removing J1. If the effective rank of Q_V is not substantially larger than that of Q_S, or if both are low, the inference 'N+K+2 >> N reduces collapse' fails empirically. (3) As a control, construct synthetic outer products with K much larger than N but all equal to one vector; this shows the count alone is vacuous.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that VQMoE 'intrinsically mitigates' representation collapse—rests on the Jacobian comparison in Section 4.2 (Eqs. 10–11 and Appendix A.1.1). For SMoE, Eq. 10 writes J_SMoE = S_k J_FFN + sum_{j=1}^N c_j e_j^T and then asserts that because the second term lies in a subspace of dimension N, the output space is restricted from R^d to R^N. This ignores the first term S_k J_FFN, whose columns generally span R^d; a low-rank additive term does not restrict the full Jacobian's column space unless the FFN Jacobian also lies in span{e_j}. For VQMoE, Eq. 11 obtains J_VQMoE = J1 + sum_{j=1}^{N+K+2} o_j e_j^T and concludes that N+K+2 >> N 'implies VQMoE is better ... in solving representation collapse.' Counting summands does not bound or characterize the subspace: K extra outer products can all be parallel (rank contribution 1), and even if they are linearly independent, the column space is at most min(d, N+K+2). No argument is given that the discrete experts' Jacobians are diverse or orthogonal to the continuous experts' embeddings. The claimed mechanism—that adding K codebook-related terms dilutes collapse—is therefore not derived. Since the abstract and Section 6 state that VQMoE is 'theoretically-grounded' and overcomes representation collapse 'by design,' this invalid inference is load-bearing: without it, the paper offers only empirical comparisons, which are small-scale, partly single-run, and with thin margins.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes VQMoE, a sparse mixture-of-experts architecture that replaces the learned router with a vector-quantized discrete representation: inputs are quantized to codebook entries and then mapped to experts via a modulo-based hash, with a gated combination of a continuous SMoE path and a discrete expert path during pre-training and a discrete-only path during fine-tuning. The authors claim theoretical results (Theorem 4.3, Proposition 4.4, and a Jacobian analysis in Section 4.2) showing that conventional routers are inconsistent and that VQMoE intrinsically mitigates representation collapse. They report experiments on language modeling and vision datasets, claiming better or comparable performance to SMoE, StableMoE, XMoE, SMoE-Dropout, and a 28% reduction in fine-tuning FLOPs.","tokens_in":20433,"tokens_out":5972,"duration_ms":51928,"significance":"If the theoretical claims were correct, the paper would make a substantive contribution by showing that learned routing is unnecessary and that discrete representation learning offers a principled alternative for expert selection. The paper is commendable for proposing a concrete architecture and for assembling a broad set of empirical comparisons, including parameter-efficiency robustness plots, a small-scale multi-run variance check, and a training-free evaluation on the 7B OLMoE across MTEB tasks. However, the theoretical core is not established: the proofs in Appendix A.1 contain gaps and circular reasoning, and the Jacobian analysis in Section 4.2 rests on a count-of-terms fallacy. Because the abstract and Section 6 present VQMoE as 'theoretically-grounded' and claim that it overcomes representation collapse 'by design,' the unsupported theoretical results are load-bearing for the paper's primary contribution. The empirical gains over SMoE are thin in most comparisons, and several key claims (e.g., the 28% figure) are mis-stated in the abstract.","major_comments":[{"comment":"The Jacobian argument is based on a rank/count fallacy. For SMoE, Eq. (10) writes J_SMoE = S_k J_FFN + sum_{j=1}^N c_j e_j^T and then asserts that because the second term lies in an N-dimensional subspace, the output space is restricted from R^d to R^N. This is false: adding a low-rank matrix to S_k J_FFN does not restrict the column space of the sum unless the first term also lies in that subspace, which is not shown (and is not true in general since J_FFN usually has rank up to d). For VQMoE, Eq. (11) obtains J_VQMoE = J1 + sum_{j=1}^{N+K+2} o_j e_j^T and concludes that N+K+2 >> N 'implies VQMoE is better than SMoE in solving the representation collapse issue.' Counting summands does not control rank or subspace dimension: the K extra outer products could all be parallel to one another, contributing rank one, and even if independent they occupy a subspace of dimension at most min(d, N+K+2). No argument is given that the codebook-related Jacobians are diverse or orthogonal to the collapsed expert embeddings. The central claim that VQMoE mitigates representation collapse 'by design' is therefore not derived from the Jacobian analysis.","section":"Section 4.2, Eqs. (10)-(11), Appendix A.1.1"},{"comment":"The proof of Theorem 4.3 is not justified. It considers three MHA outputs x1, x2, x3 and collapsed expert embeddings e1, e2, e3, then writes dist(x2, u2) ≤ min(dist(x1,e1), dist(x2,e2), dist(x3,e3)) ≤ dist(e1,e3). The first inequality is unexplained and the second is not generally true; most importantly, convergence of the expert embeddings (dist(e1,e3) → 0) does not imply convergence of the MHA output x2 to the expert embedding u2, so the claimed limit lim_{te→tm} dist(x2,u2)=0 does not follow. The proof then concludes that the MHA output y converges at step te, contradicting tm >> te, but this inference is exactly what the inequality chain was supposed to establish. In addition, the premise tm >> te is supported only qualitatively by the 2D convergence plots in Appendix A.3, with no quantitative threshold or statistical comparison. Thus Theorem 4.3 does not demonstrate that 'existing router methods are inconsistent.'","section":"Appendix A.1.2, proof of Theorem 4.3"},{"comment":"The proof of Proposition 4.4 is circular. It assumes that after tm-1 training steps the diagonal assignment satisfies L(Ej(xj)) = min_c L(Ec(xj)), and then, under the contrary assumption that some non-diagonal pair (xj, Ei) with i≠j has lower loss, it asserts that 'by definition of the loss minimization process' the inequality L(Ej(xj)) ≤ L(Ei(xj)) must hold. This is precisely the statement being proved, so the contradiction is manufactured rather than derived. Moreover, the proposition is stated with the index range i ∈ [1,k] although there are N clusters and N experts, and the proof never connects the VQ mechanism (Eqs. (3) and (6)) to the assumed cluster-to-expert assignment. The proposition therefore does not establish that VQMoE performs 'optimal expert selection.'","section":"Appendix A.1.3, proof of Proposition 4.4"},{"comment":"The entire consistency framework rests on the assumption that the MoE input is generated from N distinct clusters with a one-to-one correspondence between clusters and experts. Definitions 4.1 and 4.2 are stated in terms of cluster centers u_i, but Theorem 4.3 is stated in terms of expert embeddings e_i; the paper does not show that these coincide or that actual MHA outputs satisfy the cluster assumption. The empirical plots in Appendix A.3 show trajectory divergences, not cluster quality, so there is no evidence that real transformer hidden states are cleanly clusterable. Since Proposition 4.4's optimality claim and the inconsistency claim both depend on this assumption, the theoretical framework has no demonstrated applicability to the models evaluated experimentally.","section":"Section 4.1 and Theorem 4.3"}],"minor_comments":[{"comment":"Equation (4) contains LaTeX artifacts ('/bracehtipupleft', '/bracehtipdownright') that must be removed; the displayed formula as written is not readable.","section":"Equation (4)"},{"comment":"The fine-tuning description states that 'we freeze the router' following Chen et al. (2023a), but VQMoE has no router in the discrete-only fine-tuning path; please clarify which parameters (codebook, expert weights, gates) are frozen and which are updated.","section":"Section 5.3"},{"comment":"The abstract says VQMoE 'achieves a 28% improvement in robustness,' while Section 5.3 and Section 6 describe a 28% reduction in computational resources in fine-tuning; these are different claims and should be worded consistently.","section":"Abstract and Section 6"},{"comment":"In Equation (8), the L2 loss and commitment loss are written with subscript norms (e.g., \\|sg[zv(x)]-v\\|_2) rather than squared norms; the squared form is standard in VQVAE and appears implied but not written.","section":"Equation (8)"},{"comment":"The 'consistency score' in Figure 4a and the 'coverage rate' mentioned in Section 5.5 are never formally defined; please provide the exact computation used to produce the score.","section":"Appendix A.3 and Figure 4a"}],"recommendation":"reject","confidential_remarks":"The manuscript's core theoretical contributions are not sound, and the flaws are not localized: the proof of Theorem 4.3 relies on an unjustified chain of inequalities, the proof of Proposition 4.4 is circular, and the Jacobian analysis equates the number of terms in a decomposition with the dimension of a subspace. Removing these unsupported theoretical claims would change the paper's scope and contribution, so a major revision is not a straightforward fix. The empirical results, while broad, are on small models with thin margins over SMoE, single-run results for ImageNet, and an abstract that mis-states the 28% figure as robustness rather than FLOPs reduction. I see no evidence of misconduct; the issues are technical and presentational."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: VQMoE is a real design idea, and the empirical work is honest and reasonably broad, but the theory that supposedly grounds it does not hold up. If the authors stripped out the theorems and presented this as an empirical study of a VQ-based router-free MoE, it would be a solid workshop paper or the basis for a major revision.\n\nWhat is actually new: the VQ codebook with a modulo mapping to experts, the dual continuous/discrete path during pretraining, and the discrete-only fine-tuning. I don't see that combination in the cited literature. The ablations on VQ variants, codebook size, and alpha are useful, and they honestly report that discrete-only training fails at scale (Appendix A.2, Q6). That kind of negative result is worth publishing. The evaluation spans language, vision, fine-tuning, and even a zero-shot MTEB comparison, though the OLMoE comparison uses the pretrained router for codebook embedding rather than actually training VQMoE, so treat it as suggestive.\n\nThe soft spots are the load-bearing theory. Theorem 4.3's proof in A.1.2 is a chain of unjustified inequalities: it jumps from distances between expert embeddings to convergence of the MHA output without any bridging argument, and the te << tm premise is supported only by a 2D plot. Proposition 4.4's proof in A.1.3 is circular: it assumes the loss-minimal assignment and then uses minimality to derive a contradiction. And the Jacobian argument in Section 4.2 is the worst: the claim that the SMoE output space is restricted to R^N because one additive term lies in an N-dimensional subspace ignores the first term S_k J_FFN, which generally spans R^d. Counting N+K+2 summands in Eq. 11 says nothing about the rank or diversity of those terms. The stress-test note is right: this is a rank/count fallacy. The abstract's '28% improvement in robustness' is also a misstatement: the body reports 28% FLOPs savings in fine-tuning, and 'robustness' means performance at 80% parameters. That mismatch matters because the abstract is what most readers will see.\n\nIf I were handling this, I would send it to review, because the architecture is genuinely new and the empirical story is not dismissible. But I would expect the theory claims to be removed or replaced with a real analysis, code to be released, and the OLMoE comparison to be redone. As it stands, the central claim of theoretical grounding is not true.","headline":"A novel and plausible router-free MoE design with honest but small-scale empirical work, undermined by a theoretical section that does not hold up.","tokens_in":21100,"tokens_out":3611,"would_cite":false,"duration_ms":31259,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"Replacing the learned MoE router with vector-quantized discrete codes improves training consistency, mitigates representation collapse, and cuts fine-tuning cost.","keywords":["sparse mixture of experts","vector quantization","discrete representation","expert routing","representation collapse","language modeling","vision transformers"],"falsifier":"A direct test is to train VQMoE and a router-based SMoE on data whose MoE-layer inputs are deliberately non-clustered, for example tokens drawn uniformly from a high-dimensional sphere with no N distinct modes; if VQMoE still wins, the clustering-based optimality argument is not the operative mechanism. A complementary check is to measure on real checkpoints whether the expert embeddings do converge much earlier than the MHA output, since Theorem 4.3 hinges on t_m >> t_e.","tokens_in":19867,"feed_emoji":"🧠","tokens_out":8145,"duration_ms":63993,"temperature":0.7,"pith_summary":"This paper seeks to establish that the learned router in sparse Mixture of Experts (SMoE) — a linear layer that scores tokens against expert embeddings — is not merely improvable but is itself the cause of two training failures: inconsistent expert selection and representation collapse. The proposed architecture, VQMoE, removes the router and instead selects experts through discrete codes learned by vector quantization, using a deterministic codebook-to-expert mapping. The authors prove that a standard router is inconsistent in the sense of Definition 4.1 when the attention output converges later than the expert embeddings, and that assigning each input cluster to its own expert is optimal. They also show that VQMoE's Jacobian spans a strictly larger subspace than SMoE's, which they argue mitigates collapse by design. Empirically, VQMoE matches or outperforms SMoE, StableMoE, XMoE, and SMoE-Dropout across language and vision pre-training and fine-tuning, and fine-tunes with 28% lower FLOPs.","feed_headline":"Discrete codes beat learned routing in sparse Mixture of Experts","feed_subtitle":"VQMoE replaces the fragile learned router with vector-quantized codes, improving robustness and cutting fine-tuning cost by 28%.","key_machinery":"The mechanism that carries the argument is the vector-quantized codebook used as a router-free indexer. A vector quantizer assigns each input representation to one of K learned codebook vectors, and a modulo hash, i_exp = i_cb mod N, deterministically maps each codebook index to one of N expert networks, so no learned router scores are needed. During pre-training, VQMoE runs a standard continuous SMoE path in parallel with the discrete quantized path, combining them through a two-dimensional learned gate, and trains with the task loss plus the standard vector-quantization $\\ell^2$ and commitment losses. During fine-tuning, only the discrete path is active, which is the source of the reported 28% reduction in fine-tuning FLOPs.","core_discovery":"The central discovery, on the paper's own terms, is that discrete representation learning and expert selection are the same operation. Given a clustering of MoE inputs into N clusters, the optimal routing is to assign each cluster to one expert; VQMoE realizes this by quantizing each token to a codebook vector and hashing the codebook index to an expert index. Under the assumption that MHA outputs converge at step t_m much later than the expert embeddings converge at step t_e, the paper proves that the linear router of a conventional SMoE is inconsistent, because the converging embeddings force the router to select experts by distances that vanish. The Jacobian analysis adds a structural claim: the SMoE output projection spans a subspace of dimension N (the number of experts), while VQMoE's spans N+K+2, where K is the codebook size, so the discrete pathway widens the representational subspace and thereby avoids collapse. The empirical section then argues that this design is not just principled but practical, with consistent wins over learned-router baselines in language modeling, vision classification, and parameter-efficient fine-tuning.","pith_inferences":["The consistency framework suggests a practical diagnostic: tracking the router consistency score during training could serve as an early-warning signal for collapse, letting practitioners switch to VQ assignment before the router degrades.","The 28% fine-tuning saving is architectural — the discrete path is cheaper than the combined path — so the same VQ module could be retrofitted onto already-trained router-based MoEs, which is exactly what the appendix's OLMoE experiment explores.","The ablation showing best performance when the codebook size equals the expert count implies the modulo hash is a bridge for engineering convenience; in the aligned case the mapping becomes the identity, so the method reduces to direct cluster-to-expert assignment.","One testable extension is to replace the fixed modulo hash with a learned or data-dependent permutation from codebooks to experts, which could relax the requirement that codebook size match expert number while preserving the theoretical guarantees."],"forward_implications":["If the router is truly unnecessary, future MoE designs can drop the learned router and its auxiliary balancing losses, replacing them with a codebook and a hash, which simplifies the architecture.","VQMoE's reported pre-training gains on enwik8, text8, WikiText-103, and lm1b imply that discrete routing scales with dataset size, since the gap over baselines widens from character-level to word-level benchmarks.","The fine-tuning result implies that a discrete expert-selection pathway can serve as a parameter-efficient adapter: freezing the codebook and updating only experts yields a 28% FLOP reduction while improving downstream accuracy.","The vision results suggest the discrete-routing benefit is not language-specific; it transfers to image classification and even to a segmentation task on ADE20K reported in the appendix.","The training-free 7B-model experiment implies that a pre-trained MoE's router can be replaced at inference time by codebook assignment, improving MTEB scores by 4.6% relative without any training."],"supporting_citations":[{"why":"Supplies vector quantization and the l2 and commitment losses that VQMoE's training objective extends.","marker":"(van den Oord et al., 2017)"},{"why":"Provides the SMoE-Dropout baseline, the experimental configuration, and the freeze-router fine-tuning protocol used for comparison.","marker":"(Chen et al., 2023a)"},{"why":"Defines representation collapse in SMoE, supplies the XMoE baseline, and gives the Jacobian viewpoint the paper's Section 4.2 builds on.","marker":"(Chi et al., 2022)"},{"why":"Gives the StableMoE baseline whose two-phase stable routing is compared against VQMoE.","marker":"(Dai et al., 2022)"},{"why":"Provides the clustering perspective on routing that the consistency definitions and Proposition 4.4 are modeled on.","marker":"(Dikkala et al., 2023)"},{"why":"Supplies the plain SMoE baseline architecture whose router VQMoE replaces.","marker":"(Jiang et al., 2024)"},{"why":"Supports the design choice to keep only the feed-forward expert pathway during fine-tuning.","marker":"(Geva et al., 2021)"}],"fun_headline_variants":["Discrete codes beat learned routers in sparse MoE","Vector quantization replaces MoE router for robustness","VQMoE: discrete codes select experts without a router","Indirection via discrete codes stabilizes MoE routing","Vector-quantized MoE improves robustness and fine-tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that MoE inputs are generated from exactly N well-separated clusters, one per expert, so that assigning each cluster to its own expert is guaranteed optimal; if real token representations are not that cleanly clustered, the consistency and optimality proofs lose their grip.","fun_headline_variants_meta":{"raw":{"variants":["Discrete codes beat learned routers in sparse MoE","Vector quantization replaces MoE router for robustness","VQMoE: discrete codes select experts without a router","Indirection via discrete codes stabilizes MoE routing","Vector-quantized MoE improves robustness and fine-tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000684,"raw_usage":{"total_tokens":3106,"prompt_tokens":947,"completion_tokens":2159,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":563,"completion_tokens_details":{"reasoning_tokens":2082}},"tokens_in":563,"tokens_out":2159,"duration_ms":13654,"temperature":1.0,"reasoning_tokens":2082,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:12:41.993574+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test is to train VQMoE and a router-based SMoE on data whose MoE-layer inputs are deliberately non-clustered, for example tokens drawn uniformly from a high-dimensional sphere with no N distinct modes; if VQMoE still wins, the clustering-based optimality argument is not the operative mechanism. A complementary check is to measure on real checkpoints whether the expert embeddings do converge much earlier than the MHA output, since Theorem 4.3 hinges on t_m >> t_e.","supporting_citations":[{"cited_title":"Neural discrete representation learning","cited_arxiv_id":null,"evidence_quote":"Supplies vector quantization and the l2 and commitment losses that VQMoE's training objective extends."}],"review_version":1}