Pith. sign in

REVIEW 4 major objections 6 minor 32 references

Hecto: Modular Sparse Experts for Adaptive and Interpretable Reasoning

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Hecto claims that a sparse mixture of a GRU expert and a feedforward expert can match homogeneous MoE baselines on four NLP tasks while routing inputs in a way that is interpretable as a choice between temporal and static reasoning.

desk verdict Heterogeneous MoE with a confounded comparison: the GRU expert sees full sequences, the FFNN sees only a CLS vector, so the temporal-vs-static specialization claim is untested; still, the ablations and released artifacts make it a legitimate, if flawed, submission. read the letter →

arxiv 2506.22919 v2 pith:FVTG3KFK submitted 2025-06-28 cs.AI

Sanskar Pandey , Ruhaan Chopra , Saad Murtaza Bhat , Ark Abhyudaya This is my paper
classification cs.AI
keywords Mixture-of-ExpertsheterogeneousexpertssparseroutingTop-1gatinginterpretabilitytemporalreasoningGRUfeedforwardnetwork
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Hecto is a compact mixture-of-experts model that routes each input to exactly one of two deliberately different experts: a GRU for sequence-shaped, temporal reasoning and a feedforward network for static, pattern-style reasoning. The paper's claim is that this architectural heterogeneity delivers accuracy that matches or closely trails homogeneous baselines on AG News, SST-2, HotpotQA, and STS-B, while producing routing decisions that are interpretable at the level of reasoning type. A Top-1 gating network, regularized to be confident per input and balanced across experts, is what turns architecture into semantics: over training, the gate increasingly sends compositional and sequential inputs to the GRU and reserves the FFNN for a smaller set of static cases. If the claim holds, expert specialization need not be an emergent side effect of large scale; it can be designed into small models, giving low-resource deployments both efficiency and legibility.

What carries the argument

The load-bearing mechanism is the dual projection plus Top-1 gating scheme. The [CLS] vector is projected to a 256-dimensional space and feeds both the gating MLP and the FFNN expert, while the full token sequence is projected separately into the same 256-dimensional space and feeds the GRU expert. Because the two experts receive different inputs, the gate is forced to choose between a static summary and a sequential reading of the same text. A two-layer MLP with softmax temperature 1.5 produces expert probabilities, straight-through sampling keeps the discrete selection differentiable at train time, and argmax chooses the expert at inference. An entropy penalty pushes per-input confidence and a diversity penalty prevents batch-level collapse, and together they make the routing distribution the main interpretable output of the model.

What would settle it

Train a variant in which the FFNN expert also receives the full projected sequence, or the GRU receives only the [CLS] vector. If the gate's division of labor—GRU for temporal, FFNN for static—does not survive equalizing inputs, then the reported specialization is an artifact of input asymmetry rather than architectural inductive bias.

Watch

Extended reading notes

Core claim

The central discovery Hecto reports is that replacing a pool of identical experts with one GRU and one FFNN, each fed an isolated representation from a shared DistilBERT encoder, produces a sparse expert model that stays within about a point of homogeneous baselines while splitting computation into recognizable roles. On fine-tuned runs, Hecto reaches 90.02% on AG News versus 90.62% for GRU+GRU, 88.64% accuracy on SST-2 versus 89.11%, and 79.22% on HotpotQA versus 80.18% for FFNN+FFNN, with faster per-sample inference than the recurrent baseline on two of the three classification tasks. Routing is the evidence for specialization: the GRU receives 79.9% of AG News inputs, 63.3% of HotpotQA inputs, and 99.93% of SST-2 inputs, and a frozen-encoder variant pushes GRU usage to 92.5% while keeping 87.76% accuracy. Ablations show that removing entropy and diversity regularization collapses the gate onto one expert, doubling expert count collapses it to a single GRU, and swapping the GRU for a temporal convolutional network keeps performance, which the paper reads as evidence that heterogeneity itself, not any specific expert type, carries the benefit.

Load-bearing premise

The load-bearing premise is that feeding the GRU the full token sequence and the FFNN only the [CLS] vector is a fair comparison that isolates architecture; equalize the inputs and the observed routing split, and the temporal-versus-static interpretation with it, could vanish.

Editorial extensions

If this is right

  • At batch size 64 on AG News, Hecto reaches 90.82% accuracy and surpasses every homogeneous baseline trained at batch size 16, implying heterogeneous experts benefit more than uniform pools from relaxed optimization constraints.
  • With a frozen DistilBERT encoder, Hecto keeps 87.76% accuracy and routes 92.5% of inputs to the GRU, implying the gate and experts can still specialize even when the encoder cannot adapt.
  • Top-2 routing and four-expert variants both drive gate entropy to near zero and funnel nearly all inputs to one GRU, implying that sparse Top-1 routing with two heterogeneous experts is the configuration that keeps specialization legible.
  • Replacing the GRU with a temporal convolutional network yields 90.47% on AG News, implying the framework's benefit is architectural diversity rather than recurrence specifically.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The routing numbers are aggregate batch-level percentages, not per-sample labels of reasoning type; a stronger test of the temporal-versus-static story would annotate individual inputs and ask whether the gate's choice matches a human labeling of reasoning demand.
  • A testable extension would give the gate features beyond the [CLS] vector, such as input length, lexical cues, or predicted class, to see whether the class-wise routing patterns reported on AG News become sharper or instead wash out.
  • The HectoRegressor result of Pearson r = 0.84 on a 5k STS-B subset suggests the two-expert split transfers to scalar prediction, but the paper does not compare against homogeneous regressors; a direct comparison would show whether the interpretability benefit carries any performance cost in regression.
  • Because the paper only uses a frozen or fine-tuned DistilBERT backbone, the generalizability of the heterogeneity claim to larger encoders and generative tasks remains untested; the same routing setup applied to a larger backbone would be a straightforward check.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces Hecto, a sparse Top-1 mixture-of-experts architecture that combines a GRU expert and an FFNN expert, with the GRU processing the full token sequence and the FFNN processing only the projected [CLS] vector. The gating network is trained with cross-entropy plus hand-weighted entropy and diversity penalties. The authors claim that this heterogeneous design matches or closely trails homogeneous MoE baselines on AG News, SST-2, HotpotQA, and STS-B while achieving interpretable specialization aligned with temporal versus static reasoning. The empirical sections report accuracy, expert usage, latency, and a series of ablations on routing, regularization, expert count, gating inputs, frozen encoders, and batch size.

Significance. If the central claim were well supported, the paper would make a useful contribution: showing that expert architectures can be heterogeneous and produce interpretable routing without sacrificing performance, especially in low-resource settings. The paper also ships code and a model checkpoint, and the ablation suite is broad, covering frozen encoders, routing variants, expert counts, and an alternative TCN expert. These are genuine strengths. However, the significance is currently limited because the main interpretability claim rests on an experimental design that confounds architecture with input information content, and several performance claims rely on unfair comparisons.

major comments (4)
  1. [Section 3, Dual Projection, and Table 3] The GRU expert receives the full projected token sequence H′ while the FFNN expert receives only the projected [CLS] vector z. Because the two experts see fundamentally different amounts of information, the observed routing imbalance (79.9% GRU on AG News, 63.3% on HotpotQA, and 99.93% on SST-2) cannot be attributed to architectural inductive bias. The central claim that routing aligns with temporal-vs-static reasoning is therefore untested. To isolate the effect of architecture, the authors should either give both experts the full sequence or both experts the CLS vector, or explicitly include such an equalized-input condition in the experiments.
  2. [Section 5.3 and Appendix C] The claim that at batch size 64 Hecto 'surpasses all homogeneous baselines' and delivers 'state-of-the-art performance' is not supported by the reported evidence, because the comparison is against baselines trained with batch size 16 only. With different batch sizes, differences in performance can reflect optimization conditions rather than architectural merit. The authors should retrain the homogeneous baselines (FFNN+FFNN and GRU+GRU) at batch size 64 under the same protocol, or substantially qualify the claim as a comparison across different training settings.
  3. [Section B.2, Table 6, and Section B.4, Table 8] The routing behavior is highly unstable across small design changes: removing the entropy and diversity losses flips usage to 99.7% FFNN, and switching from hard to soft routing flips usage between approximately 99% FFNN and 99.7% GRU. This contradicts the notion of a stable, emergent 'temporal vs static' specialization. Moreover, the diversity penalty (λdiv) directly coerces balanced usage, so any balance observed in the main results is at least partly a consequence of the loss hand-tuning, not emergent specialization. The authors should present routing stability as a function of regularization strength and gate design, and discuss the interpretation accordingly.
  4. [Section 5.1, Table 3, and Section 7] On SST-2, 99.93% of inputs are routed to the GRU expert; this is effectively a single-expert model, not a specialized mixture with two active reasoning styles. The paper labels this 'clear expert specialization,' but a mixture with near-total collapse provides no evidence of adaptive, task-aligned reasoning per input. Similarly, the frozen-encoder ablation (Appendix A) gives 92.5% GRU usage, and the 4-expert variant (Appendix B.3) collapses to 100% GRU. These results should be interpreted as routing collapse under the current design, and the claim that heterogeneity yields specialized and interpretable routing should be reconsidered or substantially conditioned.
minor comments (6)
  1. [Abstract and Section 3] The phrase 'isolated input representations' obscures the fact that the two experts receive unequal information; the authors should describe this design choice explicitly as the GRU seeing the full sequence and the FFNN seeing only the CLS vector, and discuss its implications.
  2. [Section 4] The text refers to 'Figure 5.3' and 'Figure 3.2' for core results and routing behavior, but the actual figures are numbered differently; please fix these references.
  3. [Table 3 and related routing tables] The tables use 'E0/E1' without consistently defining which expert is FFNN and which is GRU in each configuration; add a note to the table caption to make the mapping unambiguous.
  4. [Appendix D, final paragraph] The sentence 'Further interpretability analyses for the original Hecto (FFNN+GRU) configuration are explored in Appendix D' appears to be a leftover reference error, since Appendix D is about FFNN+TCN; the intended pointer is probably Appendix E.
  5. [Section 5.2] The text says Figure 2, Figure 3, and Figure 4 depict expert selection patterns, but the captions only describe usage distributions; please clarify the figure content or unify the terminology.
  6. [Appendix B.1] In the Top-2 routing observation, the statement that 'the FFNN path fires on only 3.5% of inputs' is inconsistent with the table's usage ratio of 3.5:96.5 for FFNN:GRU; clarify whether 'fires' refers to being selected as one of the two active experts.

Circularity Check

1 steps flagged · score 4.0 of 10

The benchmark numbers are external and not circular, but the paper's central 'temporal vs static' specialization claim is partially self-fulfilling: the GRU is the only expert that receives full token sequences, so observed GRU-heavy routing is built into the input design rather than independently demonstrating reasoning-type alignment.

  1. self definitional [Section 3 (Architectural Design), Section 5.2 (Expert Usage Visualizations), and Appendix E.2]
    "By design, the FFNN expert operates exclusively on projected [CLS] vector, making it inductively suited for static, pattern-based reasoning, while the GRU expert processes full token sequences, equipping it to capture temporal and sequential dependencies. ... Notably, Hecto consistently routes a majority of the samples to Expert 1 (GRU expert), suggesting that the gating mechanism prioritizes temporal feature extraction and perceives sequential reasoning as more beneficial than shallow transformations across tasks."

    The 'temporal' label is assigned to the GRU by construction, since only the GRU receives the full projected token sequence H′; the FFNN receives only the projected CLS vector z. The observed routing preference for the GRU is therefore exactly what the input asymmetry predicts, and cannot serve as independent evidence that the gate selects a 'temporal' reasoning mode. No control equalizes the inputs (e.g., GRU on CLS only or FFNN on the full sequence), so the claimed alignment between expert type and reasoning type is a restatement of the design choice, not a measured discovery.

full rationale

This is an experimental paper with no formal derivation chain, so most circularity categories do not apply. The performance comparisons against FFNN+FFNN and GRU+GRU baselines are external benchmark numbers and are not circular. The central interpretability claim, however, is partly self-fulfilling: the GRU expert is defined as temporal because it sees the full token sequence, the FFNN as static because it sees only the CLS vector, and the observed GRU-heavy routing is then presented as confirmation of temporal-vs-static specialization. The paper's own Limitations section concedes that 'quantifying why a specific expert was chosen for a given input remains challenging' and that usage distributions only offer 'high-level trends,' which further weakens the strong specialization language. Because the performance numbers are external and the routing percentages are honestly reported, this is partial circularity in interpretation rather than a fully forced derivation; a moderate score is appropriate.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claims rest on hand-chosen regularization terms, an unstated data split, and an imposed temporal/static dichotomy; the experimental comparison is confounded by asymmetric expert inputs.

free parameters (4)
  • lambda_ent (entropy penalty weight) = 0.05
    Hand-chosen in Section 4.3; directly shapes gate confidence and thus the 'interpretable routing' observation.
  • lambda_div (diversity penalty weight) = 0.08
    Hand-chosen in Section 4.3; enforces balanced expert usage, so 'balanced specialization' is partly manufactured.
  • Softmax temperature tau = 1.5
    Hand-chosen in Section 4.3; affects gate confidence and entropy.
  • AG News training subset size = not stated in main text; '5k' in Appendix B.4
    The main experiments never state dataset size; Appendix B.4 says '5k AG News split'. This choice likely explains the low accuracy (~90%) and affects all main results.
assumptions (4)
  • domain assumption GRU is the right inductive bias for temporal reasoning and FFNN for static reasoning.
    Asserted in Section 1 and Section 3, never tested against tasks with known reasoning labels.
  • ad hoc to paper Giving the GRU the full token sequence and the FFNN only the CLS vector is a fair way to compare expert architectures.
    This asymmetry is presented without discussion in Section 3; it biases routing in favor of the GRU because the GRU sees more information.
  • domain assumption HotpotQA can be reduced to a classification task with accuracy/F1.
    The paper reports accuracy/F1 for HotpotQA but never describes how multi-hop QA is converted to classification (Section 4.2, Table 3).
  • ad hoc to paper Batch size 64 results for Hecto can be compared with batch size 16 results for baselines.
    Appendix C compares Hecto at batch 64 to all baselines at batch 16, implicitly assuming batch size is not a confound.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Hecto: Modular Sparse Experts for Adaptive and Interpretable Reasoning." pith.science (2026). https://pith.science/paper/FVTG3KFK

@misc{pith2026250622919,
  author       = {Pith},
  title        = {Pith review of: Hecto: Modular Sparse Experts for Adaptive and Interpretable Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FVTG3KFK}},
  note         = {Machine review of arXiv:2506.22919}
}
read the original abstract

Mixture-of-Experts (MoE) models enable conditional computation by routing inputs to specialized experts, but these experts rely on identical inductive biases, thus limiting representational diversity. This static computation pathway is inefficient for inputs that require different types of reasoning and limits specialization and interpretability. We propose Hecto, a lightweight MoE architecture that leverages architectural heterogeneity by combining a GRU expert for temporal reasoning and an FFNN expert for static abstraction under a sparse Top-1 gating mechanism. Evaluated on three reasoning benchmarks (AG News, SST-2, HotpotQA) and a regression task (STS-B), Hecto matches or closely trails homogeneous baselines in performance despite receiving isolated input representations, while achieving clear expert specialization, with each expert aligning to distinct reasoning types (temporal vs static). At larger batch sizes, Hecto exhibits improved performance, benefiting from relaxed computational constraints that allow its heterogeneous architecture to optimize more effectively. Ablation results isolate architectural diversity as the source of Hecto's stability and interpretability across diverse reasoning tasks. Overall, Hecto establishes itself as a new benchmark for conditional computation, offering a principled framework for specialized reasoning in low-resource regimes with its model strength derived from principled specialization.

Figures

Figures reproduced from arXiv: 2506.22919 by the authors.

Figure 1
Figure 1. Hecto architecture. The CLS projection feeds [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 5
Figure 5. , routing behavior is detailed in Figure 3.2, [PITH_FULL_IMAGE:figures/full_fig_p004_5.png] view at source ↗
Figure 3
Figure 3. Expert usage distributions (% of samples) for [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Expert usage distributions (% of samples) for [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 6
Figure 6. Figure 6: Expert–selection histogram under Top–2 rout [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Classwise routing trends across training epochs [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

32 extracted references · 17 canonical work pages

  1. [1]

    Le, and Geoffrey Hinton

    Noam Shazeer, Ardavan Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, and Geoffrey Hinton. Outrageously large neural networks: The mixture-of-experts layer. InInternational Confer- ence on Learning Representations (ICLR), 2017

    work page 2017

  2. [2]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.Journal of Machine Learning Research, 23(120):1–39, 2022

    William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.Journal of Machine Learning Research, 23(120):1–39, 2022

    2022

  3. [3]

    Gshard: Scaling giant models with conditional com- putation and automatic sharding

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional com- putation and automatic sharding. arXiv preprint arXiv:2006.16668, 2020

    arXiv 2006

  4. [4]

    Deepseekmoe: Towards efficient mixture-of-experts with generalist routing

    DeepSeek AI. Deepseekmoe: Towards efficient mixture-of-experts with generalist routing. arXiv preprint arXiv:2401.06066, 2024. 7

    arXiv 2024

  5. [5]

    Spatial Mixture-of-Experts

    Chen Li, Linlin Yang, Wei Zhang, Pengcheng Ren, Xuetao Wang, and Xiao Sun. Dynamoe: Dynamic mixture of experts with learnable routing for mul- titask learning. arXiv preprint arXiv:2211.13491, 2022

    work page Pith review arXiv 2022

  6. [6]

    Mixture-of-experts with expert choice routing.arXiv preprint arXiv:2202.09368, 2022

    Qiyuan Zhou, Yutian Chen, Mostafa Dehghani, An- dreas Steiner, Neil Houlsby, and Jakob Uszkoreit. Mixture-of-experts with expert choice routing.arXiv preprint arXiv:2202.09368, 2022

    arXiv 2022

  7. [7]

    Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference

    Raghav Ravindran, Bin Zhao, Linh Le, Mu Li, Amir Gholami, and Joseph E. Gonzalez. Sparse mixer: A sparse mixture-of-experts architecture. arXiv preprint arXiv:2308.12066, 2023

    work page Pith review arXiv 2023

  8. [8]

    Spectra: Specialized experts from prompt tuning

    Ryan Shin, Minjoon Park, Lianhui Qin, Yizhong Wang, Caiming Xiong, and Hannaneh Hajishirzi. Spectra: Specialized experts from prompt tuning. arXiv preprint arXiv:2504.19925, 2024

    arXiv 2024

Show all 32 references
  1. [9]

    Mofe: Modular fine-tuning of large language mod- els via mixture of frozen experts

    Chenguang Zhu, Yujia Xie, Michael Zeng, and et al. Mofe: Modular fine-tuning of large language mod- els via mixture of frozen experts. arXiv preprint arXiv:2503.06491, 2024

    2024 arXiv

  2. [10]

    Moduleformer: Modu- larization of pretrained transformers with self- supervised mixture-of-experts

    Wenxuan Zhou, Shiyue Zhang, Hong Wang, and William Yang Wang. Moduleformer: Modu- larization of pretrained transformers with self- supervised mixture-of-experts. arXiv preprint arXiv:2306.04640, 2023

    2023 arXiv

  3. [11]

    Automoe: Neural architecture search for efficient mixture of experts.arXiv preprint arXiv:2210.07535, 2022

    Zhiqiu Lin, Junda He, Haotian Zhang, and et al. Automoe: Neural architecture search for efficient mixture of experts.arXiv preprint arXiv:2210.07535, 2022

    2022 arXiv

  4. [12]

    Hetumoe: Towards training and serving heterogeneous moe models efficiently

    Jiayu Ye, Xudong Wang, Yunbo Wang, Zeyuan Jiang, and Ce Zhang. Hetumoe: Towards training and serving heterogeneous moe models efficiently. arXiv preprint arXiv:2203.14685, 2022

    2022 arXiv

  5. [13]

    Prompt- moe: Prompt-based expert routing in mixture-of- experts models

    Wei Zhang, Qing Liu, Jianfeng Wang, et al. Prompt- moe: Prompt-based expert routing in mixture-of- experts models. arXiv preprint arXiv:2501.05313, 2024

    2024 arXiv

  6. [14]

    Modular prompting via expert aggregation.arXiv preprint arXiv:2411.08982, 2024

    Jiacheng Guo, Qihang Li, Wei Wang, et al. Modular prompting via expert aggregation.arXiv preprint arXiv:2411.08982, 2024

    2024 arXiv

  7. [15]

    Moe-x: Mixture of ex- perts made intrinsically interpretable.arXiv preprint arXiv:2503.07639, 2024

    Kushal Chawla, Sajant Anand Mathew, Ximing Guo, and Nitish Srivastava. Moe-x: Mixture of ex- perts made intrinsically interpretable.arXiv preprint arXiv:2503.07639, 2024

    2024 arXiv

  8. [16]

    Deepspeed-moe: Advanc- ing mixture-of-experts inference and training to power next-generation ai scale

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Deepspeed-moe: Advanc- ing mixture-of-experts inference and training to power next-generation ai scale. arXiv preprint arXiv:2201.05596, 2022

    2022 arXiv

  9. [17]

    Expertrag: Augmenting large language models with specialized experts for retrieval

    Han Fang, Xilun Chen, Faisal Ladhak, and et al. Expertrag: Augmenting large language models with specialized experts for retrieval. arXiv preprint arXiv:2504.08744, 2024

    2024 arXiv

  10. [18]

    Ders: Decoding with expert routing and selec- tion for mixture-of-experts models.arXiv preprint arXiv:2503.01359, 2024

    Jingtao Zhan, Yining Wang, Chunting Zhou, and et al. Ders: Decoding with expert routing and selec- tion for mixture-of-experts models.arXiv preprint arXiv:2503.01359, 2024

    2024 arXiv

  11. [19]

    Manning, Andrew Ng, and Christopher Potts

    Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for se- mantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empiri- cal Methods in Natural Language Pr...

    2013

  12. [20]

    Gating is weighting: Understanding gated linear attention through in-context learning.arXiv preprint arXiv:2504.04308, 2024

    Zhuohan Teng, Junxian Lin, and Graham Neubig. Gating is weighting: Understanding gated linear attention through in-context learning.arXiv preprint arXiv:2504.04308, 2024

    2024 arXiv

  13. [21]

    Moesys: A distributed and efficient mixture-of- experts training and inference system for internet services

    Shuchang Zheng, Yiming Zhang, Qiang Wang, Yang Song, Renhong Xu, Qirui Tan, Shuai Wang, and Jie Liu. Moesys: A distributed and efficient mixture-of- experts training and inference system for internet services. arXiv preprint arXiv:2205.10034, 2022

    2022 arXiv

  14. [22]

    Molex: Modular and lightweight experts for efficient fine- tuning of llms

    Lu Chen, Tianle Cai, Wenkai Yang, and et al. Molex: Modular and lightweight experts for efficient fine- tuning of llms. arXiv preprint arXiv:2503.11144, 2024. A Frozen-Encoder Ablations We conducted an ablation study on the AG News bench- mark, keeping the Hecto framework unch...

    2024 arXiv

  15. [23]

    Two-expert routing adds a modest +0.58pp over Top–1, still below the GRU+GRU upper- bound (90.62%)

    Accuracy. Two-expert routing adds a modest +0.58pp over Top–1, still below the GRU+GRU upper- bound (90.62%)

  16. [24]

    Entropy plummets from 0.65 to 0.15 bits, indicating almost deterministic gating

    Confidence. Entropy plummets from 0.65 to 0.15 bits, indicating almost deterministic gating

  17. [25]

    always run the GRU

    Specialisation. The FFNN path fires on only 3.5% of inputs; Top–2 effectively degenerates into “always run the GRU.”

  18. [26]

    Because two experts fire per sample, single-expert attribution is lost - diluting the inter- pretability benefit of sparse MoEs

    Transparency. Because two experts fire per sample, single-expert attribution is lost - diluting the inter- pretability benefit of sparse MoEs

  19. [27]

    Executing both lightweight heads adds < 0.1 ms on a T4 GPU; compute cost is negligible

    Efficiency. Executing both lightweight heads adds < 0.1 ms on a T4 GPU; compute cost is negligible. Given these outcomes,Top–1remains the default routing policy in the main text, while Top–2 is retained here for completeness. B.2 Regularization Ablations We ablate the two loss...

  20. [28]

    Removing both losses scarcely changes accuracy (+0.13pp difference)

    Accuracy. Removing both losses scarcely changes accuracy (+0.13pp difference)

  21. [29]

    Entropy collapses from 0.40 bits to 0.02 bits—the gate becomes almost deterministic

    Gate behaviour. Entropy collapses from 0.40 bits to 0.02 bits—the gate becomes almost deterministic. 9 Table 6: Hecto with and without gate regularisation (fine- tuned encoder). Setting Acc. (%) H↓ Usage (% FF:GRU) λent=0.05, λdiv=0.08 90.60 0.401 57.1 : 42.9 λent=0, λdiv=0 90...

  22. [30]

    Without losses, the FFNN path receives 99.7% of inputs; the GRU is effectively bypassed

    Expert specialisation. Without losses, the FFNN path receives 99.7% of inputs; the GRU is effectively bypassed

  23. [31]

    Regularisation preserves a bal- anced and interpretable routing pattern, while the un-regularised variant hides all computation in one expert

    Interpretability. Regularisation preserves a bal- anced and interpretable routing pattern, while the un-regularised variant hides all computation in one expert. We therefore retain the entropy and load-balancing terms in all main-paper experiments to maintain meaningful expert...

  24. [32]

    across epochs for Fine-Tuned Hecto (averaged over 3 seeds). Epoch W orld (%) Sports (%) Business (%) Sci/T ech (%) 1 28.23 26.97 29.70 28.30 2 20.90 20.90 24.07 22.37 3 18.67 19.07 21.60 18.13 4 22.20 17.50 19.23 15.70 5 34.87 14.50 19.27 12.40 This behavior is a characteristi...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.