THOR-MoE: Hierarchical Task-Guided and Context-Responsive Routing for Neural Machine Translation

Fandong Meng; Jie Zhou; Yunlong Liang

REVIEW 4 major objections 6 minor 24 references

THOR-MoE: Hierarchical Task-Guided and Context-Responsive Routing for Neural Machine Translation

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A hierarchical routing method that combines predicted task labels with context-aware token routing improves BLEU and reduces activated experts in translation MoE models.

desk verdict The BLEU gains from task-guided and context-responsive routing look real across two architectures, but the 'less than 22% activated parameters' efficiency claim is not supported by the reported experiments. read the letter →

arxiv 2505.14173 v1 pith:VSVRYVHS submitted 2025-05-20 cs.CL

Yunlong Liang , Fandong Meng , Jie Zhou This is my paper · ORCID

classification cs.CL

keywords routingtranslationdomainthor-moetokenciteexpertshierarchical

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Translation models built with sparse mixture-of-experts (MoE) use a router to send each word, or token, to a small number of specialized sub-models, called experts, instead of sending everything through one large network. The router usually looks only at the vector for the current token. THOR-MoE changes this in two ways. First, it adds a classifier that reads the whole input sentence and guesses which domain (for example law or medicine) or language the text belongs to. That guess is turned into a soft mixture of task embeddings, and a task-level router uses it to preselect a small set of experts for the sentence. Second, at the token level the model mixes the current token vector with the average of the already generated prefix before routing, through a learned gate. This lets the routing decision see the surrounding context. The authors test the method on German-to-English translation across five domains and on multilingual translation between 16 languages. They report that THOR-MoE improves BLEU scores over vanilla Top-k and Top-p routing while activating fewer experts on average. For example, on the multi-domain benchmark the context-aware version uses about 1.45 experts per token instead of 1.87, which is roughly 22 percent fewer, while scoring higher. They also report that ablations removing either the hierarchical task guidance or the context injection cause performance to drop. The core idea is a modest engineering change to how routing decisions are made in MoE translation models. The gains are real if the experiments are reproducible, but the paper does not release code, and some baselines are not ideal, so independent verification is needed.

Extended reading notes

Core claim

The central claim, stated in the abstract and Section 5.1, is that THOR-MoE 'can achieve an average improvement of 0.75 BLEU with less than 22% activated parameters' over vanilla Top-p, and that THOR-MoE (Top-2) 'outperforms the Trim-MoE (Top-2) by averaged 1.74 BLEU scores'. If correct, the routing changes yield better translation quality and fewer activated experts across both decoder-only and encoder-decoder MoE architectures.

Load-bearing premise

The design requires knowing the number of tasks and language groups in advance, as acknowledged in the Limitations: 'the design relies on the prior (the number of tasks and language groups)'. The task-level router preselects experts using a learned embedding per known task, so an input from an unseen domain or language has no learned task vector; the classifier is forced to map it onto the known set, and the 64.89% language-prediction accuracy reported in Appendix A suggests substantial misclassification. If the task prior is wrong, or the classifier's errors are not absorbed by the mixed representation, the hierarchical routing can only hurt.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The translation-quality story here is in decent shape: the proposed THOR-MoE routing, which predicts a soft task label, builds a mixed task embedding, preselects experts hierarchically, and then gates token routing with context, delivers consistent BLEU improvements over Top-p and Top-k baselines on both decoder-only and encoder-decoder MoE architectures. The ablations show both components matter, and the comparison against Lingual-MoE is the right one. That part I buy.

The soft spot is the efficiency claim in the abstract and intro: “0.75 BLEU with less than 22% activated parameters.” That pairing does not survive contact with the reported numbers. The 0.75 BLEU gain is from Table 1, which used the trimmed model defined in Section 4.2 as 3.5B total and 2.3B activated parameters — that is about 65.7% activated, not under 22%. The under-22% figure matches the full Qwen1.5-MoE-A2.7B in Appendix B, but that table compares against Top-8, not vanilla Top-p, and shows a different BLEU gap. Table 7 counts only non-shared experts (1.45 vs 1.87), leaving out the four always-active shared experts and the extra routing modules (task predictor, task-level router, context gate) that run for every token and every layer. So the headline efficiency claim is unverified as written, and the authors should be asked to redo the parameter accounting before anyone quotes the 22% number.

Other soft spots are more minor. The decoder-only dense baseline is confounded because Trim-MoE is initialized from a larger model (Qwen1.5-MoE-A2.7B) and compared against a dense model of different total size. Ablations have no error bars, and no code or data is released. The task predictor is only 64.89% accurate on language prediction; the paper argues the mixed representation provides fault tolerance, which the Table 5 results support, but it is still a limitation that the method assumes a fixed task set. The authors acknowledge this in the Limitations section, which is honest.

The paper is a solid extension of existing MoE routing ideas, not a new framework. I would send it to peer review — the core routing idea is useful to the NMT MoE community and the empirical gains are consistent. But a serious referee should require the efficiency numbers to be recomputed with shared experts and routing overhead included, and ideally release code or at least clarify the baseline confound. As written, the abstract overstates what is measured.

Referee Report

4 major / 6 minor

Summary. This paper proposes THOR-MoE, a routing method for sparse mixture-of-experts neural machine translation. It adds two components to existing Top-k/Top-p routing: (i) a hierarchical task-guided stage that predicts a domain/language label, forms a mixed task representation, and uses a task-level router to preselect a candidate expert set per query; and (ii) a context-responsive stage that injects an averaged prefix representation into the token-level router so that tokens are selected from the preselected set using context. The method is evaluated on multi-domain De-En translation with a trimmed decoder-only Qwen1.5-MoE model and on OPUS-16 multilingual translation with an encoder-decoder Transformer, reporting consistent BLEU improvements over Trim-MoE and Lingual-MoE baselines and an average activated-expert reduction from 1.87 to 1.45. The abstract further claims an average 0.75 BLEU gain over vanilla Top-p with 'less than 22% activated parameters.'

Significance. If the central claims held, the contribution would be practically valuable: a routing mechanism that improves translation quality while reducing activated experts, and that is compatible with both Top-k and Top-p routing across two architecture families. The paper deserves credit for evaluating on two benchmarks, for including a controlled Top-p comparison with a matched threshold, for ablating the two proposed components, and for validating on the full Qwen1.5-MoE-A2.7B in Appendix B. However, the efficiency half of the headline claim is not supported by the reported numbers, and the decoder-only comparison is not capacity-matched. These issues are fixable in revision, but they are load-bearing for the abstract's central claim.

major comments (4)

[Abstract / §4.2 / §6.4 / Table 7] The central efficiency claim 'less than 22% activated parameters' is unsupported and internally inconsistent. The 0.75 BLEU improvement in the abstract is computed on the trimmed model in Table 1, which Section 4.2 defines as having 3.5B total and 2.3B activated parameters (approximately 65.7% activated), not below 22%. The 'less than 22%' figure matches the full Qwen1.5-MoE-A2.7B in Appendix B, but Appendix B has no Top-p comparison and does not report the 0.75 BLEU gain. Moreover, Table 7's 1.45 versus 1.87 activated experts counts only non-shared experts and omits the four always-active shared experts and the added routing modules (task predictor, task-level router, context gate) that execute for every token and every layer. Please report a complete activated-parameter budget for each model in Table 1 and Appendix B, or revise the abstract so that the efficiency claim is tied to the setting in which it is measured.
[§4.2 / §5.1 / Table 1] The decoder-only comparison is confounded. Trim-MoE is initialized from the trimmed Qwen1.5-MoE-A2.7B, and THOR-MoE is built on the same trimmed model but adds trainable parameters (task predictor, task-level router, context gate), while the dense SFT-3B baseline is a different architecture with no MoE. The reported 1.74 BLEU advantage of THOR-MoE (Top-2) over Trim-MoE (Top-2) therefore conflates the new routing mechanism with added capacity and different fine-tuning conditions. Please add a baseline with the same auxiliary modules but without hierarchical/context routing, or otherwise match parameter counts, and report multiple seeds with confidence intervals.
[§3.1 / Appendix A / Limitations] The method requires the number of domains/language groups to be fixed in advance, and the task predictor is trained on golden labels for those known categories. The high-level claim that the method 'automatically' obtains task knowledge is therefore only true within a fixed task inventory. With language-prediction accuracy of 64.89% (Appendix A), the routing is driven by a noisy classifier, and the paper does not quantify how misclassification affects expert selection or translation quality. Please report routing robustness to classifier confidence and evaluate on at least one held-out domain or language pair not in the training inventory, or explicitly restrict the applicability claims in the abstract and introduction.
[§6.1 / Table 4] The ablation table reports single-run BLEU scores without variance or significance testing. The contribution of the context-responsive component is only 0.74 BLEU on multi-domain and 0.79 on multilingual (Table 4), and no confidence intervals are given. Please provide confidence intervals, bootstrap tests, or multiple seeds for the ablations, and specify how the '†/‡' markers in the main tables were computed (e.g., paired bootstrap over sentences).

minor comments (6)

[§3.3 / Eq. (14)] The symbols L_dp and L_bp in Eq. (14) do not match the earlier definitions L_tp, L_bd, and L_bt; please unify the notation.
[§3.3 / Eq. (10)] The min operator in Eq. (10) appears to be a typo; the task prediction loss should be -log P_t[t_g], not min(-log P_t[t_g]).
[Table 10] The caption of Table 10 says 'BLEU (top block) / COMET (below block)' but the table reports a domain similarity matrix; please correct the caption.
[§4.3] The text says 'we fine-tune two dense models based on Qwen2.5-3B' but Table 1 reports only one SFT-3B model; please clarify.
[§8 / §1] There are typos: 'task-gudied' in the Conclusion should be 'task-guided', and 'a innovative' in the Introduction should be 'an innovative'.
[Figure 2 / Table 7] The y-axis label 'Activated Experts' and the caption of Table 7 should state explicitly whether shared experts are included; the current text is ambiguous and inconsistent with the model description in §4.2.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim rests on three domain assumptions (known task inventory, informative pooled sentence representation, and sufficient averaged context) and on several tuned hyperparameters. No new physical or abstract entities are introduced; the mixed task representation is a learned embedding matrix, not a separately postulated entity.

free parameters (3)

loss weights alpha, beta, gamma (and delta for Top-p) = 1e-2, 1e-2, 1e-2, 1e-4 (multi-domain); 1e-2, 5e-2, 5e-2, 1e-4 (multilingual)
Tuned via grid search on each benchmark; the exact grid is not reported.
Top-p threshold p = 0.5
Hand-chosen threshold controlling how many experts are activated; results reported only for p=0.5 for both baseline and method.
task-level expert set size
The paper defines task-level preselection with TopK but never states the value of K (or N/K) used in experiments.

assumptions (3)

domain assumption The number of tasks/language groups is known and fixed before training.
Explicitly stated in Limitations; the task router and task embeddings are built for a fixed set of K tasks.
domain assumption A single [CLS] pooled representation captures enough sentence-level information to predict the correct task label.
Used in Eq. 8 for task prediction; reported accuracies are 82.45% for domains and 64.89% for languages.
domain assumption The average of prefix hidden states is a sufficient context representation for token routing.
Defined in Section 3.2 as H_ctx and injected into token routing; no alternative context encoding is studied.

how reviews work

0 comments

Cite this review

Pith. "Pith review of THOR-MoE: Hierarchical Task-Guided and Context-Responsive Routing for Neural Machine Translation." pith.science (2026). https://pith.science/paper/VSVRYVHS

@misc{pith2026250514173,
  author       = {Pith},
  title        = {Pith review of: THOR-MoE: Hierarchical Task-Guided and Context-Responsive Routing for Neural Machine Translation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VSVRYVHS}},
  note         = {Machine review of arXiv:2505.14173}
}

abstract

The sparse Mixture-of-Experts (MoE) has achieved significant progress for neural machine translation (NMT). However, there exist two limitations in current MoE solutions which may lead to sub-optimal performance: 1) they directly use the task knowledge of NMT into MoE (\emph{e.g.}, domain/linguistics-specific knowledge), which are generally unavailable at practical application and neglect the naturally grouped domain/linguistic properties; 2) the expert selection only depends on the localized token representation without considering the context, which fully grasps the state of each token in a global view. To address the above limitations, we propose THOR-MoE via arming the MoE with hierarchical task-guided and context-responsive routing policies. Specifically, it 1) firstly predicts the domain/language label and then extracts mixed domain/language representation to allocate task-level experts in a hierarchical manner; 2) injects the context information to enhance the token routing from the pre-selected task-level experts set, which can help each token to be accurately routed to more specialized and suitable experts. Extensive experiments on multi-domain translation and multilingual translation benchmarks with different architectures consistently demonstrate the superior performance of THOR-MoE. Additionally, the THOR-MoE operates as a plug-and-play module compatible with existing Top-$k$~\cite{shazeer2017} and Top-$p$~\cite{huang-etal-2024-harder} routing schemes, ensuring broad applicability across diverse MoE architectures. For instance, compared with vanilla Top-$p$~\cite{huang-etal-2024-harder} routing, the context-aware manner can achieve an average improvement of 0.75 BLEU with less than 22\% activated parameters on multi-domain translation tasks.

Figures

Figures reproduced from arXiv: 2505.14173 by the authors.

**Figure 2.** Average activated experts number across train [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 18 canonical work pages

[2]

In Proceedings of the 22nd Nordic Conference on Com- putational Linguistics, NoDaLiDa 2019, Turku, Fin- land, September 30 - October 2, 2019, pages 389–

The OPUS resource repository: An open package for creating parallel corpora and machine translation services. In Proceedings of the 22nd Nordic Conference on Com- putational Linguistics, NoDaLiDa 2019, Turku, Fin- land, September 30 - October 2, 2019, pages 389–

work page 2019
[3]

BERT: Pre-training of deep bidirectional transformers for language under- standing. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, V olume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. Maha E...

work page 2019
[8]

DEMix layers: Disentangling domains for modular language modeling. InProceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 5557–5576, Seattle, United States. Association for Computational Linguistics. Quzhe Huang, Zhenwei An, Nan Zhuang, Mingxu Tao, Chen Zhan...

work page 2022
[10]

InFindings of the Associ- ation for Computational Linguistics: EMNLP 2024, pages 3843–3860, Miami, Florida, USA

Med-MoE: Mixture of domain-specific experts for lightweight medical vision-language models. InFindings of the Associ- ation for Computational Linguistics: EMNLP 2024, pages 3843–3860, Miami, Florida, USA. Association for Computational Linguistics. Philipp Koehn

work page 2024
[13]

Jiamin Li, Qiang Su, Yitao Yang, Yimin Jiang, Cong Wang, and Hong Xu

OpenRe- view.net. Jiamin Li, Qiang Su, Yitao Yang, Yimin Jiang, Cong Wang, and Hong Xu. 2023a. Adaptive gating in mixture-of-experts based language models. InPro- ceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing, pages 3577– 3587, Singapore. Association for Computational Lin- guistics. Margaret Li, Suchin Gururangan, T...

work page 2023
[14]

Samyam Rajbhandari, Conglong Li, Zhewei Yao, Min- jia Zhang, Reza Yazdani Aminabadi, Ammar Ah- mad Awan, Jeff Rasley, and Yuxiong He

O1 replication journey: A strategic progress report–part 1.arXiv preprint arXiv:2410.18982. Samyam Rajbhandari, Conglong Li, Zhewei Yao, Min- jia Zhang, Reza Yazdani Aminabadi, Ammar Ah- mad Awan, Jeff Rasley, and Yuxiong He

arXiv
[15]

Preprint, arXiv:2201.05596

Deepspeed-moe: Advancing mixture-of-experts in- ference and training to power next-generation ai scale. Preprint, arXiv:2201.05596. Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He

arXiv
[16]

InProceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 2685–2702, Online

COMET: A neural framework for MT evaluation. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics. Noam Shazeer, *Azalia Mirhoseini, *Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017a. Outrageously large neural net-...

work page 2020

Show all 24 references

[17]

Junhong Wu, Yuchen Liu, and Chengqing Zong

Hmoe: Heterogeneous mixture of experts for language modeling.Preprint, arXiv:2408.10681. Junhong Wu, Yuchen Liu, and Chengqing Zong

arXiv
[18]

F-MALLOC: Feed-forward memory allocation for continual learning in neural machine translation. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies (V olume 1: Long Papers), pages 71...

2024
[19]

InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7266–7279, Online and Punta Cana, Dominican Republic

Im- proving multilingual translation by representation and gradient regularization. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7266–7279, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. ...

2021
[20]

InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 6223–6235, Miami, Florida, USA

AdaMoE: Token-adaptive routing with null experts for mixture-of-experts lan- guage models. InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 6223–6235, Miami, Florida, USA. Association for Computational Linguistics. Biao Zhang, Ankur Bapna, Rico Se...

2024
[21]

In Findings of the Association for Computational Lin- guistics: NAACL 2024, pages 2381–2392, Mexico City, Mexico

A lightweight mixture-of-experts neural machine trans- lation model with stage-wise training strategy. In Findings of the Association for Computational Lin- guistics: NAACL 2024, pages 2381–2392, Mexico City, Mexico. Association for Computational Lin- guistics. Hao Zhao, Zihan...

2024
[22]

Simiao Zuo, Xiaodong Liu, Jian Jiao, Young Jin Kim, Hany Hassan, Ruofei Zhang, Jianfeng Gao, and Tuo Zhao

St-moe: Designing stable and transferable sparse expert models.arXiv preprint arXiv:2202.08906. Simiao Zuo, Xiaodong Liu, Jian Jiao, Young Jin Kim, Hany Hassan, Ruofei Zhang, Jianfeng Gao, and Tuo Zhao

arXiv
[23]

InThe Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29,

Taming sparsely activated transformer with stochastic experts. InThe Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29,

2022
[24]

OpenReview.net. Tasks/Groups Train Valid Test Multi-Domain Translation Dataset (De→En) IT 0.22M 2000 2000 Koran 18K Law 0.47M Medical 0.25M Subtitles 0.5M Multilingual Translation Dataset 16 17,559,950 30*1000 30*1000 Table 8: The data statistic of the multi-domain trans- lati...

2000
[1991]

Ganesh Jawahar, Subhabrata Mukherjee, Xiaodong Liu, Young Jin Kim, Muhammad Abdul-Mageed, Laks Lakshmanan, V .S., Ahmed Hassan Awadallah, Se- bastien Bubeck, and Jianfeng Gao

Adaptive mixtures of local experts.Neural Comput., 3(1):79–87. Ganesh Jawahar, Subhabrata Mukherjee, Xiaodong Liu, Young Jin Kim, Muhammad Abdul-Mageed, Laks Lakshmanan, V .S., Ahmed Hassan Awadallah, Se- bastien Bubeck, and Jianfeng Gao. 2023a. Auto- MoE: Heterogeneous mixtur...

2023
[2017]

InProceedings of the First Workshop on Neural Machine Transla- tion, NMT@ACL 2017, V ancouver , Canada, August 4, 2017, pages 28–39

Six chal- lenges for neural machine translation. InProceedings of the First Workshop on Neural Machine Transla- tion, NMT@ACL 2017, V ancouver , Canada, August 4, 2017, pages 28–39. Sneha Kudugunta, Yanping Huang, Ankur Bapna, Maxim Krikun, Dmitry Lepikhin, Minh-Thang Lu- ong,...

2017
[2019]

Massively multilingual neural machine translation. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, V olume 1 (Long and Short Papers), pages 3874–3884, Minneapolis, Minnesota. As...

2019
[2020]

William Fedus, Barret Zoph, and Noam Shazeer

Be- yond english-centric multilingual machine transla- tion.CoRR, abs/2010.11125. William Fedus, Barret Zoph, and Noam Shazeer

2010 arXiv
[2021]

InFindings of the Association for Computational Linguistics: EMNLP 2021, pages 3577–3599, Punta Cana, Dominican Republic

Beyond distillation: Task-level mixture-of-experts for efficient inference. InFindings of the Association for Computational Linguistics: EMNLP 2021, pages 3577–3599, Punta Cana, Dominican Republic. Association for Compu- tational Linguistics. Dmitry Lepikhin, HyoukJoong Lee, Y...

2021
[2022]

InProceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 1707–1718, Abu Dhabi, United Arab Emirates

Continual learning of neural machine translation within low for- getting risk regions. InProceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 1707–1718, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics. Suc...

2022
[2023]

InFindings of the Association for Computational Linguistics: ACL 2023, pages 14237–14253, Toronto, Canada

Fixing MoE over-fitting on low-resource languages in multilingual machine translation. InFindings of the Association for Computational Linguistics: ACL 2023, pages 14237–14253, Toronto, Canada. Associ- ation for Computational Linguistics. Angela Fan, Shruti Bhosale, Holger Sch...

2023
[2024]

Shuhao Gu and Yang Feng

Bet- ter & faster large language models via multi-token prediction.Preprint, arXiv:2404.19737. Shuhao Gu and Yang Feng

arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [2]

In Proceedings of the 22nd Nordic Conference on Com- putational Linguistics, NoDaLiDa 2019, Turku, Fin- land, September 30 - October 2, 2019, pages 389–

The OPUS resource repository: An open package for creating parallel corpora and machine translation services. In Proceedings of the 22nd Nordic Conference on Com- putational Linguistics, NoDaLiDa 2019, Turku, Fin- land, September 30 - October 2, 2019, pages 389–

work page 2019

[2] [3]

BERT: Pre-training of deep bidirectional transformers for language under- standing. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, V olume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. Maha E...

work page 2019

[3] [8]

DEMix layers: Disentangling domains for modular language modeling. InProceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 5557–5576, Seattle, United States. Association for Computational Linguistics. Quzhe Huang, Zhenwei An, Nan Zhuang, Mingxu Tao, Chen Zhan...

work page 2022

[4] [10]

InFindings of the Associ- ation for Computational Linguistics: EMNLP 2024, pages 3843–3860, Miami, Florida, USA

Med-MoE: Mixture of domain-specific experts for lightweight medical vision-language models. InFindings of the Associ- ation for Computational Linguistics: EMNLP 2024, pages 3843–3860, Miami, Florida, USA. Association for Computational Linguistics. Philipp Koehn

work page 2024

[5] [13]

Jiamin Li, Qiang Su, Yitao Yang, Yimin Jiang, Cong Wang, and Hong Xu

OpenRe- view.net. Jiamin Li, Qiang Su, Yitao Yang, Yimin Jiang, Cong Wang, and Hong Xu. 2023a. Adaptive gating in mixture-of-experts based language models. InPro- ceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing, pages 3577– 3587, Singapore. Association for Computational Lin- guistics. Margaret Li, Suchin Gururangan, T...

work page 2023

[6] [14]

Samyam Rajbhandari, Conglong Li, Zhewei Yao, Min- jia Zhang, Reza Yazdani Aminabadi, Ammar Ah- mad Awan, Jeff Rasley, and Yuxiong He

O1 replication journey: A strategic progress report–part 1.arXiv preprint arXiv:2410.18982. Samyam Rajbhandari, Conglong Li, Zhewei Yao, Min- jia Zhang, Reza Yazdani Aminabadi, Ammar Ah- mad Awan, Jeff Rasley, and Yuxiong He

arXiv

[7] [15]

Preprint, arXiv:2201.05596

Deepspeed-moe: Advancing mixture-of-experts in- ference and training to power next-generation ai scale. Preprint, arXiv:2201.05596. Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He

arXiv

[8] [16]

InProceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 2685–2702, Online

COMET: A neural framework for MT evaluation. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics. Noam Shazeer, *Azalia Mirhoseini, *Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017a. Outrageously large neural net-...

work page 2020