Routing-Aligned Fine-Tuning for Multilingual Downstream Tasks in Mixture-of-Experts Models

Guanzhi Deng; Haibo Wang; Kuan Wu; Linqi Song; Shing Yin Wong; Sichun Luo

arxiv: 2605.28306 · v1 · pith:SQ2R5XYKnew · submitted 2026-05-27 · 💻 cs.CL · cs.AI

Routing-Aligned Fine-Tuning for Multilingual Downstream Tasks in Mixture-of-Experts Models

Guanzhi Deng , Kuan Wu , Haibo Wang , Shing Yin Wong , Sichun Luo , Linqi Song This is my paper

Pith reviewed 2026-06-29 13:07 UTC · model grok-4.3

classification 💻 cs.CL cs.AI

keywords mixture-of-expertsmultilingual fine-tuningrouting alignmentdownstream taskslanguage-universal layerssupervised fine-tuningexpert activationparallel examples

0 comments

The pith

RA-MoE aligns middle-layer routing in MoE models to English task-expert patterns on ci-type examples to close multilingual performance gaps.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper shows that standard fine-tuning of Mixture-of-Experts models overlooks the routing decisions shaped during pretraining, which creates uneven performance across languages. It identifies middle layers as a language-universal zone where routing divergence between English and target-language examples predicts task accuracy differences. RA-MoE adds a routing alignment loss that forces target-language routing on ci examples (correct in English, incorrect in target language) to match the English expert activation pattern. This produces consistent gains over plain supervised fine-tuning and prior routing methods across models, tasks, and languages. The share of ci examples in a task-language pair also predicts how much the alignment step helps.

Core claim

Middle layers form a language-universal alignment zone; routing divergence there predicts per-language task performance gaps. RA-MoE therefore categorizes parallel examples into a four-way taxonomy (cc/ci/ic/ii), locates task-relevant experts in those layers, and augments supervised fine-tuning with a routing alignment loss that makes target-language routing on ci examples follow the English task-expert pattern.

What carries the argument

The routing alignment loss applied only to ci-type examples, which encourages target-language activations in middle-layer experts to match the English task-expert pattern identified from parallel data.

If this is right

RA-MoE improves accuracy over standard SFT and baselines on three MoE models, three tasks, and six target languages.
The proportion of ci examples for a given task-language pair reliably predicts the size of the benefit from the alignment step.
Task-relevant experts can be identified from English performance in the middle layers and then reused for target-language alignment.
The four-way taxonomy (cc/ci/ic/ii) isolates the subset of examples where routing alignment is most useful.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same taxonomy and alignment loss could be applied to other routing-based architectures beyond the three MoE models tested.
If the middle-layer zone is stable across pretraining runs, the method might transfer to new MoE models without retraining the identifier step.
Extending the taxonomy to more than two languages could reveal whether the alignment benefit scales with the number of mismatched languages.

Load-bearing premise

Middle layers form a language-universal alignment zone where routing divergence strongly predicts per-language task performance gaps.

What would settle it

An experiment that measures routing divergence in middle layers on held-out parallel data and finds no correlation with per-language accuracy gaps on the downstream tasks.

Figures

Figures reproduced from arXiv: 2605.28306 by Guanzhi Deng, Haibo Wang, Kuan Wu, Linqi Song, Shing Yin Wong, Sichun Luo.

**Figure 2.** Figure 2: Overview of the RA-MoE framework. the median of the per-layer divergence distribution, a threshold that adapts automatically to each model’s divergence profile and consistently identifies approximately the middle third of transformer layers across all three models in our experiments. Task expert identification. We identify a set of task experts in each middle layer ℓ ∈ Lmid—the experts that M preferentia… view at source ↗

**Figure 3.** Figure 3: Sensitivity analysis on Qwen1.5-MoE (GSM8K). Solid lines show RA-MoE; dashed lines indicate SFT baseline. Left: varying λ with K=8. Right: varying K with λ=1.0. of improvement. Extending alignment beyond middle layers also leads to a substantial degradation, consistent with the finding that early and late layers are language-specific and should not be constrained toward English routing. Replacing task ex… view at source ↗

**Figure 4.** Figure 4: Relative gain of RA-MoE over SFT vs. ci proportion across six target languages and three models (GSM8K). The dashed line shows the linear fit. a K fails to cover sufficient task-relevant experts, while too large a K introduces task-irrelevant experts that dilute the alignment target. 4.4 Analysis 4.4.1 Effect of ci Proportion We examine whether the proportion of ci examples in a task-language pair predict… view at source ↗

**Figure 5.** Figure 5: Layer-wise mean routing divergence between English and target-language inputs on Qwen1.5-MoE [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗

**Figure 6.** Figure 6: Eval CE loss (solid lines, left axis) and task [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗

read the original abstract

Mixture-of-Experts (MoE) models have emerged as a dominant paradigm for efficient LLM scaling, yet adapting them to non-English downstream tasks remains challenging. Existing fine-tuning approaches treat MoE models as monolithic learners, ignoring the heterogeneous routing structure that develops during pretraining. We validate across multiple MoE models and downstream tasks that middle layers form a language-universal alignment zone where routing divergence strongly predicts per-language task performance gaps. Building on this observation, we propose RA-MoE (Routing-Aligned MoE Fine-Tuning), a three-stage framework that categorizes parallel task examples into a four-way taxonomy (cc/ci/ic/ii) based on correctness in English and the target language, identifies task-relevant experts in the middle layers, and augments standard SFT with a routing alignment loss that encourages target-language routing on ci-type examples to follow the English task-expert activation pattern. Experiments across three MoE models, three tasks, and six target languages demonstrate that RA-MoE consistently outperforms standard SFT and strong baselines including Routing Steering and RISE, with the ci proportion of a task-language pair serving as a reliable predictor of alignment benefit.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

RA-MoE introduces a taxonomy-driven routing alignment loss for MoE multilingual fine-tuning that reportedly beats SFT and baselines, but the middle-layer observation needs direct verification.

read the letter

The core of this paper is a three-stage fine-tuning method called RA-MoE. It first builds a four-way taxonomy of parallel examples (cc/ci/ic/ii) from English and target-language correctness, then adds a routing alignment loss that pushes ci-type examples to use the same middle-layer experts as their English counterparts.

What stands out as new is the explicit taxonomy tied to a claimed middle-layer language-universal zone, plus the use of ci proportion as a predictor of how much alignment will help. The abstract says this yields consistent gains over standard SFT and two routing baselines across three MoE models, three tasks, and six languages.

The approach addresses a real practical gap: MoE models route differently by language, and standard fine-tuning ignores that structure. If the middle-layer observation and the reported gains hold up in the full experiments, the method gives practitioners a targeted way to close some performance gaps without full retraining.

The soft spots are the lack of visible experimental details in the abstract—no error bars, no statistical tests, no layer-wise routing plots. The claim that middle layers form a reliable alignment zone rests on an unreported validation step, and the ci proportion is derived from the same taxonomy used to pick examples for the loss, even if downstream metrics stay independent. Those points need checking in the full text.

This work is aimed at groups already running MoE models and trying to make them usable across languages. It is worth sending to peer review because it supplies a concrete, testable procedure for a scaling problem that matters in practice; referees can verify the layer analysis and the strength of the gains.

Referee Report

2 major / 0 minor

Summary. The paper claims that middle layers in MoE models constitute a language-universal alignment zone where routing divergence predicts per-language task performance gaps. It introduces RA-MoE, a three-stage framework that builds a four-way taxonomy (cc/ci/ic/ii) of parallel examples based on English vs. target-language correctness, identifies task-relevant experts in middle layers, and augments SFT with a routing alignment loss that steers ci-type target-language routing toward the English expert pattern. Experiments across three MoE models, three tasks, and six languages are said to show consistent gains over SFT and baselines (Routing Steering, RISE), with the ci proportion of a task-language pair serving as a reliable predictor of alignment benefit.

Significance. If the empirical claims hold, the work would supply a routing-aware fine-tuning method that exploits rather than ignores MoE heterogeneity, together with an observable predictor (ci proportion) of when alignment is useful. This could matter for efficient multilingual adaptation of large MoE models.

major comments (2)

[Abstract] Abstract: the central claims of consistent outperformance and a predictive relationship are asserted without any experimental details, error bars, statistical tests, tables, or verification of the middle-layer observation; the soundness of the contribution therefore cannot be assessed from the provided text.
[Method description (taxonomy and loss)] Taxonomy and loss construction: the ci proportion used as a predictor is computed from the same four-way taxonomy that selects examples for the alignment loss, introducing partial dependence between the predictor and the training signal; downstream metrics are measured independently, but the circularity risk for the predictive claim is not addressed.

Simulated Author's Rebuttal

2 responses · 0 unresolved

Thank you for the opportunity to respond to the referee's report. We address each major comment point by point below. We believe the concerns can be addressed through clarifications and minor revisions to the manuscript.

read point-by-point responses

Referee: [Abstract] Abstract: the central claims of consistent outperformance and a predictive relationship are asserted without any experimental details, error bars, statistical tests, tables, or verification of the middle-layer observation; the soundness of the contribution therefore cannot be assessed from the provided text.

Authors: The abstract serves as a high-level summary of the paper's contributions and findings. Detailed experimental setups, results with error bars (standard deviations from multiple seeds), statistical tests, tables comparing RA-MoE to baselines, and verification of the middle-layer alignment zone (including routing divergence metrics and layer-wise analysis) are provided in the main body of the manuscript, specifically in Sections 3, 4, and 5. We can revise the abstract to include brief mentions of the experimental scale (three MoE models, three tasks, six languages) if recommended. revision: partial
Referee: [Method description (taxonomy and loss)] Taxonomy and loss construction: the ci proportion used as a predictor is computed from the same four-way taxonomy that selects examples for the alignment loss, introducing partial dependence between the predictor and the training signal; downstream metrics are measured independently, but the circularity risk for the predictive claim is not addressed.

Authors: The taxonomy is built from correctness labels obtained by evaluating the pre-trained model on parallel English and target-language examples, which is done independently of the fine-tuning process. The ci proportion is a static characteristic of each task-language pair derived from this pre-evaluation. The routing alignment loss then leverages this taxonomy during training to align ci examples. The predictive relationship is assessed by measuring how well this pre-computed proportion correlates with the gains in downstream performance on held-out test data after applying RA-MoE. We will add explicit discussion in the revised manuscript to clarify this separation and mitigate any appearance of circularity. revision: yes

Circularity Check

0 steps flagged

No significant circularity detected

full rationale

The abstract and description outline a taxonomy (cc/ci/ic/ii) used both to select ci-type examples for the routing alignment loss and to compute the ci proportion as a predictor of alignment benefit. However, no equations, derivations, or explicit reductions are present that demonstrate any claimed prediction or result being equivalent to its inputs by construction. Downstream task metrics are described as measured independently, and the framework is presented as an empirical augmentation to SFT rather than a self-referential definition. Without load-bearing self-citations, fitted inputs renamed as predictions, or ansatzes smuggled via prior work, the derivation chain remains self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the empirical observation that middle layers constitute a language-universal alignment zone; no free parameters or invented entities are stated in the abstract.

axioms (1)

domain assumption Middle layers of MoE models form a language-universal alignment zone where routing divergence strongly predicts per-language task performance gaps.
Presented as a validated observation that directly motivates the routing alignment loss.

pith-pipeline@v0.9.1-grok · 5750 in / 1363 out tokens · 32466 ms · 2026-06-29T13:07:56.414450+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

4 extracted references · 3 canonical work pages · 3 internal anchors

[1]

DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model

Deepseek-v2: A strong, economical, and effi- cient mixture-of-experts language model.Preprint, arXiv:2405.04434. Guanzhi Deng, Bo Li, Ronghao Chen, Xiujin Liu, Zhuo Han, Huacan Wang, Lijie Wen, and Linqi Song. 2026. Dr-lora: Dynamic rank lora for fine-tuning mixture- of-experts models.Preprint, arXiv:2601.04823. Naman Goyal, Cynthia Gao, Vishrav Chaudhary...

work page internal anchor Pith review Pith/arXiv arXiv 2026
[2]

Measuring Massive Multitask Language Understanding

Measuring massive multitask language under- standing.Preprint, arXiv:2009.03300. Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. InInternational Conference on Learning Representations. Albert Q Jiang, Alexandre Sablayrolles, Antoine Rou...

work page internal anchor Pith review Pith/arXiv arXiv 2009
[3]

DeepSeek-V3 Technical Report

Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437. Danni Liu and Jan Niehues. 2025. Middle-layer repre- sentation alignment for cross-lingual transfer in fine- tuned llms. InProceedings of the 63rd Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15979–15996. 9 Niklas Muennighoff, Luca Soldaini, D...

work page internal anchor Pith review Pith/arXiv arXiv 2025
[4]

All training-based methods (SFT, RISE, RA-MoE) share identical training schedules, optimization set- tings, and data orders for a fair comparison

applied to the up and down projections of every FFN layer, with all other parameters frozen. All training-based methods (SFT, RISE, RA-MoE) share identical training schedules, optimization set- tings, and data orders for a fair comparison. Table 8 reports the hyperparameter settings shared across all models and tasks; Table 9 reports the per-model, per-ta...

2026

[1] [1]

DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model

Deepseek-v2: A strong, economical, and effi- cient mixture-of-experts language model.Preprint, arXiv:2405.04434. Guanzhi Deng, Bo Li, Ronghao Chen, Xiujin Liu, Zhuo Han, Huacan Wang, Lijie Wen, and Linqi Song. 2026. Dr-lora: Dynamic rank lora for fine-tuning mixture- of-experts models.Preprint, arXiv:2601.04823. Naman Goyal, Cynthia Gao, Vishrav Chaudhary...

work page internal anchor Pith review Pith/arXiv arXiv 2026

[2] [2]

Measuring Massive Multitask Language Understanding

Measuring massive multitask language under- standing.Preprint, arXiv:2009.03300. Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. InInternational Conference on Learning Representations. Albert Q Jiang, Alexandre Sablayrolles, Antoine Rou...

work page internal anchor Pith review Pith/arXiv arXiv 2009

[3] [3]

DeepSeek-V3 Technical Report

Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437. Danni Liu and Jan Niehues. 2025. Middle-layer repre- sentation alignment for cross-lingual transfer in fine- tuned llms. InProceedings of the 63rd Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15979–15996. 9 Niklas Muennighoff, Luca Soldaini, D...

work page internal anchor Pith review Pith/arXiv arXiv 2025

[4] [4]

All training-based methods (SFT, RISE, RA-MoE) share identical training schedules, optimization set- tings, and data orders for a fair comparison

applied to the up and down projections of every FFN layer, with all other parameters frozen. All training-based methods (SFT, RISE, RA-MoE) share identical training schedules, optimization set- tings, and data orders for a fair comparison. Table 8 reports the hyperparameter settings shared across all models and tasks; Table 9 reports the per-model, per-ta...

2026