The reviewed record of science sign in
Pith

arxiv: 2607.06109 · v1 · pith:6VGBVS3W · submitted 2026-07-07 · cs.CV · cs.AI

RoME: Robust Mixture of Low-Rank Experts against Multiple Adversarial Perturbations

Woo Jae Kim , Kyle Min , Suhyeon Ha , Joonsung Jeon , Sung-eui Yoon This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-07-08 16:16 UTCglm-5.2pith:6VGBVS3Wrecord.jsonopen to challenge →

classification cs.CV cs.AI
keywords adversarial robustnessmulti-perturbation adversarial trainingmixture of expertslow-rank adaptationthreat-specific routingdual-scale gatingunion robustness
0
0 comments X

The pith

Low-rank experts route each adversarial threat type through a separ

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that multi-perturbation adversarial training suffers from cross-threat trade-offs because existing methods force all adversarial perturbation types through a single shared model pathway, creating conflicting optimization objectives. The authors identify a specific failure mode when naively applying mixture-of-experts (MoE) to this problem: threat-agnostic routing, where the gating network cannot distinguish between different perturbation types and assigns nearly identical expert combinations regardless of threat. To address this, the paper proposes RoME, which implements each expert as a low-rank additive update (LoRA-style) to a shared backbone so the backbone captures threat-common features while experts specialize on threat-specific information. The gating mechanism uses dual-scale features (local patch-level and global image-level) because the authors observe that different perturbation types are separable at different feature scales, and a diversification loss explicitly enforces distinct routing patterns across threats. The combination achieves state-of-the-art union robustness across CIFAR-10, ImageNet-100, and ImageNet-1K with only 1.04x parameter overhead and 1.17x training time, while also improving robustness against unseen threat types and natural accuracy.

Core claim

The central finding is that adversarial perturbation types (l1, l2, l_inf) exhibit complementary discriminative structure across feature scales: l1 threats are more separable in local patch-level features while l_inf threats are more separable in global image-level features. By building a gating mechanism that exploits both scales simultaneously, and by implementing experts as low-rank additive updates rather than disjoint feed-forward networks, the paper shows that distinct threat-specific model pathways can be constructed within a single model. This routing specialization directly mitigates the cross-threat robustness trade-off that has limited multi-perturbation adversarial training, and,

What carries the argument

Three components carry the argument: (1) Low-rank experts implemented as LoRA-style additive updates to shared backbone weights, separating threat-common from threat-specific features; (2) Dual-scale gating combining local patch-level gating (per-token MLP) with global image-level gating (averaged-token MLP), merged via a layer-adaptive coefficient that emphasizes global features in early Transformer layers and local features in deeper layers; (3) Threat-guided gating diversification loss that maximizes pairwise Euclidean distance between average gating patterns across threat types, with a projection layer for global gating to a higher dimension for finer discrimination. Threat labels are用于d

Load-bearing premise

The dual-scale gating design assumes that the observed complementarity in threat separability between local and global features (l1 better separated locally, l_inf globally) is a stable structural property that generalizes beyond the specific ViT-B/CIFAR-10 setup used for the analysis, and that the fixed prior about Transformer layer hierarchy (global features in early layers, local in deep layers) holds across architectures and datasets. Additionally, the diversification

What would settle it

If the complementarity between local and global feature separability does not hold on other architectures (e.g., CNNs beyond the tested WideResNets) or datasets beyond CIFAR-10 and ImageNet, the dual-scale gating would provide no additional discriminative signal over single-scale gating, weakening the central routing mechanism. The ablation in Table 4 showing that inverted or learned layer weights underperform the fixed prior provides partial evidence but only on CIFAR-10 with ViT-B. A direct falsification would be demonstrating that single-scale gating matches or exceeds dual-scale gating on

Figures

Figures reproduced from arXiv: 2607.06109 by Joonsung Jeon, Kyle Min, Suhyeon Ha, Sung-Eui Yoon, Woo Jae Kim.

Figure 2
Figure 2. Figure 2: Conceptual visualization of conventional mixture of experts and our approach. Conventional methods (left) face the threat-agnostic routing issue and route different threats through similar ex￾pert combinations, resulting in a single shared model pathway for distinct threats. Our approach (right) effectively routes each threat through distinct expert combinations, constructing multiple threat-specific model… view at source ↗
Figure 3
Figure 3. Figure 3: Gaussian KDE analysis on expert activations across different threat types on CIFAR-10. (a) Naïve MoE suffers from routing collapse (Expert 1 dominates) and threat-agnostic routing (similar routing across threats). (b) Load balancing [56] resolves routing collapse but exhibits threat-agnostic routing. (c) Our RoME addresses threat￾agnostic routing issue and learns diverse, threat-specific expert combination… view at source ↗
Figure 4
Figure 4. Figure 4: Overview of RoME. (i) We train a mixture of robust low-rank experts, where a shared backbone captures threat-common features while low-rank experts focus on threat-specific information. To address threat-agnostic routing, where the gating as￾signs similar expert combinations across threats, we propose (ii) threat-distinguishing dual-scale gating, which leverages local patch-level and global image-level fea… view at source ↗
Figure 5
Figure 5. Figure 5: t-SNE visualization of threat separability at different feature levels. (a) Local patch-level features better separate ℓ1 threats (silhouette: 0.394) but show overlap for ℓ∞ and ℓ2. (b) Global image-level features show the complementary pattern, better distinguishing ℓ∞ (silhouette: 0.611) over ℓ1 and ℓ2. Gating weight gi = [gi,1, . . . , gi,K] ⊤ ∈ R K is predicted for each token i (i.e., patch) in an imag… view at source ↗
Figure 6
Figure 6. Figure 6: t-SNE visualization of (a) backbone features and (b) backbone + expert outputs from the final ViT-B layer [PITH_FULL_IMAGE:figures/full_fig_p014_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Expert activations for various ℓp threats applied to different regions of im￾age. While standard MoE (rows 1, 3) routes threats to same experts, RoME (rows 2, 4) utilizes different experts for each threat [PITH_FULL_IMAGE:figures/full_fig_p015_7.png] view at source ↗
read the original abstract

Multi-perturbation adversarial training (MAT) aims to achieve robustness against multiple $\ell_p$ perturbations but suffers from robustness trade-offs between different threats. To address this, we employ a mixture of experts (MoE) to route different threats through distinct model pathways. However, naive application of MoE encounters two critical challenges: experts tend to overlook threat-specific features and redundantly capture features shared across threats, and gating networks suffer from threat-agnostic routing where they learn nearly identical routing patterns across threats, thus preventing the construction of threat-specific model pathways. To this end, we propose Robust Mixture of Low-Rank Experts (RoME), where each expert is a low-rank additive update to the shared backbone, allowing it to capture threat-common features while experts focus on threat-specific information. To address threat-agnostic routing, RoME introduces (i) dual-scale gating that exploits threat-discriminative signals from local and global level features, and (ii) threat-guided gating diversification that enforces diverse expert utilization across threats. Extensive experiments demonstrate that RoME outperforms existing state-of-the-art MAT in union robustness and natural accuracy and improves robustness against unseen threats. Codes are available at https://github.com/wkim97/RoME.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 7 minor

Summary. This paper proposes RoME (Robust Mixture of Low-Rank Experts), a framework for multi-perturbation adversarial training (MAT) that addresses cross-threat robustness trade-offs. The method introduces three components: (1) low-rank additive experts (LoRA-style) on a shared backbone to separate threat-common from threat-specific features, (2) dual-scale gating that combines local patch-level and global image-level features to provide threat-discriminative signals, and (3) threat-guided gating diversification loss that enforces distinct expert routing across different threat types. Experiments are conducted on CIFAR-10, ImageNet-100, and ImageNet-1K using ViT-B, DeiT-B, Swin-B, WRN, and XCiT architectures, comparing against 7+ MAT baselines under PGD and APGD. The method also evaluates robustness against unseen threats (common corruptions, non-ℓp attacks, adaptive GMA attack) and demonstrates applicability to robust fine-tuning and non-ℓp perceptual adversarial training.

Significance. The paper addresses a well-known problem in multi-perturbation adversarial training: the robustness trade-off between different ℓp threat types. The identification and characterization of the 'threat-agnostic routing' problem in conventional MoE applied to MAT is a genuine conceptual contribution. The solution is well-motivated: low-rank experts provide parameter efficiency (only 1.04× parameters, 1.17× training time over baseline), and the dual-scale gating is grounded in an empirical observation about feature separability. The experimental evaluation is thorough, spanning multiple datasets, architectures (including CNNs), threat models, and an adaptive white-box attack (GMA). The modular design is validated by applying RoME on top of multiple base methods (MAX, RANDOM, E-AT, RAMP, PAT, VR). Code is publicly available, which supports reproducibility. The gains over baselines are consistent, though modest in some settings.

major comments (1)
  1. §4.3, Fig. 5, and Table 4: The dual-scale gating mechanism is motivated by the observation (Fig. 5) that ℓ1 threats are more separable at patch-level features (silhouette 0.394) while ℓ∞ threats are more separable at image-level features (silhouette 0.611). This observation is made exclusively on ViT-B trained with RANDOM on CIFAR-10. The layer-adaptive coefficient β^(l) in Eq. (5) encodes a fixed prior about Transformer layer hierarchy (global in early layers, local in deep layers) based on NLP citations [20, 37], and the ablation isolating dual-scale gating (Table 4: 41.1 → 39.4 local-only, 39.8 global-only) is also CIFAR-10 only. While the method achieves consistent gains across datasets and architectures (Tables 1, 3, A1, A2), the paper does not verify whether the complementarity in feature separability holds on ImageNet or CNN architectures (WRN in Table 3), where the semantics of '
minor comments (7)
  1. §4.4, Eqs. (6)–(7): The diversification loss requires threat labels during training. The paper acknowledges this in §4.4 (last paragraph) and §A4 (limitations), noting that new threat types require retraining. This is a reasonable design choice for the MAT setting, but the limitation could be stated more prominently in the main text rather than primarily in the appendix.
  2. Table 1: The MORE baseline [5] shows notably low performance (e.g., 31.1 union on CIFAR-10/PGD vs. 38.6 for RANDOM). Since MORE also uses MoE, a brief discussion of why MORE underperforms — beyond the general statement in §5.2 about 'gating without explicit threat-aware guidance' — would help readers understand the specific failure mode.
  3. §5.1: The perturbation budgets for ImageNet-100/1K list ϵ1 = 255, which appears to be a typo (likely should be 255/255 = 1.0 in [0,1] scale, or 255 in [0,255] scale). Please clarify the normalization convention.
  4. Fig. 5: The silhouette scores are reported as mean ± std for patch-level features but as single values for image-level features. For consistency, please report the same statistics for both.
  5. Table 4: The 'Gating classification' variant uses 3 experts (marked with ‡) while the main configuration uses 4. The footnote explains this is for fair comparison, but the difference in expert count makes the comparison less clean. Consider noting the 4-expert gating classification result as well, if available.
  6. §4.3, Eq. (5): The notation switches between g_i (final gating weight for token i) and g_{k,i} (weight for expert k) without explicit definition of the relationship. Please clarify.
  7. References [45, 49] are from 2025 and appear to be concurrent or very recent work on MoE and adversarial robustness. A brief discussion of how these relate to RoME would strengthen the related work section.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the careful reading and constructive feedback. The referee correctly identifies that our dual-scale gating motivation (Fig. 5, silhouette scores) and the corresponding ablation (Table 4) are conducted exclusively on ViT-B with RANDOM on CIFAR-10, and that the layer-adaptive coefficient beta encodes a fixed prior based on NLP Transformer layer hierarchy citations. We agree that verifying whether the feature-separability complementarity generalizes to ImageNet and CNN architectures (e.g., WRN) would strengthen the paper. We will add this analysis to the revision.

read point-by-point responses

  1. Referee: §4.3, Fig. 5, and Table 4: The dual-scale gating mechanism is motivated by the observation (Fig. 5) that ℓ1 threats are more separable at patch-level features (silhouette 0.394) while ℓ∞ threats are more separable at image-level features (silhouette 0.611). This observation is made exclusively on ViT-B trained with RANDOM on CIFAR-10. The layer-adaptive coefficient β^(l) in Eq. (5) encodes a fixed prior about Transformer layer hierarchy (global in early layers, local in deep layers) based on NLP citations [20, 37], and the ablation isolating dual-scale gating (Table 4: 41.1 → 39.4 local-only, 39.8 global-only) is also CIFAR-10 only. While the method achieves consistent gains across datasets and architectures (Tables 1, 3, A1, A2), the paper does not verify whether the complementarity in feature separability holds on ImageNet or CNN architectures (WRN in Table 3), where the semantics of '

    Authors: The referee raises a valid point. The feature-separability analysis in Fig. 5 and the dual-scale gating ablation in Table 4 are indeed limited to ViT-B on CIFAR-10, and the layer-adaptive coefficient beta^(l) is encoded as a fixed prior based on established Transformer layer hierarchy properties [20, 37]. We acknowledge that the paper does not directly verify whether the complementarity in feature separability (local features better separating ℓ1, global features better separating ℓ∞) holds on ImageNet or on CNN architectures like WRN. We note the following mitigating points from the current manuscript: (1) The consistent robustness gains of RoME across architectures — including WRN-28-10, WRN-94-16 (Table 3), DeiT-B, Swin-B (Table A1), and XCiT-S (Table 3) — suggest that dual-scale gating provides useful discriminative signals beyond ViT-B on CIFAR-10. (2) For CNN architectures, we adapt the local/global feature extraction to convolutional feature maps (per-spatial-location and global average pooling, as described in Sec. 5.1), so the dual-scale concept transfers naturally. (3) Fig. A4 in the appendix provides additional separability analysis across multiple patch positions and layers, though still on ViT-B/CIFAR-10. However, the referee is correct that a direct verification of the separability complementarity on ImageNet and CNN backbones is missing. We will address this in the revision by: (a) adding silhouette score analysis on ImageNet-100 with ViT-B and on CIFAR-10 with WRN, and (b) discussing the applicability and limitations of the fixed beta^(l) prior for non-Transformer architectures. If the complementarity does not hold as strongly on some architectures, we will transparently report this and note that the consistent empirical gains may stem from the gatingdiv revision: no

Circularity Check

0 steps flagged

No circularity found: derivation is self-contained and validated against external benchmarks

full rationale

The paper's three core components—low-rank experts (Sec. 4.2), dual-scale gating (Sec. 4.3), and threat-guided gating diversification (Sec. 4.4)—are architectural and training-time design choices, not fitted parameters later repackaged as predictions. The low-rank expert design is inspired by external work [69, Yang et al. CVPR 2024] and implemented via standard LoRA [24, Hu et al. ICLR 2022], with no self-citation in load-bearing positions. The dual-scale gating motivation (Fig. 5) is derived from a separately trained model (ViT-B with RANDOM, without any RoME components), so the observation is not circularly dependent on the method it motivates. The layer-adaptive coefficient β^(l) in Eq. 5 encodes a Transformer layer hierarchy prior citing external work [20, Geva et al.; 37, Liao et al.], not the authors' own prior results. The diversification loss (Eqs. 6–8) is a training regularizer that encourages diverse expert routing; it is not a parameter fitted to evaluation data and then 'predicted.' All quantitative claims are validated against external benchmarks (AutoAttack, OODRobustBench) and compared with independently developed baselines (RANDOM, MAX, MSD, E-AT, RAMP, MORE). No step in the derivation chain reduces to its own inputs by construction.

Axiom & Free-Parameter Ledger

7 free parameters · 4 axioms · 0 invented entities

RoME introduces no new physical entities, particles, or forces. The 'low-rank experts,' 'dual-scale gating,' and 'threat-guided gating diversification' are architectural components built from standard neural network primitives (LoRA matrices, MLPs, softmax, Euclidean distance). The GMA attack is a new evaluation procedure, not a postulated entity. All free parameters are hyperparameters selected via sweeps on CIFAR-10 and applied to other datasets without re-tuning, which is standard practice but means the method's performance depends on these choices transferring across settings.

free parameters (7)
  • K (number of experts) = 4
    Selected via hyperparameter sweep (Fig. A3a), chosen for best union robustness on CIFAR-10.
  • r (expert rank) = 16
    Selected via hyperparameter sweep (Fig. A3b).
  • lambda (diversification loss weight) = 0.1
    Selected via hyperparameter sweep (Fig. A3c).
  • s (layer-adaptive transition rate) = 4
    Selected via hyperparameter sweep (Fig. A3d).
  • b (layer-adaptive shift) = 2
    Selected via hyperparameter sweep (Fig. A3e).
  • K' (global gating projection dimension) = 100
    Stated in Sec. 5.1 without systematic sweep; used for computing diversification loss (Eq. 7).
  • Load balancing loss coefficient (for baseline comparison) = 0.5
    Set following existing protocol [64] for the conventional MoE baseline in Fig. 3.
axioms (4)
  • domain assumption Adversarial examples from different threat types share underlying image content but exhibit threat-specific perturbation patterns.
    Foundational premise for the paper's approach (Sec. 1, citing [26,78]). If threats did not share content, the shared backbone would be unnecessary; if they did not have threat-specific patterns, expert specialization would be impossible.
  • domain assumption Different threat types exhibit discriminative cues at different feature levels: l1 at local/patch-level, l_inf at global/image-level.
    Empirically observed on ViT-B/CIFAR-10 (Fig. 5) and used to justify dual-scale gating (Sec. 4.3). Generalization to other architectures/datasets is assumed but not extensively verified.
  • standard math Transformer layer hierarchy encodes local features in early layers and global semantic features in deeper layers.
    Invoked in Sec. 4.3 (citing [20,37]) to justify the fixed layer-adaptive coefficient beta^(l). These citations are from NLP; the property is assumed to transfer to vision Transformers.
  • domain assumption Threat labels are available during training to compute the diversification loss.
    The diversification loss (Eq. 6-7) requires grouping samples by threat type B_p. This is standard in MAT but limits applicability to scenarios with known, labeled threat types.

pith-pipeline@v1.1.0-glm · 26542 in / 3146 out tokens · 244302 ms · 2026-07-08T16:16:31.387806+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

79 extracted references · 79 canonical work pages · 6 internal anchors

  1. [1]

    In: NeurIPS (2021)

    Ali, A., Touvron, H., Caron, M., Bojanowski, P., Douze, M., Joulin, A., Laptev, I., Neverova, N., Synnaeve, G., Verbeek, J., et al.: Xcit: Cross-covariance image transformers. In: NeurIPS (2021)

    work page 2021

  2. [2]

    In: NeurIPS (2017)

    Brown, T.B., Mané, D., Roy, A., Abadi, M., Gilmer, J.: Adversarial patch. In: NeurIPS (2017)

    work page 2017

  3. [3]

    In: ICCV (2023)

    Chen, T., Chen, X., Du, X., Rashwan, A., Yang, F., Chen, H., Wang, Z., Li, Y.: Adamv-moe: Adaptive multi-task vision mixture-of-experts. In: ICCV (2023)

    work page 2023

  4. [4]

    In: CVPR (2023)

    Chen, Z., Shen, Y., Ding, M., Chen, Z., Zhao, H., Learned-Miller, E.G., Gan, C.: Mod-squad: Designing mixtures of experts as modular multi-task learners. In: CVPR (2023)

    work page 2023

  5. [5]

    Mixture of Robust Experts (MoRE):A Robust Denoising Method towards multiple perturbations

    Cheng, H., Xu, K., Wang, C., Kailkhura, B., Lin, X., Goldhahn, R.: Mixture of robust experts (more): A robust denoising method towards multiple perturbations. arXiv preprint arXiv:2104.10586 (2021)

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  6. [6]

    Chun-HsiaoYeh,Y.C.:IN100pytorch:Pytorchimplementation:Trainingresnetson imagenet-100.https://github.com/danielchyeh/ImageNet-100-Pytorch(2022), accessed: 2026-03-04

    work page 2022

  7. [7]

    In: NeurIPS Datasets and Benchmarks Track (2021)

    Croce, F., Andriushchenko, M., Sehwag, V., Debenedetti, E., Flammarion, N., Chi- ang, M., Mittal, P., Hein, M.: Robustbench: a standardized adversarial robustness benchmark. In: NeurIPS Datasets and Benchmarks Track (2021)

    work page 2021

  8. [8]

    In: ICML (2020)

    Croce, F., Hein, M.: Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In: ICML (2020)

    work page 2020

  9. [9]

    In: ICML (2022)

    Croce, F., Hein, M.: Adversarial robustness against multiple and singlel_p-threat models via quick fine-tuning of robust classifiers. In: ICML (2022)

    work page 2022

  10. [10]

    In: CVPR (2023)

    Croce, F., Rebuffi, S.A., Shelhamer, E., Gowal, S.: Seasoning model soups for robustness to adversarial and natural distribution shifts. In: CVPR (2023)

    work page 2023

  11. [11]

    DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models

    Dai, D., Deng, C., Zhao, C., Xu, R., Gao, H., Chen, D., Li, J., Zeng, W., Yu, X., Wu, Y., et al.: Deepseekmoe: Towards ultimate expert specialization in mixture- of-experts language models. arXiv preprint arXiv:2401.06066 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  12. [12]

    In: ICML (2025)

    Dai, S., Cianfarani, C., Bhagoji, A., Sehwag, V., Mittal, P.: Adapting to evolving adversaries with regularized continual robust training. In: ICML (2025)

    work page 2025

  13. [13]

    In: NeurIPS (2022)

    Dai, S., Mahloujifar, S., Mittal, P.: Formulating robustness against unforeseen at- tacks. In: NeurIPS (2022)

    work page 2022

  14. [14]

    In: CVPR (2009)

    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: CVPR (2009)

    work page 2009

  15. [15]

    In: ICLR (2021)

    Dosovitskiy, A.: An image is worth 16x16 words: Transformers for image recogni- tion at scale. In: ICLR (2021)

    work page 2021

  16. [16]

    In: ACL (2024)

    Dou, S., Zhou, E., Liu, Y., Gao, S., Shen, W., Xiong, L., Zhou, Y., Wang, X., Xi, Z., Fan, X., et al.: Loramoe: Alleviating world knowledge forgetting in large language models via moe-style plugin. In: ACL (2024)

    work page 2024

  17. [17]

    In: NeurIPS (2022)

    Fan, Z., Sarkar, R., Jiang, Z., Chen, T., Zou, K., Cheng, Y., Hao, C., Wang, Z., et al.: M3vit: Mixture-of-experts vision transformer for efficient multi-task learning with model-accelerator co-design. In: NeurIPS (2022)

    work page 2022

  18. [18]

    Journal of Machine Learning Research (2022)

    Fedus,W.,Zoph,B.,Shazeer,N.:Switchtransformers:Scalingtotrillionparameter models with simple and efficient sparsity. Journal of Machine Learning Research (2022)

    work page 2022

  19. [19]

    In: NAACL (2025) RoME: Robust Mixture of Low-Rank Experts 17

    Gao, C., Chen, K., Rao, J., Liu, R., Sun, B., Zhang, Y., Peng, D., Guo, X., Subrah- manian, V.: Mola: Moe lora with layer-wise expert allocation. In: NAACL (2025) RoME: Robust Mixture of Low-Rank Experts 17

    work page 2025

  20. [20]

    In: EMNLP (2021)

    Geva, M., Schuster, R., Berant, J., Levy, O.: Transformer feed-forward layers are key-value memories. In: EMNLP (2021)

    work page 2021

  21. [21]

    In: ICLR (2015)

    Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples. In: ICLR (2015)

    work page 2015

  22. [22]

    In: ICLR (2019)

    Hendrycks,D.,Dietterich,T.:Benchmarkingneuralnetworkrobustnesstocommon corruptions and perturbations. In: ICLR (2019)

    work page 2019

  23. [23]

    In: ICML (2019)

    Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Ges- mundo, A., Attariyan, M., Gelly, S.: Parameter-efficient transfer learning for nlp. In: ICML (2019)

    work page 2019

  24. [24]

    In: ICLR (2022)

    Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al.: Lora: Low-rank adaptation of large language models. In: ICLR (2022)

    work page 2022

  25. [25]

    In: COLM (2023)

    Huang, C., Liu, Q., Lin, B.Y., Pang, T., Du, C., Lin, M.: Lorahub: Efficient cross- task generalization via dynamic lora composition. In: COLM (2023)

    work page 2023

  26. [26]

    In: NeurIPS (2019)

    Ilyas, A., Santurkar, S., Tsipras, D., Engstrom, L., Tran, B., Madry, A.: Adversarial examples are not bugs, they are features. In: NeurIPS (2019)

    work page 2019

  27. [27]

    Neural computation (1991)

    Jacobs, R.A., Jordan, M.I., Nowlan, S.J., Hinton, G.E.: Adaptive mixtures of local experts. Neural computation (1991)

    work page 1991

  28. [28]

    In: NeurIPS (2024)

    Jiang, E., Singh, G.: Ramp: Boosting adversarial robustness against multiplel_p perturbations for universal robustness. In: NeurIPS (2024)

    work page 2024

  29. [29]

    PMoE: Progressive Mixture of Experts with Asymmetric Transformer for Continual Learning

    Jung, M.J., Kim, J.: Pmoe: Progressive mixture of experts with asymmetric trans- former for continual learning. arXiv preprint arXiv:2407.21571 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  30. [30]

    Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images (2009)

    work page 2009

  31. [31]

    In: NeurIPS (2012)

    Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep con- volutional neural networks. In: NeurIPS (2012)

    work page 2012

  32. [32]

    In: NeurIPS (2019)

    Laidlaw, C., Feizi, S.: Functional adversarial attacks. In: NeurIPS (2019)

    work page 2019

  33. [33]

    In: ICLR (2021)

    Laidlaw, C., Singla, S., Feizi, S.: Perceptual adversarial robustness: Defense against unseen threat models. In: ICLR (2021)

    work page 2021

  34. [34]

    In: ICLR (2021)

    Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang, Y., Krikun, M., Shazeer, N., Chen, Z.: Gshard: Scaling giant models with conditional computation and automatic sharding. In: ICLR (2021)

    work page 2021

  35. [35]

    In: ICLR (2025)

    Li, H., Lin, S., Duan, L., Liang, Y., Shroff, N.B.: Theory on mixture-of-experts in continual learning. In: ICLR (2025)

    work page 2025

  36. [36]

    In: ICML (2024)

    Li, L., Wang, Y., Sitawarin, C., Spratling, M.: Oodrobustbench: a benchmark and large-scale analysis of adversarial robustness under distribution shift. In: ICML (2024)

    work page 2024

  37. [37]

    In: ICLR (2025)

    Liao, M., Chen, W., Shen, J., Guo, S., Wan, H.: Hmora: Making llms more effective with hierarchical mixture of lora experts. In: ICLR (2025)

    work page 2025

  38. [38]

    In: ICCV (2021)

    Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin trans- former: Hierarchical vision transformer using shifted windows. In: ICCV (2021)

    work page 2021

  39. [39]

    In: ICLR (2019)

    Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: ICLR (2019)

    work page 2019

  40. [40]

    In: KDD (2018)

    Ma, J., Zhao, Z., Yi, X., Chen, J., Hong, L., Chi, E.H.: Modeling task relationships in multi-task learning with multi-gate mixture-of-experts. In: KDD (2018)

    work page 2018

  41. [41]

    In: JMLR (2008)

    Maaten, L.v.d., Hinton, G.: Visualizing data using t-sne. In: JMLR (2008)

    work page 2008

  42. [42]

    In: ICML (2021)

    Madaan, D., Shin, J., Hwang, S.J.: Learning to generate noise for multi-attack robustness. In: ICML (2021)

    work page 2021

  43. [43]

    In: ICLR (2018)

    Madry, A., Makelov, A., Schmidt, L., Tsipras, D., Vladu, A.: Towards deep learning models resistant to adversarial attacks. In: ICLR (2018)

    work page 2018

  44. [44]

    In: ICML (2020) 18 W

    Maini,P.,Wong,E.,Kolter,Z.:Adversarialrobustnessagainsttheunionofmultiple perturbation models. In: ICML (2020) 18 W. Kimet al

    work page 2020

  45. [45]

    arXiv preprint arXiv:2512.20821 (2025)

    Meymani, M., Razavi-Far, R.: Defending against adversarial attacks using mixture of experts. arXiv preprint arXiv:2512.20821 (2025)

    work page arXiv 2025

  46. [46]

    In: CVPR (2019)

    Modas, A., Moosavi-Dezfooli, S.M., Frossard, P.: Sparsefool: a few pixels make a big difference. In: CVPR (2019)

    work page 2019

  47. [47]

    arXiv preprint arXiv:2503.07137 (2025)

    Mu, S., Lin, S.: A comprehensive survey of mixture-of-experts: Algorithms, theory, and applications. arXiv preprint arXiv:2503.07137 (2025)

    work page arXiv 2025

  48. [48]

    In: 2024 Interna- tional Conference on Machine Learning and Applications (ICMLA) (2024)

    Pavlitska, S., Eisen, E., Zöllner, J.M.: Towards adversarial robustness of model- level mixture-of-experts architectures for semantic segmentation. In: 2024 Interna- tional Conference on Machine Learning and Applications (ICMLA) (2024)

    work page 2024

  49. [49]

    In: ICCV (2025)

    Pavlitska, S., Fan, H., Ditschuneit, K., Zöllner, J.M.: Robust experts: the effect of adversarial training on cnns with sparse mixture-of-experts layers. In: ICCV (2025)

    work page 2025

  50. [50]

    In: NeurIPS (2022)

    Puigcerver, J., Jenatton, R., Riquelme, C., Awasthi, P., Bhojanapalli, S.: On the adversarial robustness of mixture of experts. In: NeurIPS (2022)

    work page 2022

  51. [51]

    In: ICLR (2024)

    Puigcerver,J.,Riquelme,C.,Mustafa,B.,Houlsby,N.:Fromsparsetosoftmixtures of experts. In: ICLR (2024)

    work page 2024

  52. [52]

    In: NeurIPS (2021)

    Riquelme, C., Puigcerver, J., Mustafa, B., Neumann, M., Jenatton, R., Su- sano Pinto, A., Keysers, D., Houlsby, N.: Scaling vision with sparse mixture of experts. In: NeurIPS (2021)

    work page 2021

  53. [53]

    Journal of computational and applied mathematics (1987)

    Rousseeuw, P.J.: Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. Journal of computational and applied mathematics (1987)

    work page 1987

  54. [54]

    Progressive Neural Networks

    Rusu, A.A., Rabinowitz, N.C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., Hadsell, R.: Progressive neural networks. arXiv preprint arXiv:1606.04671 (2016)

    work page internal anchor Pith review Pith/arXiv arXiv 2016

  55. [55]

    Shafahi, A., Najibi, M., Ghiasi, M.A., Xu, Z., Dickerson, J., Studer, C., Davis, L.S., Taylor, G., Goldstein, T.: Adversarial training for free! In: NeurIPS (2019)

    work page 2019

  56. [56]

    In: ICLR (2017)

    Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., Dean, J.: Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In: ICLR (2017)

    work page 2017

  57. [57]

    In: NeurIPS (2023)

    Singh, N.D., Croce, F., Hein, M.: Revisiting adversarial training for imagenet: Architectures, training and generalization across threat models. In: NeurIPS (2023)

    work page 2023

  58. [58]

    In: ICML (2021)

    Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., Jégou, H.: Training data-efficient image transformers & distillation through attention. In: ICML (2021)

    work page 2021

  59. [59]

    In: NeurIPS (2019)

    Tramer, F., Boneh, D.: Adversarial training and robustness for multiple perturba- tions. In: NeurIPS (2019)

    work page 2019

  60. [60]

    In: ICLR (2020)

    Wang, Y., Zou, D., Yi, J., Bailey, J., Ma, X., Gu, Q.: Improving adversarial ro- bustness requires revisiting misclassified examples. In: ICLR (2020)

    work page 2020

  61. [61]

    In: CVPR (2024)

    Wang, Z., Li, X., Zhu, H., Xie, C.: Revisiting adversarial training at scale. In: CVPR (2024)

    work page 2024

  62. [62]

    Wightman, R.: Pytorch image models.https://github.com/rwightman/pytorch- image-models(2019).https://doi.org/10.5281/zenodo.4414861

    work page doi:10.5281/zenodo.4414861 2019

  63. [63]

    In: ICLR (2020)

    Wong, E., Rice, L., Kolter, J.Z.: Fast is better than free: Revisiting adversarial training. In: ICLR (2020)

    work page 2020

  64. [64]

    In: ICLR (2024)

    Wu, X., Huang, S., Wei, F.: Mixture of lora experts. In: ICLR (2024)

    work page 2024

  65. [65]

    In: ICLR (2018)

    Xiao, C., Zhu, J.Y., Li, B., He, W., Liu, M., Song, D.: Spatially transformed ad- versarial examples. In: ICLR (2018)

    work page 2018

  66. [66]

    In: ICLR (2020)

    Xie, C., Yuille, A.: Intriguing properties of adversarial training at scale. In: ICLR (2020)

    work page 2020

  67. [67]

    Qwen3 Technical Report

    Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al.: Qwen3 technical report. arXiv preprint arXiv:2505.09388 (2025) RoME: Robust Mixture of Low-Rank Experts 19

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  68. [68]

    Leveraging Open Knowledge for Advancing Task Expertise in Large Language Models

    Yang, Y., Qin, Y., Wu, T., Xu, Z., Li, G., Guo, P., Shao, H., Shi, Y., Li, K., Sun, X., et al.: Leveraging open knowledge for advancing task expertise in large language models. arXiv preprint arXiv:2408.15915 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  69. [69]

    In: CVPR (2024)

    Yang, Y., Jiang, P.T., Hou, Q., Zhang, H., Chen, J., Li, B.: Multi-task dense prediction via mixture of low-rank experts. In: CVPR (2024)

    work page 2024

  70. [70]

    In: CVPR (2024)

    Yu, J., Zhuge, Y., Zhang, L., Hu, P., Wang, D., Lu, H., He, Y.: Boosting contin- ual learning of vision-language models via mixture-of-experts adapters. In: CVPR (2024)

    work page 2024

  71. [71]

    In: BMVC (2016)

    Zagoruyko, S., Komodakis, N.: Wide residual networks. In: BMVC (2016)

    work page 2016

  72. [72]

    In: ICML (2019)

    Zhang, H., Yu, Y., Jiao, J., Xing, E., El Ghaoui, L., Jordan, M.: Theoretically principled trade-off between robustness and accuracy. In: ICML (2019)

    work page 2019

  73. [73]

    In: ICML (2025)

    Zhang, X., Xu, K., Hu, Z., Wang, R.: Optimizing robustness and accuracy in mixture of experts: A dual-model approach. In: ICML (2025)

    work page 2025

  74. [74]

    In: ICCV (2023)

    Zhang, Y., Cai, R., Chen, T., Zhang, G., Zhang, H., Chen, P.Y., Chang, S., Wang, Z., Liu, S.: Robust mixture-of-expert training for convolutional neural networks. In: ICCV (2023)

    work page 2023

  75. [75]

    In: ICLR (2024)

    Zhao, X., Chen, X., Cheng, Y., Chen, T.: Sparse moe with language guided routing for multilingual machine translation. In: ICLR (2024)

    work page 2024

  76. [76]

    In: AAAI (2025)

    Zhou, H., Wang, Z., Huang, S., Huang, X., Han, X., Feng, J., Deng, C., Luo, W., Chen, J.: Moe-lpr: Multilingual extension of large language models through mixture-of-experts with language priors routing. In: AAAI (2025)

    work page 2025

  77. [77]

    In: NeurIPS (2022)

    Zhou, Y., Lei, T., Liu, H., Du, N., Huang, Y., Zhao, V., Dai, A.M., Le, Q.V., Laudon, J., et al.: Mixture-of-experts with expert choice routing. In: NeurIPS (2022)

    work page 2022

  78. [78]

    Nature commu- nications (2019) RoME: Robust Mixture of Low-Rank Experts 1 In this appendix, we first elaborate on additional implementation details (Sec

    Zhou, Z., Firestone, C.: Humans can decipher adversarial images. Nature commu- nications (2019) RoME: Robust Mixture of Low-Rank Experts 1 In this appendix, we first elaborate on additional implementation details (Sec. A1). We report results of applying our RoME to other Transformer-based visionmodelsandImageNet-1K(Sec.A2).Wealsoprovideadditionalanalysiso...

    work page 2019

  79. [79]

    For semantic ReColorAdv [32] attack, we use bound of0.06with20iterations

    For spatial StAdv [65] attack, we set flow regularization loss weightτto0.05. For semantic ReColorAdv [32] attack, we use bound of0.06with20iterations. For gating misrouting attack (GMA), we combine the original classification loss with a loss that minimizes cross-entropy loss between gating outputs and least-likely expert with the same weight for each lo...

    work page