Pith. sign in

REVIEW 2 major objections 1 minor 15 references

CLEAR-MoE: Shared-Basis Expert Extraction from Frozen Vision Transformers via Calibration-Driven Layer Selection

T0 review · 2 major / 1 minor · reviewed 2026-06-30 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The shared low-rank SVD basis, not the routing mechanism, preserves accuracy when turning frozen Vision Transformers into Mixture-of-Experts models.

desk verdict CLEAR-MoE shows a post-training SVD-plus-clustering route to turn frozen ViTs into MoE models while keeping accuracy on the calibration set, with the shared basis doing most of the work. read the letter →

arxiv 2606.28516 v1 pith:K5HCITCH submitted 2026-06-26 cs.CV cs.DC

Md Irtiza Hossain , Humaira Ayesha , Junaid Ahmed Sifat This is my paper · ORCID
classification cs.CVcs.DC
keywords Mixture-of-ExpertsVisionTransformerpost-trainingconversionSVDdecompositionlayerselectionfrozenmodelsparsification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CLEAR-MoE is a post-training pipeline that converts a frozen Vision Transformer into a sparse Mixture-of-Experts model by scoring FFN layers on sparsity, clusterability, and output sensitivity, then decomposing selected layers into a shared low-rank SVD basis plus per-cluster residual experts via k-means. Lightweight routers are trained on the resulting cluster labels and tokens are dispatched through pluggable CUDA kernels. The central empirical result is that this shared SVD basis accounts for accuracy retention, since random routing, learned routing, and multiple router designs all yield performance within 0.06 percentage points of one another.

What carries the argument

Shared low-rank SVD basis extracted from selected FFN weight matrices, paired with k-means-derived per-cluster residual experts.

What would settle it

If accuracy on a new image-classification dataset drops more than 0.5 percentage points below the dense baseline when the shared-basis model is paired with random routing, the claim that the basis alone preserves behavior would be falsified.

Watch

Extended reading notes

Core claim

The shared SVD basis is the primary factor responsible for preserving accuracy. Random routing, learned routing, and three different router architectures produce nearly identical performance, with accuracy varying by at most 0.06 percentage points (86.62%-86.68%). This behavior generalizes across five ViT backbones covering 5.7M to 86.6M parameters, with accuracy differences at most 0.10 percentage points from the dense models, and remains stable under changes in SVD rank, expert count, calibration set size, and random seed.

Load-bearing premise

That the three layer-scoring criteria reliably identify FFN layers whose decomposition into a shared low-rank SVD basis plus per-cluster residuals will preserve the original model's behavior on unseen inputs and tasks beyond the calibration set.

Editorial extensions

If this is right

  • The converted model retains 99.9 percent of the dense accuracy on Imagenette with DeiT-Small.
  • Accuracy stays stable for expert counts from 2 to 8 and across calibration sets of 50 to 500 samples.
  • The same accuracy retention holds for all tested ViT sizes from 5.7M to 86.6M parameters.
  • The MoE FFN runs 1.3-1.7 times slower than the dense version on a GTX 960 GPU because dispatch is memory-bound.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The low-rank structure uncovered by the SVD step may be a general property of many transformer FFN layers.
  • Future speedups would require fused kernels that reduce the memory-bound routing overhead identified in the dispatch microbenchmark.
  • The finding that routing choice is secondary could motivate simpler expert-selection schemes in other post-training conversion pipelines.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 1 minor

Summary. The manuscript presents CLEAR-MoE, a four-phase post-training pipeline that converts a frozen pretrained Vision Transformer into a sparse MoE model without updating backbone weights. FFN layers are scored by sparsity, clusterability, and output sensitivity; selected layers are decomposed into a shared low-rank SVD basis plus per-cluster residual experts via k-means; lightweight routers are trained on cluster labels; and tokens are dispatched through pluggable CUDA kernels. On Imagenette the method retains 99.9% of dense accuracy for DeiT-Small (86.70% vs 86.73%) and comparable retention across five backbones (5.7M–86.6M parameters), with ablations showing that accuracy varies by at most 0.06 pp across random, learned, and alternative routers and remains stable across SVD ranks, expert counts, and calibration sizes.

Significance. If the central empirical finding holds, the work supplies a practical route to sparse ViT inference that avoids any backbone fine-tuning and isolates the shared low-rank SVD basis as the dominant accuracy-preserving component. The reported routing-insensitivity and hyperparameter stability constitute a concrete, falsifiable observation that could guide future MoE design. The multi-backbone coverage and explicit timing measurements are additional strengths.

major comments (2)
  1. [Abstract] Abstract: The claim that the shared SVD basis is the primary factor responsible for accuracy preservation rests on the observation that random routing, learned routing, and three router architectures produce nearly identical results (variation ≤ 0.06 pp). All such ablations, however, are performed exclusively on the Imagenette calibration distribution; without OOD or cross-task results it remains untested whether the decomposition (and the three scoring criteria used to select layers) generalizes beyond calibration statistics.
  2. [Abstract] Abstract: The three layer-scoring criteria (sparsity, clusterability, output sensitivity) and the k-means clustering step are both computed on the calibration set, yet the manuscript supplies no explicit formulas, pseudocode, or protocol for these quantities. This omission directly affects reproducibility of the layer-selection step that underpins the entire pipeline and the claim that the resulting experts preserve original behavior on unseen inputs.
minor comments (1)
  1. [Abstract] Abstract: The reported standard deviation (±0.02%) implies repeated runs, but the number of seeds, exact statistical test, and whether the same calibration set was reused across runs are not stated.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive feedback. We address the two major comments point by point below. We agree that explicit formulas and pseudocode are required for reproducibility and will add them. On the generalization point, we maintain that the current experiments are scoped to the calibration distribution and multi-backbone evaluation, which is consistent with the manuscript's claims.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The claim that the shared SVD basis is the primary factor responsible for accuracy preservation rests on the observation that random routing, learned routing, and three router architectures produce nearly identical results (variation ≤ 0.06 pp). All such ablations, however, are performed exclusively on the Imagenette calibration distribution; without OOD or cross-task results it remains untested whether the decomposition (and the three scoring criteria used to select layers) generalizes beyond calibration statistics.

    Authors: The ablations on routing variants were intentionally performed on the Imagenette calibration set to isolate the contribution of the shared SVD basis versus routing. The manuscript's central empirical claim is that, within this distribution, the shared basis dominates accuracy preservation, as evidenced by the ≤0.06 pp variation. We do not claim or test generalization to OOD or cross-task settings in the current work; the multi-backbone results (DeiT-Tiny through ViT-Base) demonstrate consistency of the pipeline across model scales but remain within the same evaluation protocol. We therefore do not plan to expand the scope with new OOD experiments for this revision. revision: no

  2. Referee: [Abstract] Abstract: The three layer-scoring criteria (sparsity, clusterability, output sensitivity) and the k-means clustering step are both computed on the calibration set, yet the manuscript supplies no explicit formulas, pseudocode, or protocol for these quantities. This omission directly affects reproducibility of the layer-selection step that underpins the entire pipeline and the claim that the resulting experts preserve original behavior on unseen inputs.

    Authors: We agree that the absence of explicit definitions and pseudocode for the three scoring criteria and the clustering procedure limits reproducibility. In the revised manuscript we will add: (i) the precise mathematical definitions of sparsity (fraction of near-zero activations), clusterability (silhouette score on activation vectors), and output sensitivity (L2 norm of output change under perturbation); (ii) the exact protocol for computing these scores on the calibration set; and (iii) pseudocode for the layer-selection and k-means decomposition steps. These additions will be placed in the Methods section and an appendix. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity; purely empirical pipeline with direct held-out measurements.

full rationale

The manuscript contains no mathematical derivations, uniqueness theorems, or first-principles predictions. It presents a four-phase empirical pipeline (layer scoring, SVD decomposition + k-means, router training, dispatch) and reports accuracy numbers plus ablations measured on held-out Imagenette data. The key observation that random routing matches learned routing is a direct empirical comparison on the same test split, not a fitted parameter renamed as a prediction. No self-citation chains, ansatzes smuggled via prior work, or self-definitional reductions appear. The work is therefore self-contained against its external benchmarks.

Assumptions & free parameters 0 free parameters · 0 assumptions · 0 invented entities

The approach is driven by empirical calibration and clustering rather than explicit axioms or fitted parameters; SVD rank and expert count are varied experimentally but not optimized against a loss that would create circularity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CLEAR-MoE: Shared-Basis Expert Extraction from Frozen Vision Transformers via Calibration-Driven Layer Selection." pith.science (2026). https://pith.science/paper/K5HCITCH

@misc{pith2026260628516,
  author       = {Pith},
  title        = {Pith review of: CLEAR-MoE: Shared-Basis Expert Extraction from Frozen Vision Transformers via Calibration-Driven Layer Selection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/K5HCITCH}},
  note         = {Machine review of arXiv:2606.28516}
}
read the original abstract

We present CLEAR-MoE, a four-phase post-training pipeline that converts a frozen pretrained Vision Transformer (ViT) into a sparse Mixture-of-Experts (MoE) model without updating backbone weights. The pipeline (i) scores feed-forward network (FFN) layers by sparsity, clusterability, and output sensitivity; (ii) decomposes selected layers into a shared low-rank SVD basis and per-cluster residual experts using k-means clustering; (iii) trains lightweight routers supervised by cluster labels; and (iv) dispatches tokens through pluggable CUDA backends. On Imagenette with DeiT-Small, CLEAR-MoE retains 99.9% of the dense model's accuracy (86.70 +/- 0.02% versus 86.73%). Extensive ablation studies reveal a consistent empirical finding: the shared SVD basis is the primary factor responsible for preserving accuracy. Random routing, learned routing, and three different router architectures produce nearly identical performance, with accuracy varying by at most 0.06 percentage points (86.62%-86.68%). Accuracy also remains stable across different SVD ranks, expert counts (2-8), calibration set sizes (50-500), and random seeds. This behavior generalizes across five ViT backbones (DeiT-Tiny, DeiT-Small, DeiT-Base, ViT-Small, and ViT-Base), covering models from 5.7M to 86.6M parameters, with accuracy differences <= 0.10 percentage points from their dense counterparts. On a GTX 960 GPU, routing and scatter-gather overhead make the CLEAR-MoE FFN 1.3-1.7x slower than the dense implementation. A dispatch microbenchmark further shows that routing is an order of magnitude more memory-bound than expert matrix multiplications, identifying fused dispatch kernels as a promising direction for future optimization.

Figures

Figures reproduced from arXiv: 2606.28516 by the authors.

Figure 1
Figure 1. CLEAR-MoE pipeline. A frozen pretrained ViT yields calibration [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Roofline for GTX 960. Blue = compute-bound; orange/red = memory [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Mean per-token routing entropy (nats) per expertized layer. Dashed [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Accuracy gap (∆pp relative to dense) vs. router training across 13 configurations. D5 (random, no trained router) at x=0; D6 and cross-backbone routers at measured routing accuracy. No strong monotonic relationship is visible, consistent with the hypothesis that shared…
Figure 5
Figure 5. Figure 5: Accuracy ∆ vs. dense for five ViT backbones under D5 (random) and D6 (learned linear) routing (Imagenette, E=4, N=200, seed 42). All deltas within |∆|≤0.10 pp. D6 provides marginal numerical advantage in 4 of 5 backbones; D5 outperforms D6 only on ViT-B (highest load s…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 3 canonical work pages

  1. [1]

    An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,”CoRR, vol. abs/2010.11929, 2020. [Online]. Available: https://arxiv.org/abs/2010.11929

    work page Pith review arXiv 2010

  2. [2]

    Scaling vision with sparse mixture of experts,

    C. Riquelme, J. Puigcerver, B. Mustafa, M. Neumann, R. Jenatton, A. Pinto, D. Keysers, and N. Houlsby, “Scaling vision with sparse mixture of experts,” 06 2021

    2021

  3. [3]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,

    G. Hinton, N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, and J. Dean, “Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,” 01 2017

    2017

  4. [4]

    Adamv-moe: Adaptive multi-task vision mixture-of-experts,

    T. Chen, X. Chen, X. Du, A. Rashwan, F. Yang, H. Chen, Z. Wang, and Y . Li, “Adamv-moe: Adaptive multi-task vision mixture-of-experts,” 10 2023, pp. 17 300–17 311

    2023

  5. [5]

    Moefication: Transformer feed-forward layers are mixtures of experts,

    Z. Zhang, Y . Lin, Z. Liu, P. Li, M. Sun, and J. Zhou, “Moefication: Transformer feed-forward layers are mixtures of experts,” 01 2022, pp. 877–890

    2022

  6. [6]

    Exploiting activation sparsity with dense to dynamic-k mixture-of- experts conversion,

    F. Szatkowski, B. W’ojcik, M. Pi’orczy’nski, and S. Scardapane, “Exploiting activation sparsity with dense to dynamic-k mixture-of- experts conversion,”Advances in Neural Information Processing Systems 37, 2023. [Online]. Available: https://api.semanticscholar.org/CorpusID: 263831552

    2023

  7. [7]

    Efficient data driven mixture-of-expert extraction from trained networks,

    U. Berisha, J. Mehnert, and A. P. Condurache, “Efficient data driven mixture-of-expert extraction from trained networks,” 2025. [Online]. Available: https://arxiv.org/abs/2505.15414

    work page arXiv 2025

  8. [8]

    Sparse upcycling: Training mixture-of-experts from dense checkpoints,

    A. Komatsuzaki, J. Puigcerver, J. Lee-Thorp, C. Ruiz, B. Mustafa, J. Ainslie, Y . Tay, M. dehghani, and N. Houlsby, “Sparse upcycling: Training mixture-of-experts from dense checkpoints,” 12 2022

    2022

Show all 15 references
  1. [9]

    Dynamicvit: Efficient vision transformers with dynamic token sparsification,

    Y . Rao, W. Zhao, B. Liu, J. Lu, J. Zhou, and C.-J. Hsieh, “Dynamicvit: Efficient vision transformers with dynamic token sparsification,” 06 2021

    2021

  2. [10]

    A-vit: Adaptive tokens for efficient vision transformer,

    H. Yin, A. Vahdat, J. M. Alvarez, A. Mallya, J. Kautz, and P. Molchanov, “A-vit: Adaptive tokens for efficient vision transformer,” 06 2022, pp. 10 799–10 808

    2022

  3. [11]

    Tutel: Adaptive mixture-of-experts at scale,

    C. Hwang, W. Cui, Y . Xiong, Z. Yang, Z. Liu, H. Hu, Z. Wang, R. Salas, J. Jose, P. Ram, J. Chau, P. Cheng, F. Yang, M. Yang, and Y . Xiong, “Tutel: Adaptive mixture-of-experts at scale,” 06 2022

    2022

  4. [12]

    Optimizing dynamic neural networks with brainstorm optimizing dynamic neural networks with brainstorm,

    W. Cui, S. Jiao, T. University, Z. Han, M. Research, Y . Wang, N. Zheng, L. Ma, Y . Yang, F. Yang, J. Xue, L. Qiu, L. Zhou, A. Quan, H. Tan, M. Guo, L. Ouyang, and Q. Chen, “Optimizing dynamic neural networks with brainstorm optimizing dynamic neural networks with brainstorm,” 07 2023

    2023

  5. [13]

    Lancet: Accelerating mixture-of-experts training via whole graph computation-communication overlapping,

    C. Jiang, Y . Tian, Z. Jia, S. Zheng, C. Wu, and Y . Wang, “Lancet: Accelerating mixture-of-experts training via whole graph computation-communication overlapping,”ArXiv, vol. abs/2404.19429,

  6. [14]

    Available: https://api.semanticscholar.org/CorpusID: 269456940

    [Online]. Available: https://api.semanticscholar.org/CorpusID: 269456940

  7. [15]

    Mixture-of-experts with expert choice routing,

    Y . Zhou, T. Lei, H. Liu, N. Du, Y . Huang, V . Zhao, A. Dai, Z. Chen, Q. Le, and J. Laudon, “Mixture-of-experts with expert choice routing,” 02 2022. APPENDIXA METHODOLOGYDETAILS A. Glossary of Key Terms Table A1 defines domain-specific terminology used through- out the paper...

    2022

Pith tools

Reviewed June 30, 2026 · model on record in the stance chip above.