REVIEW 2 major objections 5 minor 26 references
MoGA: Searching Beyond MobileNetV3
T0 review · 2 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read GPU-aware architecture search yields mobile models that beat MobileNetV3 on ImageNet
desk verdict A credible GPU-aware mobile NAS pipeline with real searched models, but the missing supernet-to-standalone rank check means the claimed gains over MobileNetV3 are not yet fully established. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is the weighted NSGA-II loop with a three-objective fitness: maximize accuracy, minimize mobile-GPU latency, and maximize parameter count. Each candidate's latency is read from a layerwise lookup table built by benchmarking the 12 possible block choices per layer once on the GPU, so latency is the sum of per-layer entries; each candidate's accuracy comes from a one-shot supernet trained with a strict-fairness sampling strategy, which the paper relies on to rank candidates stably. The crowding distance is weighted by preferences $w_{\mathrm{acc}}=w_{\mathrm{lat}}=0.4$, $w_{\mathrm{params}}=0.2$, so accuracy and latency dominate but parameter count still steers the search. The distinctive third objective rests on the observation that mobile-scale models are underfitted rather than overfitted, so enlarged parameter counts buy representational power without proportionally raising latency.
What would settle it
Train, say, fifty architectures randomly sampled from the $12^{14}$ search space, or from the final Pareto front, from scratch and compare their final top-1 accuracies with the supernet's validation scores; if the rank correlation is weak or negative, the supernet is not a faithful selector and the reported MoGA advantages are not attributable to the search.
Extended reading notes
Core claim
This paper claims that neural architecture search for mobile devices should optimize for the mobile GPU rather than the CPU, because CPU latency does not predict GPU latency: the authors measure 100 random models on both platforms and find no obvious linear correspondence. It builds a search over inverted-bottleneck blocks with $12^{14}$ possible architectures, scores candidates with a one-shot supernet trained under a fairness-enforced sampling scheme, predicts GPU latency by summing per-block times from a layerwise lookup table, and evolves models with weighted NSGA-II over three objectives: accuracy, latency, and number of parameters. The resulting MoGA-A, MoGA-B, and MoGA-C reach 75.9%, 75.5%, and 75.3% top-1 ImageNet accuracy under mobile-GPU latency budgets comparable to MobileNetV3's, at 75.0% in the authors' reproduction, and the whole search costs about 12 GPU days, roughly 200 times fewer than MnasNet. MoGA-C in particular is slower on CPU but faster on GPU than MobileNetV3, which the paper presents as direct evidence that the optimization target changes the architecture.
Load-bearing premise
The search's accuracy signal is a one-shot supernet's validation score, so the pipeline assumes that score ranks candidate architectures in the same order as training each candidate from scratch would.
Editorial extensions
If this is right
- Optimizing for mobile GPU latency rather than CPU latency changes the searched architectures: MoGA-C is slower on CPU but faster on GPU than MobileNetV3, so hardware choice is a first-class design axis rather than a measurement detail.
- Adding parameter count as a maximization objective, with lower weight than accuracy and latency, shifts the Pareto front toward models with higher capacity and better accuracy within the same latency budget.
- A one-shot supernet plus a layerwise latency lookup table makes the search cost about 12 GPU days, and re-targeting to another mobile device only requires a refreshed latency table rather than retraining the supernet.
- The searched models give concrete accuracy/latency points above MobileNetV3 on ImageNet: 75.9% for MoGA-A, 75.5% for MoGA-B, and 75.3% for MoGA-C.
Reading between the lines
- If the CPU-versus-GPU latency divergence holds beyond the phones tested, earlier CPU-aware NAS results should be re-examined on GPU Pareto fronts; architectures that look bad by CPU latency may be competitive when GPU latency is the constraint.
- The parameter-maximizing objective is tailored to the underfitting regime of small mobile models; on larger datasets, harder distillation targets, or much larger capacity budgets, the sign of that objective would likely need to flip.
- The layerwise latency sum assumes per-operator costs add independently; operator fusion, framework updates, or thermal throttling would violate additivity, so the reported latency numbers are tied to the specific runtime and device measured.
- The paper does not report the rank correlation between supernet validation accuracy and stand-alone trained accuracy for the searched population; that correlation is the quantity that would determine whether the search's selections, rather than the training recipe, are responsible for the final accuracies.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Mobile GPU-Aware Neural Architecture Search (MoGA), a one-shot NAS method that targets mobile GPU latency rather than CPU latency. The search uses a FairNAS-style supernet as a fast accuracy evaluator, a layerwise latency lookup table for mobile GPU (MACE) latency, and a weighted NSGA-II with three objectives: accuracy, latency, and number of parameters. The authors report three searched models, MoGA-A/B/C, achieving 75.9/75.5/75.3% ImageNet top-1 accuracy, which they claim outperform MobileNetV3-large under similar mobile GPU latency, with a total search cost of 12 GPU days, about 200x less than MnasNet. The paper also presents ablations on the weighted objective and the hierarchical mutator, and discusses the GPU/CPU latency discrepancy.
Significance. If the results hold, MoGA provides evidence that hardware-aware NAS should target the actual deployment hardware (mobile GPU) rather than CPU, and that adding a parameter-maximization objective can improve accuracy without increasing latency. The public release of models and test code is a strength. However, the central claim that the searched models outperform MobileNetV3 depends on the supernet's validation accuracy being a reliable ranking signal for stand-alone trained models, and this is not demonstrated in the paper. The reported margins over the reimplemented MobileNetV3 are small and come from single training runs, so the significance of the empirical contribution is currently uncertain.
major comments (2)
- [§4.2, Algorithm 1, §6.1] The entire search pipeline selects candidates based on the supernet's validation accuracy, but the paper never validates that this proxy correlates with stand-alone trained accuracy in the 12^14 search space used here. The manuscript itself concedes in §6.1 that the rank relationship between the supernet predictor and stand-alone training 'can be affected by initialization techniques and suboptimal training hyperparameters.' Since all NSGA-II selection, the Figure 7 ablation, and the final choice of MoGA-A/B/C rely on supernet validation accuracy, while Table 3 reports stand-alone trained accuracy, the claimed advantage over MobileNetV3 is not yet established. The authors should sample a set of architectures from the search space, train them stand-alone, and report the rank correlation (e.g., Spearman's rho) against supernet scores; without this, the three final models could be arbitrary points on a noisy proxy front.
- [Table 3] The top-1 accuracies in Table 3 are single-run results with no error bars or multiple seeds. The margins over the reimplemented MobileNetV3 (75.0% vs. the official 75.2%) are 0.3 to 0.9 percentage points, which is comparable to typical run-to-run variance in ImageNet training. To support the claim that MoGA 'outperforms' MobileNetV3, the authors should either report results from multiple training runs (at least for the final models and the baseline) or provide evidence of the expected variance of the training recipe, and ideally also compare against the official MobileNetV3 pretrained weights rather than only their own reimplementation.
minor comments (5)
- [Abstract, §1] The claim of being the 'first Mobile GPU-Aware NAS' should be qualified: previous works such as Proxyless-GPU have considered GPU targets, and the novelty should be framed as targeting mobile GPUs specifically.
- [§3.3, Eq. (4)] The objective weights wacc=0.4, wlat=0.4, wparams=0.2 are stated to come from a practical application, but no sensitivity analysis or justification is provided; a brief discussion of how robust the results are to these weights would strengthen the paper.
- [§4.2, Figure 4] The latency prediction validation reports an RMSE of 0.0571 ms, but it is not stated whether the models used for validation are independent of the lookup table construction; please clarify the cross-validation procedure.
- [Table 3] The columns 'Lat SNPE', 'Lat MACE', and 'Lat c' are not consistently populated across all rows; a footnote describing which framework and device each method used would improve comparability.
- [Conclusion] The phrase 'o(1) search cost' should be 'O(1)' (or 'constant per additional platform after amortization') to avoid confusion with little-o notation.
Circularity Check
MoGA's reported accuracies come from independent stand-alone evaluation; only a self-cited FairNAS ranking assumption is left unvalidated, so circularity is at most minor.
full rationale
The central reported results (MoGA-A 75.9%, MoGA-B 75.5%, MoGA-C 75.3% top-1 accuracy) are obtained by training the selected architectures stand-alone and evaluating on the official ImageNet validation set. These numbers are external measurements, not quantities defined by the search equations, so they do not reduce to the search inputs by construction. The supernet validation accuracy in Algorithm 1 is used only as a search-time selection signal; the paper does not define final accuracy as supernet accuracy. The latency 'prediction' in Section 4.2 is a lookup-table sum of individually benchmarked per-block latencies, validated against ground truth (RMSE 0.0571 ms), so it is a measurement composition rather than a fitted parameter renamed as a prediction. The hand-chosen weights w_acc=0.4, w_lat=0.4, w_params=0.2 are preferences, not fitted to the final top-1 accuracies. The one self-referential element is the reliance on the authors' own FairNAS work for the claim that the one-shot supernet provides 'a stable ranking' (Section 4.2) that transfers to stand-alone training, with Section 6.1 conceding this rank relationship 'can be affected by initialization techniques and suboptimal training hyperparameters.' That is a load-bearing validation assumption imported via self-citation, but it is an experimental claim from prior work rather than an equation-level equivalence or a fitted input. The absence of a rank-correlation check in this specific 12^14 search space is a robustness or correctness concern, not a circularity of the derivation. Therefore the circularity score is low: 2, reflecting the minor self-citation issue while recognizing that the final accuracy claims stand on independent evaluation.
Assumptions & free parameters
free parameters (3)
- wacc =
0.4
- wlat =
0.4
- wparams =
0.2
assumptions (4)
- domain assumption The one-shot supernet trained with FairNAS provides a stable and reliable ranking of candidate architectures that correlates with final stand-alone accuracy.
- domain assumption Total mobile GPU latency can be accurately predicted as the sum of per-layer latencies from a lookup table, with no interaction effects between layers.
- domain assumption Mobile networks in the target latency range are underfitted, so increasing the number of parameters improves accuracy without necessarily increasing latency.
- domain assumption The search space based on MobileNetV3-large layers with 12 choices per layer (kernel size, expansion ratio, SE) is sufficiently rich to contain architectures that outperform MobileNetV3.
Cite this review
Pith. "Pith review of MoGA: Searching Beyond MobileNetV3." pith.science (2026). https://pith.science/paper/XZYLN2UN
@misc{pith2026190801314,
author = {Pith},
title = {Pith review of: MoGA: Searching Beyond MobileNetV3},
year = {2026},
howpublished = {\url{https://pith.science/paper/XZYLN2UN}},
note = {Machine review of arXiv:1908.01314}
}
read the original abstract
The evolution of MobileNets has laid a solid foundation for neural network applications on mobile end. With the latest MobileNetV3, neural architecture search again claimed its supremacy in network design. Unfortunately, till today all mobile methods mainly focus on CPU latencies instead of GPU, the latter, however, is much preferred in practice for it has faster speed, lower overhead and less interference. Bearing the target hardware in mind, we propose the first Mobile GPU-Aware (MoGA) neural architecture search in order to be precisely tailored for real-world applications. Further, the ultimate objective to devise a mobile network lies in achieving better performance by maximizing the utilization of bounded resources. Urging higher capability while restraining time consumption is not reconcilable. We alleviate the tension by weighted evolution techniques. Moreover, we encourage increasing the number of parameters for higher representational power. With 200x fewer GPU days than MnasNet, we obtain a series of models that outperform MobileNetV3 under the similar latency constraints, i.e., MoGA-A achieves 75.9% top-1 accuracy on ImageNet, MoGA-B meets 75.5% which costs only 0.5 ms more on mobile GPU. MoGA-C best attests GPU-awareness by reaching 75.3% and being slower on CPU but faster on GPU.The models and test code is made available here https://github.com/xiaomi-automl/MoGA.
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[1]
Abadi, M.; Agarwal, A.; Barham, P.; Brevdo, E.; Chen, Z.; Citro, C.; Corrado, G. S.; Davis, A.; Dean, J.; Devin, M.; Ghemawat, S.; Goodfellow, I.; Harp, A.; Irving, G.; Isard, M.; Jia, Y.; Jozefowicz, R.; Kaiser, L.; Kudlur, M.; Levenberg, J.; Man\' e , D.; Monga, R.; Moore, S.; Murray, D.; Olah, C.; Schuster, M.; Shlens, J.; Steiner, B.; Sutskever, I.; T...
work page 2015
-
[2]
Bender, G.; Kindermans, P.-J.; Zoph, B.; Vasudevan, V.; and Le, Q. 2018. Understanding and Simplifying One-Shot Architecture Search . In International Conference on Machine Learning , 549--558
work page 2018
-
[3]
Cai, H.; Gan, C.; and Han, S. 2019. Once for All: Train One Network and Specialize it for Efficient Deployment . arXiv preprint. arXiv:1908.09791
arXiv 2019
-
[4]
Cai, H.; Zhu, L.; and Han, S. 2019. Proxyless NAS : Direct Neural Architecture Search on Target Task and Hardware . In International Conference on Learning Representations
work page 2019
-
[5]
Chu, X.; Zhang, B.; Xu, R.; and Li, J. 2019a. FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Search . arXiv preprint. arXiv:1907.01845
arXiv 1907
-
[6]
Chu, X.; Zhang, B.; Xu, R.; and Ma, H. 2019b. Multi-Objective Reinforced Evolution in Mobile Neural Architecture Search . arXiv preprint. arXiv:1901.01074
work page Pith review arXiv 1901
-
[7]
Deb, K.; Pratap, A.; Agarwal, S.; and Meyarivan, T. 2002. A Fast and Elitist Multiobjective Genetic Algorithm: NSGA-II . IEEE Transactions on Evolutionary Computation 6(2):182--197
work page 2002
-
[8]
Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and Fei-Fei, L. 2009. ImageNet: A Large-Scale Hierarchical Image Database . In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 248--255. IEEE
work page 2009
Show all 26 references
-
[9]
Dong, J.-D.; Cheng, A.-C.; Juan, D.-C.; Wei, W.; and Sun, M. 2018. DPP-Net : Device-aware Progressive Search for Pareto-optimal Neural Architectures . In Proceedings of the European Conference on Computer Vision , 517--531
2018
-
[10]
Goyal, P.; Doll \'a r, P.; Girshick, R.; Noordhuis, P.; Wesolowski, L.; Kyrola, A.; Tulloch, A.; Jia, Y.; and He, K. 2017. Accurate, Large Minibatch SGD : Training ImageNet in 1 Hour . arXiv preprint. arXiv:1706.02677
2017 arXiv
-
[11]
Guo, Z.; Zhang, X.; Mu, H.; Heng, W.; Liu, Z.; Wei, Y.; and Sun, J. 2019. Single Path One-Shot Neural Architecture Search with Uniform Sampling . arXiv preprint. arXiv:1904.00420
2019 arXiv
-
[12]
He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016. Deep Residual Learning for Image Recognition . In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 770--778
2016
-
[13]
G.; Zhu, M.; Chen, B.; Kalenichenko, D.; Wang, W.; Weyand, T.; Andreetto, M.; and Adam, H
Howard, A. G.; Zhu, M.; Chen, B.; Kalenichenko, D.; Wang, W.; Weyand, T.; Andreetto, M.; and Adam, H. 2017. MobileNets : Efficient. Convolutional Neural Networks for Mobile Vision Applications . arXiv preprint. arXiv:1704.04861
2017 arXiv
-
[14]
Howard, A.; Sandler, M.; Chu, G.; Chen, L.-C.; Chen, B.; Tan, M.; Wang, W.; Zhu, Y.; Pang, R.; Vasudevan, V.; et al. 2019. Searching for MobileNetV3 . arXiv preprint. arXiv:1905.02244
2019 arXiv
-
[15]
Hu, J.; Shen, L.; and Sun, G. 2018. Squeeze-and-Excitation Networks . In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 7132--7141
2018
-
[16]
Liu, H.; Simonyan, K.; and Yang, Y. 2019. DARTS: Differentiable Architecture Search . In International Conference on Learning Representations
2019
-
[17]
Qualcomm. 2019. Snapdragon Neural Processing Engine SDK . https://developer.qualcomm.com/software/qualcomm-neural-processing-sdk, version: 1.27.1.382
2019
-
[18]
Sandler, M.; Howard, A.; Zhu, M.; Zhmoginov, A.; and Chen, L.-C. 2018. MobileNetV2 : Inverted Residuals and Linear Bottlenecks . In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 4510--4520
2018
-
[19]
Srivastava, N.; Hinton, G.; Krizhevsky, A.; Sutskever, I.; and Salakhutdinov, R. 2014. Dropout: A Simple Way to Prevent Neural Networks from Overfitting . The Journal of Machine Learning Research 15(1):1929--1958
2014
-
[20]
Stamoulis, D.; Ding, R.; Wang, D.; Lymberopoulos, D.; Priyantha, B.; Liu, J.; and Marculescu, D. 2019. Single-Path NAS : Designing Hardware-Efficient ConvNets in less than 4 Hours . arXiv preprint. arXiv:1904.02877
2019 arXiv
-
[21]
Tan, M.; Chen, B.; Pang, R.; Vasudevan, V.; and Le, Q. V. 2019. MnasNet: Platform-Aware Neural Architecture Search for Mobile . In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
2019
-
[22]
Wu, B.; Dai, X.; Zhang, P.; Wang, Y.; Sun, F.; Wu, Y.; Tian, Y.; Vajda, P.; Jia, Y.; and Keutzer, K. 2019. FBNet : Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search . In The IEEE Conference on Computer Vision and Pattern Recognition
2019
-
[23]
Xiaomi. 2018. Mobile AI Compute Engine . https://github.com/XiaoMi/mace, commit hashtag: 03362fa0
2018
-
[24]
Yang, T.-J.; Howard, A.; Chen, B.; Zhang, X.; Go, A.; Sandler, M.; Sze, V.; and Adam, H. 2018. NetAdapt: Platform-Aware Neural Network. Adaptation for Mobile Applications . In Proceedings of the European Conference on Computer Vision , 285--300
2018
-
[25]
Zhang, X.; Zhou, X.; Lin, M.; and Sun, J. 2018. ShuffleNet : An Extremely Efficient Convolutional Neural Network for Mobile Devices . In The IEEE Conference on Computer Vision and Pattern Recognition
2018
-
[26]
Zoph, B.; Vasudevan, V.; Shlens, J.; and Le, Q. V. 2018. Learning Transferable Architectures for Scalable Image Recognition . In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 8697--8710
2018
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.