Pith. sign in

REVIEW 2 major objections 5 minor 26 references

MoGA: Searching Beyond MobileNetV3

T0 review · 2 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Pith's one-line read GPU-aware architecture search yields mobile models that beat MobileNetV3 on ImageNet

desk verdict A credible GPU-aware mobile NAS pipeline with real searched models, but the missing supernet-to-standalone rank check means the claimed gains over MobileNetV3 are not yet fully established. read the letter →

arxiv 1908.01314 v4 pith:XZYLN2UN submitted 2019-08-04 cs.LG cs.CVcs.NEstat.ML

classification cs.LGcs.CVcs.NEstat.ML
keywords neuralarchitecturesearchmobileGPUone-shotsupernetNSGA-IImulti-objectiveoptimizationImageNetclassificationlatency-awaredesignMoGA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that mobile neural architecture search should optimize for the GPU that actually runs the app, not the CPU that benchmarks are traditionally measured on. It proposes a search pipeline, MoGA, that scores candidate architectures with a one-shot supernet, predicts GPU latency from a per-layer lookup table, and evolves models with weighted NSGA-II over accuracy, latency, and parameter count. On ImageNet, the resulting MoGA-A/B/C reach 75.9%, 75.5%, and 75.3% top-1 accuracy, all above the authors' reproduction of MobileNetV3 at 75.0%, while requiring only 12 GPU-days of search, about 200x less than MnasNet. If correct, the result matters because it suggests hardware targeting is not a detail but a first-class design axis: a network can be slower on CPU yet faster on GPU and still be the right choice on phones.

What carries the argument

The load-bearing machinery is the weighted NSGA-II loop with a three-objective fitness: maximize accuracy, minimize mobile-GPU latency, and maximize parameter count. Each candidate's latency is read from a layerwise lookup table built by benchmarking the 12 possible block choices per layer once on the GPU, so latency is the sum of per-layer entries; each candidate's accuracy comes from a one-shot supernet trained with a strict-fairness sampling strategy, which the paper relies on to rank candidates stably. The crowding distance is weighted by preferences $w_{\mathrm{acc}}=w_{\mathrm{lat}}=0.4$, $w_{\mathrm{params}}=0.2$, so accuracy and latency dominate but parameter count still steers the search. The distinctive third objective rests on the observation that mobile-scale models are underfitted rather than overfitted, so enlarged parameter counts buy representational power without proportionally raising latency.

What would settle it

Train, say, fifty architectures randomly sampled from the $12^{14}$ search space, or from the final Pareto front, from scratch and compare their final top-1 accuracies with the supernet's validation scores; if the rank correlation is weak or negative, the supernet is not a faithful selector and the reported MoGA advantages are not attributable to the search.

Watch

Extended reading notes

Core claim

This paper claims that neural architecture search for mobile devices should optimize for the mobile GPU rather than the CPU, because CPU latency does not predict GPU latency: the authors measure 100 random models on both platforms and find no obvious linear correspondence. It builds a search over inverted-bottleneck blocks with $12^{14}$ possible architectures, scores candidates with a one-shot supernet trained under a fairness-enforced sampling scheme, predicts GPU latency by summing per-block times from a layerwise lookup table, and evolves models with weighted NSGA-II over three objectives: accuracy, latency, and number of parameters. The resulting MoGA-A, MoGA-B, and MoGA-C reach 75.9%, 75.5%, and 75.3% top-1 ImageNet accuracy under mobile-GPU latency budgets comparable to MobileNetV3's, at 75.0% in the authors' reproduction, and the whole search costs about 12 GPU days, roughly 200 times fewer than MnasNet. MoGA-C in particular is slower on CPU but faster on GPU than MobileNetV3, which the paper presents as direct evidence that the optimization target changes the architecture.

Load-bearing premise

The search's accuracy signal is a one-shot supernet's validation score, so the pipeline assumes that score ranks candidate architectures in the same order as training each candidate from scratch would.

Editorial extensions

If this is right

  • Optimizing for mobile GPU latency rather than CPU latency changes the searched architectures: MoGA-C is slower on CPU but faster on GPU than MobileNetV3, so hardware choice is a first-class design axis rather than a measurement detail.
  • Adding parameter count as a maximization objective, with lower weight than accuracy and latency, shifts the Pareto front toward models with higher capacity and better accuracy within the same latency budget.
  • A one-shot supernet plus a layerwise latency lookup table makes the search cost about 12 GPU days, and re-targeting to another mobile device only requires a refreshed latency table rather than retraining the supernet.
  • The searched models give concrete accuracy/latency points above MobileNetV3 on ImageNet: 75.9% for MoGA-A, 75.5% for MoGA-B, and 75.3% for MoGA-C.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the CPU-versus-GPU latency divergence holds beyond the phones tested, earlier CPU-aware NAS results should be re-examined on GPU Pareto fronts; architectures that look bad by CPU latency may be competitive when GPU latency is the constraint.
  • The parameter-maximizing objective is tailored to the underfitting regime of small mobile models; on larger datasets, harder distillation targets, or much larger capacity budgets, the sign of that objective would likely need to flip.
  • The layerwise latency sum assumes per-operator costs add independently; operator fusion, framework updates, or thermal throttling would violate additivity, so the reported latency numbers are tied to the specific runtime and device measured.
  • The paper does not report the rank correlation between supernet validation accuracy and stand-alone trained accuracy for the searched population; that correlation is the quantity that would determine whether the search's selections, rather than the training recipe, are responsible for the final accuracies.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes Mobile GPU-Aware Neural Architecture Search (MoGA), a one-shot NAS method that targets mobile GPU latency rather than CPU latency. The search uses a FairNAS-style supernet as a fast accuracy evaluator, a layerwise latency lookup table for mobile GPU (MACE) latency, and a weighted NSGA-II with three objectives: accuracy, latency, and number of parameters. The authors report three searched models, MoGA-A/B/C, achieving 75.9/75.5/75.3% ImageNet top-1 accuracy, which they claim outperform MobileNetV3-large under similar mobile GPU latency, with a total search cost of 12 GPU days, about 200x less than MnasNet. The paper also presents ablations on the weighted objective and the hierarchical mutator, and discusses the GPU/CPU latency discrepancy.

Significance. If the results hold, MoGA provides evidence that hardware-aware NAS should target the actual deployment hardware (mobile GPU) rather than CPU, and that adding a parameter-maximization objective can improve accuracy without increasing latency. The public release of models and test code is a strength. However, the central claim that the searched models outperform MobileNetV3 depends on the supernet's validation accuracy being a reliable ranking signal for stand-alone trained models, and this is not demonstrated in the paper. The reported margins over the reimplemented MobileNetV3 are small and come from single training runs, so the significance of the empirical contribution is currently uncertain.

major comments (2)
  1. [§4.2, Algorithm 1, §6.1] The entire search pipeline selects candidates based on the supernet's validation accuracy, but the paper never validates that this proxy correlates with stand-alone trained accuracy in the 12^14 search space used here. The manuscript itself concedes in §6.1 that the rank relationship between the supernet predictor and stand-alone training 'can be affected by initialization techniques and suboptimal training hyperparameters.' Since all NSGA-II selection, the Figure 7 ablation, and the final choice of MoGA-A/B/C rely on supernet validation accuracy, while Table 3 reports stand-alone trained accuracy, the claimed advantage over MobileNetV3 is not yet established. The authors should sample a set of architectures from the search space, train them stand-alone, and report the rank correlation (e.g., Spearman's rho) against supernet scores; without this, the three final models could be arbitrary points on a noisy proxy front.
  2. [Table 3] The top-1 accuracies in Table 3 are single-run results with no error bars or multiple seeds. The margins over the reimplemented MobileNetV3 (75.0% vs. the official 75.2%) are 0.3 to 0.9 percentage points, which is comparable to typical run-to-run variance in ImageNet training. To support the claim that MoGA 'outperforms' MobileNetV3, the authors should either report results from multiple training runs (at least for the final models and the baseline) or provide evidence of the expected variance of the training recipe, and ideally also compare against the official MobileNetV3 pretrained weights rather than only their own reimplementation.
minor comments (5)
  1. [Abstract, §1] The claim of being the 'first Mobile GPU-Aware NAS' should be qualified: previous works such as Proxyless-GPU have considered GPU targets, and the novelty should be framed as targeting mobile GPUs specifically.
  2. [§3.3, Eq. (4)] The objective weights wacc=0.4, wlat=0.4, wparams=0.2 are stated to come from a practical application, but no sensitivity analysis or justification is provided; a brief discussion of how robust the results are to these weights would strengthen the paper.
  3. [§4.2, Figure 4] The latency prediction validation reports an RMSE of 0.0571 ms, but it is not stated whether the models used for validation are independent of the lookup table construction; please clarify the cross-validation procedure.
  4. [Table 3] The columns 'Lat SNPE', 'Lat MACE', and 'Lat c' are not consistently populated across all rows; a footnote describing which framework and device each method used would improve comparability.
  5. [Conclusion] The phrase 'o(1) search cost' should be 'O(1)' (or 'constant per additional platform after amortization') to avoid confusion with little-o notation.

Circularity Check

0 steps flagged · score 2.0 of 10

MoGA's reported accuracies come from independent stand-alone evaluation; only a self-cited FairNAS ranking assumption is left unvalidated, so circularity is at most minor.

full rationale

The central reported results (MoGA-A 75.9%, MoGA-B 75.5%, MoGA-C 75.3% top-1 accuracy) are obtained by training the selected architectures stand-alone and evaluating on the official ImageNet validation set. These numbers are external measurements, not quantities defined by the search equations, so they do not reduce to the search inputs by construction. The supernet validation accuracy in Algorithm 1 is used only as a search-time selection signal; the paper does not define final accuracy as supernet accuracy. The latency 'prediction' in Section 4.2 is a lookup-table sum of individually benchmarked per-block latencies, validated against ground truth (RMSE 0.0571 ms), so it is a measurement composition rather than a fitted parameter renamed as a prediction. The hand-chosen weights w_acc=0.4, w_lat=0.4, w_params=0.2 are preferences, not fitted to the final top-1 accuracies. The one self-referential element is the reliance on the authors' own FairNAS work for the claim that the one-shot supernet provides 'a stable ranking' (Section 4.2) that transfers to stand-alone training, with Section 6.1 conceding this rank relationship 'can be affected by initialization techniques and suboptimal training hyperparameters.' That is a load-bearing validation assumption imported via self-citation, but it is an experimental claim from prior work rather than an equation-level equivalence or a fitted input. The absence of a rank-correlation check in this specific 12^14 search space is a robustness or correctness concern, not a circularity of the derivation. Therefore the circularity score is low: 2, reflecting the minor self-citation issue while recognizing that the final accuracy claims stand on independent evaluation.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or conceptual entities; it rearranges known building blocks (inverted bottleneck, SE, kernel sizes) within a fixed search space. The main assumptions are the reliability of the supernet predictor, the additivity of layer latencies, the underfitting justification for the parameter objective, and the sufficiency of the search space. Three preference weights are hand-chosen and directly affect the search outcome.

free parameters (3)
  • wacc = 0.4
    Hand-chosen preference weight for accuracy in the weighted crowding distance (Equation 5). The search outcome depends on this choice; stated in Section 4.3.
  • wlat = 0.4
    Hand-chosen preference weight for latency in the weighted crowding distance (Equation 5). Set from a practical application requirement, not fitted; stated in Section 4.3.
  • wparams = 0.2
    Hand-chosen preference weight for the number of parameters. This weights the third objective lower than accuracy and latency; stated in Section 4.3.
assumptions (4)
  • domain assumption The one-shot supernet trained with FairNAS provides a stable and reliable ranking of candidate architectures that correlates with final stand-alone accuracy.
    The search uses supernet predictions as the accuracy objective without full training of each candidate. The paper cites FairNAS (Chu et al. 2019a) for this property, but its reliability is assumed and not re-validated in this search space. Invoked in Section 4.2 and Algorithm 1.
  • domain assumption Total mobile GPU latency can be accurately predicted as the sum of per-layer latencies from a lookup table, with no interaction effects between layers.
    The latency table is built per choice block and summed to predict total model latency (Section 4.2). Empirical RMSE of 0.0571 ms is reported, but interactions (memory, scheduling, kernel fusion) are assumed negligible.
  • domain assumption Mobile networks in the target latency range are underfitted, so increasing the number of parameters improves accuracy without necessarily increasing latency.
    This motivates the third objective in Equation 3. It is based on the MobileNet evolution trend (Figure 3) and the underfitting argument in Section 3.2. If false, maximizing parameter count would not help and could hurt.
  • domain assumption The search space based on MobileNetV3-large layers with 12 choices per layer (kernel size, expansion ratio, SE) is sufficiently rich to contain architectures that outperform MobileNetV3.
    The search space is fixed to that of MobileNetV3-large (Section 4.1). The claim that searched models beat MobileNetV3 depends on the space being comparable in expressiveness.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MoGA: Searching Beyond MobileNetV3." pith.science (2026). https://pith.science/paper/XZYLN2UN

@misc{pith2026190801314,
  author       = {Pith},
  title        = {Pith review of: MoGA: Searching Beyond MobileNetV3},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XZYLN2UN}},
  note         = {Machine review of arXiv:1908.01314}
}
read the original abstract

The evolution of MobileNets has laid a solid foundation for neural network applications on mobile end. With the latest MobileNetV3, neural architecture search again claimed its supremacy in network design. Unfortunately, till today all mobile methods mainly focus on CPU latencies instead of GPU, the latter, however, is much preferred in practice for it has faster speed, lower overhead and less interference. Bearing the target hardware in mind, we propose the first Mobile GPU-Aware (MoGA) neural architecture search in order to be precisely tailored for real-world applications. Further, the ultimate objective to devise a mobile network lies in achieving better performance by maximizing the utilization of bounded resources. Urging higher capability while restraining time consumption is not reconcilable. We alleviate the tension by weighted evolution techniques. Moreover, we encourage increasing the number of parameters for higher representational power. With 200x fewer GPU days than MnasNet, we obtain a series of models that outperform MobileNetV3 under the similar latency constraints, i.e., MoGA-A achieves 75.9% top-1 accuracy on ImageNet, MoGA-B meets 75.5% which costs only 0.5 ms more on mobile GPU. MoGA-C best attests GPU-awareness by reaching 75.3% and being slower on CPU but faster on GPU.The models and test code is made available here https://github.com/xiaomi-automl/MoGA.

Figures

Figures reproduced from arXiv: 1908.01314 by the authors.

Figure 1
Figure 1. Latency pie chart of MoGA-A/B/C, MobileNetV3 [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Latency relationship on mobile CPUs vs. on mo [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. The evolution of MobileNets. Moreover, for the mobile end, models tend to be underfit￾ted instead of overfitted since they carry fewer numbers of [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Mobile GPU latency measured vs. predicted ones. [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: The overall pipeline of MoGA. 5 Experiments 5.1 Mobile GPU Latency In practice, we employ SNPE (Qualcomm 2019) and Mo￾bile AI Compute Engine (MACE) for mobile GPU bench- [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 1
Figure 1. Figure 1: Additionally, there is a discrepancy when running [PITH_FULL_IMAGE:figures/full_fig_p006_1.png]
Figure 7
Figure 7. Figure 7: Pareto Front of weighted NSGA-II with hierarchi [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: The Architectures of MoGA-A, B, C. Note E [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: Histogram on numbers of parameters of models [PITH_FULL_IMAGE:figures/full_fig_p007_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 17 canonical work pages

  1. [1]

    Abadi, M.; Agarwal, A.; Barham, P.; Brevdo, E.; Chen, Z.; Citro, C.; Corrado, G. S.; Davis, A.; Dean, J.; Devin, M.; Ghemawat, S.; Goodfellow, I.; Harp, A.; Irving, G.; Isard, M.; Jia, Y.; Jozefowicz, R.; Kaiser, L.; Kudlur, M.; Levenberg, J.; Man\' e , D.; Monga, R.; Moore, S.; Murray, D.; Olah, C.; Schuster, M.; Shlens, J.; Steiner, B.; Sutskever, I.; T...

  2. [2]

    Bender, G.; Kindermans, P.-J.; Zoph, B.; Vasudevan, V.; and Le, Q. 2018. Understanding and Simplifying One-Shot Architecture Search . In International Conference on Machine Learning , 549--558

  3. [3]

    Cai, H.; Gan, C.; and Han, S. 2019. Once for All: Train One Network and Specialize it for Efficient Deployment . arXiv preprint. arXiv:1908.09791

  4. [4]

    Cai, H.; Zhu, L.; and Han, S. 2019. Proxyless NAS : Direct Neural Architecture Search on Target Task and Hardware . In International Conference on Learning Representations

  5. [5]

    Chu, X.; Zhang, B.; Xu, R.; and Li, J. 2019a. FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Search . arXiv preprint. arXiv:1907.01845

  6. [6]

    Chu, X.; Zhang, B.; Xu, R.; and Ma, H. 2019b. Multi-Objective Reinforced Evolution in Mobile Neural Architecture Search . arXiv preprint. arXiv:1901.01074

  7. [7]

    Deb, K.; Pratap, A.; Agarwal, S.; and Meyarivan, T. 2002. A Fast and Elitist Multiobjective Genetic Algorithm: NSGA-II . IEEE Transactions on Evolutionary Computation 6(2):182--197

  8. [8]

    Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and Fei-Fei, L. 2009. ImageNet: A Large-Scale Hierarchical Image Database . In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 248--255. IEEE

Show all 26 references
  1. [9]

    Dong, J.-D.; Cheng, A.-C.; Juan, D.-C.; Wei, W.; and Sun, M. 2018. DPP-Net : Device-aware Progressive Search for Pareto-optimal Neural Architectures . In Proceedings of the European Conference on Computer Vision , 517--531

  2. [10]

    Goyal, P.; Doll \'a r, P.; Girshick, R.; Noordhuis, P.; Wesolowski, L.; Kyrola, A.; Tulloch, A.; Jia, Y.; and He, K. 2017. Accurate, Large Minibatch SGD : Training ImageNet in 1 Hour . arXiv preprint. arXiv:1706.02677

  3. [11]

    Guo, Z.; Zhang, X.; Mu, H.; Heng, W.; Liu, Z.; Wei, Y.; and Sun, J. 2019. Single Path One-Shot Neural Architecture Search with Uniform Sampling . arXiv preprint. arXiv:1904.00420

  4. [12]

    He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016. Deep Residual Learning for Image Recognition . In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 770--778

  5. [13]

    G.; Zhu, M.; Chen, B.; Kalenichenko, D.; Wang, W.; Weyand, T.; Andreetto, M.; and Adam, H

    Howard, A. G.; Zhu, M.; Chen, B.; Kalenichenko, D.; Wang, W.; Weyand, T.; Andreetto, M.; and Adam, H. 2017. MobileNets : Efficient. Convolutional Neural Networks for Mobile Vision Applications . arXiv preprint. arXiv:1704.04861

  6. [14]

    Howard, A.; Sandler, M.; Chu, G.; Chen, L.-C.; Chen, B.; Tan, M.; Wang, W.; Zhu, Y.; Pang, R.; Vasudevan, V.; et al. 2019. Searching for MobileNetV3 . arXiv preprint. arXiv:1905.02244

  7. [15]

    Hu, J.; Shen, L.; and Sun, G. 2018. Squeeze-and-Excitation Networks . In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 7132--7141

  8. [16]

    Liu, H.; Simonyan, K.; and Yang, Y. 2019. DARTS: Differentiable Architecture Search . In International Conference on Learning Representations

  9. [17]

    Qualcomm. 2019. Snapdragon Neural Processing Engine SDK . https://developer.qualcomm.com/software/qualcomm-neural-processing-sdk, version: 1.27.1.382

  10. [18]

    Sandler, M.; Howard, A.; Zhu, M.; Zhmoginov, A.; and Chen, L.-C. 2018. MobileNetV2 : Inverted Residuals and Linear Bottlenecks . In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 4510--4520

  11. [19]

    Srivastava, N.; Hinton, G.; Krizhevsky, A.; Sutskever, I.; and Salakhutdinov, R. 2014. Dropout: A Simple Way to Prevent Neural Networks from Overfitting . The Journal of Machine Learning Research 15(1):1929--1958

  12. [20]

    Stamoulis, D.; Ding, R.; Wang, D.; Lymberopoulos, D.; Priyantha, B.; Liu, J.; and Marculescu, D. 2019. Single-Path NAS : Designing Hardware-Efficient ConvNets in less than 4 Hours . arXiv preprint. arXiv:1904.02877

  13. [21]

    Tan, M.; Chen, B.; Pang, R.; Vasudevan, V.; and Le, Q. V. 2019. MnasNet: Platform-Aware Neural Architecture Search for Mobile . In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

  14. [22]

    Wu, B.; Dai, X.; Zhang, P.; Wang, Y.; Sun, F.; Wu, Y.; Tian, Y.; Vajda, P.; Jia, Y.; and Keutzer, K. 2019. FBNet : Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search . In The IEEE Conference on Computer Vision and Pattern Recognition

  15. [23]

    Xiaomi. 2018. Mobile AI Compute Engine . https://github.com/XiaoMi/mace, commit hashtag: 03362fa0

  16. [24]

    Yang, T.-J.; Howard, A.; Chen, B.; Zhang, X.; Go, A.; Sandler, M.; Sze, V.; and Adam, H. 2018. NetAdapt: Platform-Aware Neural Network. Adaptation for Mobile Applications . In Proceedings of the European Conference on Computer Vision , 285--300

  17. [25]

    Zhang, X.; Zhou, X.; Lin, M.; and Sun, J. 2018. ShuffleNet : An Extremely Efficient Convolutional Neural Network for Mobile Devices . In The IEEE Conference on Computer Vision and Pattern Recognition

  18. [26]

    Zoph, B.; Vasudevan, V.; Shlens, J.; and Le, Q. V. 2018. Learning Transferable Architectures for Scalable Image Recognition . In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 8697--8710

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.