{"id":"dfeac785-b954-4c71-b82a-224d024f01fb","arxiv_id":"1908.01314","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"MoGA uses weighted NSGA-II and a one-shot supernet to search mobile GPU-aware architectures, producing models that beat MobileNetV3 on ImageNet at similar mobile GPU latency with 200x less search cost than MnasNet.","lead":"This paper describes a neural architecture search (NAS) method that designs mobile networks to run fast on phone GPUs, not just CPUs. The searched models reach slightly higher ImageNet accuracy than Google's MobileNetV3 at similar GPU speed, using only 12 GPU days of search time.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The search's selection signal is the supernet's validation accuracy, but the paper never shows that this proxy ranks stand-alone-trained models correctly, so the claimed NAS advantage over MobileNetV3 is not yet established.","rationale":"The reader's conditional verdict already centers on the unvalidated supernet ranking, and the manuscript itself contains the admission in Section 6.1, so this is not an invented concern. I agree that it is the weakest load-bearing step: every algorithmic contribution (weighted NSGA-II, the parameter objective, hierarchical mutation ablation, and final model selection) is mediated by the supernet's score. The final models' accuracies alone cannot validate the search because they are single runs and the margins over MobileNetV3 are small. A rank-correlation experiment is cheap relative to the search cost and would directly settle whether the proxy is usable. I do not see an internal inconsistency that would require rejection; the right verdict is the same conditional accept with this validation requested. The 'first GPU-aware NAS' overstatement and lack of pinned code are secondary and do not change this assessment.","tokens_in":12148,"tokens_out":5586,"duration_ms":58844,"concrete_test":"Train 30-50 architectures sampled from the last NSGA-II generation (and, for contrast, from the initial random population) from scratch with the exact MoGA stand-alone training recipe; compute the Spearman rank correlation between their supernet validation top-1 accuracy (as used in Algorithm 1) and their stand-alone top-1 accuracy, with a scatter plot. Also retrain MoGA-A/B/C and the MobileNetV3 reference at least 3 times to attach error bars. If the rank correlation is weak (e.g., rho < 0.5), the search-and-selection pipeline is not validated; if it is strong, the conditional concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is Algorithm 1's use of the one-shot supernet to score every candidate: Section 4.2 says FairNAS gives 'fast evaluation with a stable ranking,' and Section 6.1 concedes that the supernet/stand-alone rank relationship 'can be affected by initialization techniques and suboptimal training hyperparameters.' Yet the paper never measures that rank relationship in this 12^14 search space. All NSGA-II selection, the weighted-objective ablation in Figure 7, and the final choice of MoGA-A/B/C from the Pareto front are based on supernet validation accuracy, while Table 3 reports stand-alone trained accuracy. Because the supernet's absolute scores (~0.3-0.65) are far below stand-alone accuracy (~0.75), only ranking fidelity can justify transferring search decisions. Without a rank-correlation check, the 'searching beyond' claim is unsupported: the three final models could be arbitrary points on a noisy proxy front, and the small reported margins over a reimplemented MobileNetV3 (75.0 vs official 75.2) could be selection noise rather than genuine architecture advantages.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Mobile GPU-Aware Neural Architecture Search (MoGA), a one-shot NAS method that targets mobile GPU latency rather than CPU latency. The search uses a FairNAS-style supernet as a fast accuracy evaluator, a layerwise latency lookup table for mobile GPU (MACE) latency, and a weighted NSGA-II with three objectives: accuracy, latency, and number of parameters. The authors report three searched models, MoGA-A/B/C, achieving 75.9/75.5/75.3% ImageNet top-1 accuracy, which they claim outperform MobileNetV3-large under similar mobile GPU latency, with a total search cost of 12 GPU days, about 200x less than MnasNet. The paper also presents ablations on the weighted objective and the hierarchical mutator, and discusses the GPU/CPU latency discrepancy.","tokens_in":12372,"tokens_out":3420,"duration_ms":34018,"significance":"If the results hold, MoGA provides evidence that hardware-aware NAS should target the actual deployment hardware (mobile GPU) rather than CPU, and that adding a parameter-maximization objective can improve accuracy without increasing latency. The public release of models and test code is a strength. However, the central claim that the searched models outperform MobileNetV3 depends on the supernet's validation accuracy being a reliable ranking signal for stand-alone trained models, and this is not demonstrated in the paper. The reported margins over the reimplemented MobileNetV3 are small and come from single training runs, so the significance of the empirical contribution is currently uncertain.","major_comments":[{"comment":"The entire search pipeline selects candidates based on the supernet's validation accuracy, but the paper never validates that this proxy correlates with stand-alone trained accuracy in the 12^14 search space used here. The manuscript itself concedes in §6.1 that the rank relationship between the supernet predictor and stand-alone training 'can be affected by initialization techniques and suboptimal training hyperparameters.' Since all NSGA-II selection, the Figure 7 ablation, and the final choice of MoGA-A/B/C rely on supernet validation accuracy, while Table 3 reports stand-alone trained accuracy, the claimed advantage over MobileNetV3 is not yet established. The authors should sample a set of architectures from the search space, train them stand-alone, and report the rank correlation (e.g., Spearman's rho) against supernet scores; without this, the three final models could be arbitrary points on a noisy proxy front.","section":"§4.2, Algorithm 1, §6.1"},{"comment":"The top-1 accuracies in Table 3 are single-run results with no error bars or multiple seeds. The margins over the reimplemented MobileNetV3 (75.0% vs. the official 75.2%) are 0.3 to 0.9 percentage points, which is comparable to typical run-to-run variance in ImageNet training. To support the claim that MoGA 'outperforms' MobileNetV3, the authors should either report results from multiple training runs (at least for the final models and the baseline) or provide evidence of the expected variance of the training recipe, and ideally also compare against the official MobileNetV3 pretrained weights rather than only their own reimplementation.","section":"Table 3"}],"minor_comments":[{"comment":"The claim of being the 'first Mobile GPU-Aware NAS' should be qualified: previous works such as Proxyless-GPU have considered GPU targets, and the novelty should be framed as targeting mobile GPUs specifically.","section":"Abstract, §1"},{"comment":"The objective weights wacc=0.4, wlat=0.4, wparams=0.2 are stated to come from a practical application, but no sensitivity analysis or justification is provided; a brief discussion of how robust the results are to these weights would strengthen the paper.","section":"§3.3, Eq. (4)"},{"comment":"The latency prediction validation reports an RMSE of 0.0571 ms, but it is not stated whether the models used for validation are independent of the lookup table construction; please clarify the cross-validation procedure.","section":"§4.2, Figure 4"},{"comment":"The columns 'Lat SNPE', 'Lat MACE', and 'Lat c' are not consistently populated across all rows; a footnote describing which framework and device each method used would improve comparability.","section":"Table 3"},{"comment":"The phrase 'o(1) search cost' should be 'O(1)' (or 'constant per additional platform after amortization') to avoid confusion with little-o notation.","section":"Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The paper's main empirical evidence is a set of single-run accuracies over a reimplementation of the baseline, and the search signal itself is not validated in the target search space. The lack of a rank-correlation check is the key technical risk; if the authors can provide such a check and show that the supernet ranking is faithful, the paper would be considerably stronger. The heavy reliance on the authors' own FairNAS method for the core claim is also worth noting for the editor."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper to know: a mobile NAS that targets GPU latency instead of CPU, using a weighted NSGA-II over three objectives (accuracy, latency, params) and a one-shot supernet. It produces three architectures (MoGA-A/B/C) competitive with MobileNetV3, at a reported search cost of 12 GPU days. That is a real empirical contribution: the models are new, the GPU-aware comparisons are explicit, and the latency lookup table has tiny RMSE (0.057 ms). The code is released, though not pinned to a commit.\n\nWhat it does well: the hardware-awareness argument is concrete—CPU and GPU latencies are not linearly related, and MoGA-C is slower on CPU but faster on GPU than MobileNetV3. That is exactly the right way to demonstrate that target hardware matters. The param-count objective is sensible for mobile underfitting. The use of MobileNetV3's training recipe for stand-alone models is fair, and the table gives both reimplemented and official numbers.\n\nThe soft spots, in order: (1) The search selects architectures using supernet validation accuracy, but there is no rank-correlation check between supernet scores and stand-alone accuracy in this search space. The paper cites FairNAS for stable ranking and even concedes in Section 6.1 that the relationship can be affected by initialization and hyperparameters, but does not measure it. Given that the reported margins over MobileNetV3 are small (0.3–0.9 points, single runs, no error bars), selection noise could explain part of the advantage. That is the load-bearing issue. (2) The \"first Mobile GPU-Aware NAS\" claim is overstated—Proxyless GPU exists and is in their own table. (3) The MobileNetV3 baseline is a reimplementation (75.0 vs official 75.2), which weakens the comparison slightly. (4) The code is not pinned to a reproducible commit.\n\nNone of this sinks the paper. The direction is credible, the models are concrete, and the external ImageNet measurements are independent of the search's fitted weights. But the central claim—that the search genuinely found better architectures than MobileNetV3—needs the rank check and ideally repeated runs or error bars.\n\nFor whom: anyone working on hardware-aware NAS or mobile model design. It deserves a serious referee; the missing rank correlation should be a major revision request, not a desk reject.\n\nRecommendation: send to peer review.","headline":"A credible GPU-aware mobile NAS pipeline with real searched models, but the missing supernet-to-standalone rank check means the claimed gains over MobileNetV3 are not yet fully established.","tokens_in":12898,"tokens_out":2726,"would_cite":true,"duration_ms":27602,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GPU-aware architecture search yields mobile models that beat MobileNetV3 on ImageNet","keywords":["neural architecture search","mobile GPU","one-shot supernet","NSGA-II","multi-objective optimization","ImageNet classification","latency-aware design","MoGA"],"falsifier":"Train, say, fifty architectures randomly sampled from the $12^{14}$ search space, or from the final Pareto front, from scratch and compare their final top-1 accuracies with the supernet's validation scores; if the rank correlation is weak or negative, the supernet is not a faithful selector and the reported MoGA advantages are not attributable to the search.","tokens_in":11943,"feed_emoji":"📱","tokens_out":10310,"duration_ms":94133,"temperature":0.7,"pith_summary":"This paper tries to show that mobile neural architecture search should optimize for the GPU that actually runs the app, not the CPU that benchmarks are traditionally measured on. It proposes a search pipeline, MoGA, that scores candidate architectures with a one-shot supernet, predicts GPU latency from a per-layer lookup table, and evolves models with weighted NSGA-II over accuracy, latency, and parameter count. On ImageNet, the resulting MoGA-A/B/C reach 75.9%, 75.5%, and 75.3% top-1 accuracy, all above the authors' reproduction of MobileNetV3 at 75.0%, while requiring only 12 GPU-days of search, about 200x less than MnasNet. If correct, the result matters because it suggests hardware targeting is not a detail but a first-class design axis: a network can be slower on CPU yet faster on GPU and still be the right choice on phones.","feed_headline":"GPU-aware search beats MobileNetV3 on phones","feed_subtitle":"Three searched models exceed 75% top-1 on ImageNet while staying in MobileNetV3's mobile-GPU latency range.","key_machinery":"The load-bearing machinery is the weighted NSGA-II loop with a three-objective fitness: maximize accuracy, minimize mobile-GPU latency, and maximize parameter count. Each candidate's latency is read from a layerwise lookup table built by benchmarking the 12 possible block choices per layer once on the GPU, so latency is the sum of per-layer entries; each candidate's accuracy comes from a one-shot supernet trained with a strict-fairness sampling strategy, which the paper relies on to rank candidates stably. The crowding distance is weighted by preferences $w_{\\mathrm{acc}}=w_{\\mathrm{lat}}=0.4$, $w_{\\mathrm{params}}=0.2$, so accuracy and latency dominate but parameter count still steers the search. The distinctive third objective rests on the observation that mobile-scale models are underfitted rather than overfitted, so enlarged parameter counts buy representational power without proportionally raising latency.","core_discovery":"This paper claims that neural architecture search for mobile devices should optimize for the mobile GPU rather than the CPU, because CPU latency does not predict GPU latency: the authors measure 100 random models on both platforms and find no obvious linear correspondence. It builds a search over inverted-bottleneck blocks with $12^{14}$ possible architectures, scores candidates with a one-shot supernet trained under a fairness-enforced sampling scheme, predicts GPU latency by summing per-block times from a layerwise lookup table, and evolves models with weighted NSGA-II over three objectives: accuracy, latency, and number of parameters. The resulting MoGA-A, MoGA-B, and MoGA-C reach 75.9%, 75.5%, and 75.3% top-1 ImageNet accuracy under mobile-GPU latency budgets comparable to MobileNetV3's, at 75.0% in the authors' reproduction, and the whole search costs about 12 GPU days, roughly 200 times fewer than MnasNet. MoGA-C in particular is slower on CPU but faster on GPU than MobileNetV3, which the paper presents as direct evidence that the optimization target changes the architecture.","pith_inferences":["If the CPU-versus-GPU latency divergence holds beyond the phones tested, earlier CPU-aware NAS results should be re-examined on GPU Pareto fronts; architectures that look bad by CPU latency may be competitive when GPU latency is the constraint.","The parameter-maximizing objective is tailored to the underfitting regime of small mobile models; on larger datasets, harder distillation targets, or much larger capacity budgets, the sign of that objective would likely need to flip.","The layerwise latency sum assumes per-operator costs add independently; operator fusion, framework updates, or thermal throttling would violate additivity, so the reported latency numbers are tied to the specific runtime and device measured.","The paper does not report the rank correlation between supernet validation accuracy and stand-alone trained accuracy for the searched population; that correlation is the quantity that would determine whether the search's selections, rather than the training recipe, are responsible for the final accuracies."],"forward_implications":["Optimizing for mobile GPU latency rather than CPU latency changes the searched architectures: MoGA-C is slower on CPU but faster on GPU than MobileNetV3, so hardware choice is a first-class design axis rather than a measurement detail.","Adding parameter count as a maximization objective, with lower weight than accuracy and latency, shifts the Pareto front toward models with higher capacity and better accuracy within the same latency budget.","A one-shot supernet plus a layerwise latency lookup table makes the search cost about 12 GPU days, and re-targeting to another mobile device only requires a refreshed latency table rather than retraining the supernet.","The searched models give concrete accuracy/latency points above MobileNetV3 on ImageNet: 75.9% for MoGA-A, 75.5% for MoGA-B, and 75.3% for MoGA-C."],"supporting_citations":[{"why":"Supplies the MnasNet baseline, the platform-aware search formulation, and the latency-reward objective that MoGA modifies.","marker":"Tan et al. 2019"},{"why":"Supplies MobileNetV3, the accuracy and latency baseline to beat, plus the stand-alone training recipe reused for the searched models.","marker":"Howard et al. 2019"},{"why":"Supplies the fairness-enforced one-shot supernet training that MoGA uses for fast accuracy evaluation and stable ranking.","marker":"Chu et al. 2019a"},{"why":"Supplies the NSGA-II multi-objective genetic algorithm that MoGA weights toward accuracy, latency, and parameter count.","marker":"Deb et al. 2002"},{"why":"Supplies ProxylessNAS, a comparison baseline, and the kernel-size/expansion/SE choices of the search space.","marker":"Cai, Zhu, and Han 2019"},{"why":"Supplies MobileNetV2 and its inverted residual blocks, the layer primitive of the search space and a CPU-latency baseline.","marker":"Sandler et al. 2018"},{"why":"Supplies the mobile-GPU inference engine used to construct the layerwise latency lookup table and report on-device GPU latencies.","marker":"Xiaomi 2018"},{"why":"Supplies the mobile-GPU runtime used for the final latency comparisons in the results table.","marker":"Qualcomm 2019"},{"why":"Supplies the ImageNet dataset and protocol on which all accuracy numbers are measured.","marker":"Deng et al. 2009"}],"fun_headline_variants":["GPU-aware NAS outdoes MobileNetV3 on mobile","Searching for mobile GPU, not CPU, beats MobileNetV3","MoGA: 200x cheaper GPU search, beats MobileNetV3","Mobile GPU search hits 75.9% top-1, beats MobileNetV3"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The search's accuracy signal is a one-shot supernet's validation score, so the pipeline assumes that score ranks candidate architectures in the same order as training each candidate from scratch would.","fun_headline_variants_meta":{"raw":{"variants":["GPU-aware NAS outdoes MobileNetV3 on mobile","Searching for mobile GPU, not CPU, beats MobileNetV3","MoGA: 200x cheaper GPU search, beats MobileNetV3","Mobile GPU search hits 75.9% top-1, beats MobileNetV3"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000333,"raw_usage":{"total_tokens":1897,"prompt_tokens":1037,"completion_tokens":860,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":653,"completion_tokens_details":{"reasoning_tokens":779}},"tokens_in":653,"tokens_out":860,"duration_ms":8393,"temperature":1.0,"reasoning_tokens":779,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T15:16:24.267939+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train, say, fifty architectures randomly sampled from the $12^{14}$ search space, or from the final Pareto front, from scratch and compare their final top-1 accuracies with the supernet's validation scores; if the rank correlation is weak or negative, the supernet is not a faithful selector and the reported MoGA advantages are not attributable to the search.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the MnasNet baseline, the platform-aware search formulation, and the latency-reward objective that MoGA modifies."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the NSGA-II multi-objective genetic algorithm that MoGA weights toward accuracy, latency, and parameter count."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies MobileNetV2 and its inverted residual blocks, the layer primitive of the search space and a CPU-latency baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the mobile-GPU inference engine used to construct the layerwise latency lookup table and report on-device GPU latencies."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the mobile-GPU runtime used for the final latency comparisons in the results table."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the ImageNet dataset and protocol on which all accuracy numbers are measured."}],"review_version":1}