Pith. sign in

REVIEW 4 major objections 5 minor 25 references

HM-NAS: Efficient Neural Architecture Search via Hierarchical Masking

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read HM-NAS replaces hand-coded search heuristics with learned masks and finds a 1.8M-parameter CIFAR-10 net at 2.41% error.

desk verdict HM-NAS is a solid extension of weight-sharing NAS with edge-level mixing weights and hierarchical masks; the CIFAR-10 results are competitive, but the mask-selection step's use of only training loss leaves an overfitting gap that needs a validation-based baseline. read the letter →

arxiv 1909.00122 v2 pith:C7THDCWV submitted 2019-08-31 cs.LG cs.CVstat.ML

Shen Yan , Biyi Fang , Faen Zhang , Yu Zheng , Xiao Zeng , Hui Xu , Mi Zhang This is my paper · ORCID
classification cs.LGcs.CVstat.ML
keywords neuralarchitecturesearchweightsharinghierarchicalmaskingmulti-levelencodingmodelpruningsupernetdifferentiableCIFAR-10
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

HM-NAS is a neural architecture search method built on weight sharing that replaces hand-designed rules for extracting architectures with learned hierarchical masks. The paper claims these masks prune redundant operations, edges, and weights of a supernet, so the final architecture can have arbitrary numbers of incoming edges per node and multiple operations per edge. On CIFAR-10 this yields 2.41% test error with 1.8M parameters in 1.8 GPU days total, and on ImageNet 73.4% top-1 accuracy with 3.6M parameters. The central idea is to treat architecture search as model pruning: masks are optimized on the training loss of the masked supernet, and the surviving unmasked weights become the starting point for fine-tuning instead of retraining. A sympathetic reader would care because it suggests that the constraints imposed by earlier differentiable NAS methods, rather than the weight-sharing idea itself, are what limit search quality and efficiency.

What carries the argument

The machinery is a multi-level architecture encoding plus a hierarchical mask. In the supernet, $\alpha$ are real-valued operation mixing weights on each edge and $\beta$ are real-valued edge mixing weights between predecessors and intermediate nodes, so each intermediate node is a weighted sum of transformed predecessors. On top of the trained supernet, three real-valued masks $M^r_\alpha$, $M^r_\beta$, and $M^r_w$ are thresholded by a Heaviside step into binary masks and multiplied with the architecture parameters and weights; the real-valued masks are trained to minimize the training loss of the masked network, using a straight-through gradient approximation to backpropagate through the threshold. This mask search simultaneously decides how many incoming edges each node keeps, how many operations each edge keeps, and which weights survive, and the surviving unmasked weights become the fine-tuning initialization.

What would settle it

Run the hierarchical masking phase twice on CIFAR-10 with identical supernets and random seeds, once minimizing training loss and once minimizing a held-out validation loss. If the training-loss version does not match or beat the validation-loss version in test error, the assumption that training-loss mask selection generalizes is false.

Watch

Extended reading notes

Core claim

The paper's central claim is that a weight-sharing supernet with multi-level architecture encoding and learned hierarchical masks can find better architectures more efficiently than existing weight-sharing NAS. Specifically, it claims that treating architecture search as a pruning problem, optimizing binary masks over operations, edges, and weights by minimizing the training loss of the masked supernet, discovers cells in which each node may have more than two incoming edges and each edge may carry several operations, something hand-designed candidate extraction rules cannot express. On CIFAR-10 this yields 2.41% test error with 1.8M parameters and a total cost of 1.8 GPU days (0.85 for the first-order variant), versus 2.76%, 3.3M parameters and 6 GPU days for the second-order version of the differentiable baseline it extends. On ImageNet the searched model reaches 73.4% top-1 accuracy with 3.6M parameters and 482M FLOPs. The paper also claims the masked weights themselves are a good starting point, so the final network only needs fine-tuning rather than a full retrain.

Load-bearing premise

The paper selects masks by minimizing training loss on the full training set, and it assumes that what works on the training set will also work on unseen test data without a validation split for mask selection.

Editorial extensions

If this is right

  • The final architectures can contain nodes with more than two incoming edges and edges carrying multiple operations, which previous weight-sharing approaches could not express.
  • The unmasked weights left by the mask search are a good initialization, so the final model is fine-tuned for 200 epochs instead of retrained for about 600, which is what makes the total cost 1.8 GPU days.
  • Ablations show that each component contributes: multi-level encoding beats single-level encoding, learned masks beat hand-designed heuristics, and HM-NAS beats random architecture search from the same supernet.
  • The same search recipe transfers to ImageNet with competitive 73.4% top-1 accuracy while using fewer parameters and FLOPs than the compared methods.
  • The paper states that other structural constraints, such as the number of cells, channels, nodes, and the combination operation, could be relaxed by the same encoding and masking scheme, leaving this as future work.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The mask-selection criterion, minimizing the training loss of the masked supernet, is the load-bearing premise; a natural test is to compare mask selection on a held-out validation split, which the paper does not do.
  • The threshold $\tau$ is fixed and the paper notes robustness over the range [0, 1e-2]; making thresholds learnable or annealing them during mask search could improve the accuracy-efficiency tradeoff further, but this was not explored.
  • The observed pattern that edges into later intermediate nodes carry higher importance may be a transferable prior for cell design beyond NAS, though the paper does not test this outside its own searched cells.
  • Because the masks are real-valued and optimized with straight-through gradients, the approach may compose with quantization or hardware-aware pruning objectives to directly search for deployment-friendly architectures.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes HM-NAS, a weight-sharing neural architecture search method that relaxes hand-designed constraints in prior differentiable NAS approaches. It introduces a multi-level architecture encoding with separate operation-level and edge-level mixing weights, and a hierarchical masking scheme that learns binary masks over operations, edges, and network weights by minimizing the training loss of the masked supernet. After search, the final network is fine-tuned rather than retrained from scratch. On CIFAR-10, HM-NAS (second-order) reports 2.41% test error with 1.8M parameters and a total cost of 1.8 GPU days, outperforming DARTS in efficiency and parameter count; on ImageNet it reports 73.4% top-1 accuracy with 3.6M parameters. Ablation studies compare against single-level encoding, hand-designed heuristics, random architectures, and random initialization, and an architecture analysis shows the discovered cells have more than two incoming edges per node and multiple operations per edge.

Significance. If the reported results hold, HM-NAS makes a useful contribution by showing that removing hand-designed heuristics in weight-sharing NAS and replacing them with learned hierarchical masks can improve search efficiency while maintaining accuracy. The paper includes error bars on CIFAR-10, several ablations, and an analysis of the searched architectures, which are strengths. The claim of generalizing existing weight-sharing NAS approaches is credible in scope. However, the absence of released code, the single-run ImageNet result, and several underspecified hyperparameters currently limit reproducibility and make the central claims only conditionally supported.

major comments (4)
  1. [§3.3, Eq. (3) and Algorithm 1] The mask optimization objective M^r* = argmin_{M^r} L_train(P_M(α*,β*,w*)) uses only the training loss on the full training set, with no validation split, no early stopping, and no reporting of training-versus-validation error during mask training. Since M_w^r is a per-network-weight mask (Figure 2), the mask search is high-dimensional relative to the size of CIFAR-10 (≈50k training images). The central claim that learned masks select better architectures requires evidence that masks minimizing training loss also generalize. Please add an analysis comparing masks selected by training loss against masks selected by held-out validation loss, and report the train/validation curves during mask training. Without such evidence, the reported 2.41% test error could partly reflect mask overfitting rather than genuinely better architectures.
  2. [§3.2 and §4.2] The stochastic update probability p = σ(iter) for optimizing α and β is described only as 'monotonically non-increasing' with σ(0)=1, but no concrete schedule is given. This schedule controls the bilevel optimization and affects the quality of the supernet, which is the starting point for mask search. Please specify the exact schedule used in all experiments (e.g., exponential decay, step decay, the value at each epoch, and when it reaches zero). Similarly, the warm-up of 10 epochs is stated, but the mask learning rates (1e-4 for M_w, 1e-5 for M_α and M_β) and the threshold τ=5e-31 are given without sensitivity analysis; the footnote claims robustness to τ in [0, 1e-2], but no experimental evidence is shown. Without these details the method is not reproducible.
  3. [§1 and §3.3] The paper repeatedly claims that the hierarchical masks 'help correct the architecture search bias caused by bilevel optimization' of architecture parameters and network weights, but no experiment isolates or tests this claim. The ablations compare hierarchical masks against hand-designed heuristics and random architectures, but they do not compare against a supernet trained without the bilevel procedure (e.g., joint optimization or uniform sampling) to show that masks specifically compensate for bilevel bias. Please either provide direct evidence for the bias-correction mechanism or remove/weaken this claim, as it is currently unsupported.
  4. [§4.3, Table 4] The ImageNet result (73.4% top-1 accuracy) is reported from a single run, with no error bars or multiple seeds. Given the stochasticity in random initialization, the random 100-class subset used for supernet training, and the stochastic α/β updates, the ImageNet comparison to DARTS and SNAS is currently based on one realization. Please report the variance over at least three runs, or explicitly state that the reported number is a single run and discuss the expected variability.
minor comments (5)
  1. [§4.2, Table 3] The text says the searched network loses '∼1% accuracy' relative to the supernet; Table 3 shows 4.2% to 5.14%, which is 0.94 percentage points. Please use '≈0.9%' or specify that the comparison is in percentage points rather than relative error.
  2. [§4.4, Table 6] The random architecture baseline uses 2.1M parameters and 600 training epochs, whereas HM-NAS uses 1.8M parameters and 200 epochs. This is not an apples-to-apples comparison in either parameter count or training cost; please note this difference explicitly in the text to avoid misleading readers.
  3. [Figure 3] The figure caption and the text refer to 'test errors' and 'training errors' on the left and right axes, but the curves are not described in enough detail to tell which color corresponds to which metric at a glance. Please add a legend or explicit labels to the curves.
  4. [§4.2, Training Details] The mask initialization is given as 1e-2, but the threshold τ is 5e-31. Since all initialized masks are far above the threshold, the binarization initially keeps all connections; please state the intended initial state and whether the masks are real-valued throughout or are binarized only in the forward pass.
  5. [References] The related work discussion cites ProxylessNAS and FBNet, but the comparison table (Table 1) lists 'Operations' for both; please clarify that HM-NAS additionally prunes edges and weights, which is the key distinction.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: HM-NAS's accuracy claims are empirically benchmarked and its mask objective is an optimization criterion, not a disguised evaluation of the same data.

full rationale

The derivation chain in HM-NAS is self-contained for the purposes of the circularity test. The architecture search is defined in Eq. (3) as minimizing the training loss of the masked supernet; this is an optimization objective, not an evaluation procedure, and the reported CIFAR-10 and ImageNet numbers are measured on held-out test or validation sets and compared against external baselines such as DARTS, SNAS, random architecture search, and random initialization. The searched network is not a renamed version of the supernet: Table 3 separates the stages empirically, reporting 4.2% error for the supernet, 5.14% for the masked network, and 2.41% after fine-tuning, so the claimed improvement is not true by construction. The multi-level encoding and hierarchical masking are method choices rather than conclusions assumed in their own premises. The paper's assertions that hierarchical masks 'help correct the architecture search bias' and the robustness analysis that reports runs with 'best validation performance' are unsupported or weakly supported, but they are evidence-quality issues, not definitional equivalences between inputs and outputs. There are no load-bearing self-citations, no fitted parameters later relabeled as predictions, and no derived quantity that equals its own input by construction. Therefore no circular step meets the required standard.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The ledger is dominated by hand-set hyperparameters of the masking procedure and by standard NAS domain assumptions. The only ad hoc assumption specific to HM-NAS is the asserted bias-correction effect of masks, which lacks dedicated evidence. No new physical or external entity is introduced.

free parameters (5)
  • Mask threshold tau = 5e-31
    Hand-set threshold in Eq. (4); determines when a mask becomes zero. Robustness is claimed in a footnote but no sensitivity experiments are shown.
  • Mask initialization = 1e-2
    Hand-set initial value for M_alpha, M_beta, and M_w in Section 4.2; with tau equal to 5e-31 all masks start active, so initialization affects which weights survive early training.
  • Mask learning rates = 1e-4 for M_w, 1e-5 for M_alpha and M_beta, decayed 10x after 10 epochs
    Hand-set optimization schedule in Section 4.2; central to whether masks converge to a good binary selection.
  • Architecture parameter update probability schedule sigma(iter) = monotone non-increasing, sigma(0)=1, not fully specified
    Introduced in Section 3.2 to decide how often alpha and beta are updated; the paper does not give the concrete formula, making it a hidden tuning choice.
  • Warm-up epochs and data split = 10 epochs warm-up; 80/20 train/validation split
    Chosen in Section 4.2; affects the quality of supernet weights and architecture parameters.
assumptions (4)
  • domain assumption A weight-sharing supernet trained once yields candidate architectures whose inherited weights are predictive of standalone performance.
    Entered in Sections 1 and 3.1; no analysis in this paper validates this transfer, but it is standard in the DARTS and SNAS literature.
  • ad hoc to paper Bilevel optimization of architecture parameters and network weights introduces bias, and adding learned masks corrects this bias.
    Stated in Sections 1 and 5 with references [11-13]; no dedicated experiment in the paper supports the corrective effect.
  • standard math The straight-through estimator from BinaryConnect provides usable gradients for binary masks.
    Used in Section 3.3 Eq. (4) and the gradient update; justified by prior work [16-18].
  • domain assumption Cell-based search space with 7 nodes and the DARTS operation set is a sufficient and appropriate search space.
    Adopted in Sections 3.1 and 4.1 from [6-8]; the final architecture quality depends on this space.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HM-NAS: Efficient Neural Architecture Search via Hierarchical Masking." pith.science (2026). https://pith.science/paper/C7THDCWV

@misc{pith2026190900122,
  author       = {Pith},
  title        = {Pith review of: HM-NAS: Efficient Neural Architecture Search via Hierarchical Masking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/C7THDCWV}},
  note         = {Machine review of arXiv:1909.00122}
}
read the original abstract

The use of automatic methods, often referred to as Neural Architecture Search (NAS), in designing neural network architectures has recently drawn considerable attention. In this work, we present an efficient NAS approach, named HM- NAS, that generalizes existing weight sharing based NAS approaches. Existing weight sharing based NAS approaches still adopt hand-designed heuristics to generate architecture candidates. As a consequence, the space of architecture candidates is constrained in a subset of all possible architectures, making the architecture search results sub-optimal. HM-NAS addresses this limitation via two innovations. First, HM-NAS incorporates a multi-level architecture encoding scheme to enable searching for more flexible network architectures. Second, it discards the hand-designed heuristics and incorporates a hierarchical masking scheme that automatically learns and determines the optimal architecture. Compared to state-of-the-art weight sharing based approaches, HM-NAS is able to achieve better architecture search performance and competitive model evaluation accuracy. Without the constraint imposed by the hand-designed heuristics, our searched networks contain more flexible and meaningful architectures that existing weight sharing based NAS approaches are not able to discover.

Figures

Figures reproduced from arXiv: 1909.00122 by the authors.

Figure 1
Figure 1. The pipelines of (a) existing weight sharing based [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Illustration of the iterative hierarchical masking process on a single cell. In this example, each edge has 3 [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Training curves of HM-NAS (in blue color) and [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Details of learned cell for CIFAR-10. (a) cell structure. (b) number of input edges of four intermediate nodes. (c) (a)(b) (c) [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Details of learned cell for ImageNet. (a) cell structure. (b) number of input edges of four intermediate nodes. (c) [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Robustness of learned edge importance of (a) [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

25 extracted references · 16 canonical work pages

  1. [1]

    Learning transferable architectures for scalable image recognition

    Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In CVPR, pages 8697– 8710, 2018

    work page 2018

  2. [2]

    Detnas: Neural architecture search on object detection

    Yukang Chen, Tong Yang, Xiangyu Zhang, Gaofeng Meng, Chunhong Pan, and Jian Sun. Detnas: Neural architecture search on object detection. arXiv preprint arXiv:1903.10979, 2019

    arXiv 1903

  3. [3]

    Fast neural architecture search of compact semantic segmentation models via auxiliary cells

    Vladimir Nekrasov, Hao Chen, Chunhua Shen, and Ian Reid. Fast neural architecture search of compact semantic segmentation models via auxiliary cells. In CVPR, pages 9126–9135, 2019

    work page 2019

  4. [4]

    Efficient neural architecture search via pa- rameters sharing

    Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture search via pa- rameters sharing. In ICML, 2018

    work page 2018

  5. [5]

    Designing neural network architectures using reinforcement learning

    Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar. Designing neural network architectures using reinforcement learning. In ICLR, 2017

    work page 2017

  6. [6]

    Regularized evolution for image classifier architecture search

    Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. Regularized evolution for image classifier architecture search. In AAAI, 2019

    work page 2019

  7. [7]

    Darts: Differentiable architecture search

    Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. In ICLR, 2019

    2019

  8. [8]

    Snas: stochastic neural architecture search

    Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. Snas: stochastic neural architecture search. In ICLR, 2019

    work page 2019

Show all 25 references
  1. [9]

    Proxylessnas: Direct neural architecture search on target task and hardware

    Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search on target task and hardware. In ICLR, 2019

    2019

  2. [10]

    Ritchie, and Nick Weston

    Andrew Brock, Theodore Lim, James M. Ritchie, and Nick Weston. Smash: One-shot model architecture search through hypernetworks. ICLR, 2018

    2018

  3. [11]

    Evaluating the search phase of neural architecture search

    Christian Sciuto, Kaicheng Yu, Martin Jaggi, Claudiu Musat, and Mathieu Salzmann. Evaluating the search phase of neural architecture search. arXiv preprint arXiv:1902.08142, 2019

    1902 arXiv

  4. [12]

    Understanding and simplifying one-shot architecture search

    Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoc Le. Understanding and simplifying one-shot architecture search. In ICML, pages 549–558, 2018

    2018

  5. [13]

    Single path one- shot neural architecture search with uniform sampling

    Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. Single path one- shot neural architecture search with uniform sampling. arXiv preprint arXiv:1904.00420, 2019

    1904 arXiv

  6. [14]

    Neural architecture optimization

    Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie-Yan Liu. Neural architecture optimization. In Advances in neural information processing systems , pages 7816–7827, 2018

    2018

  7. [15]

    Fbnet: Hardware- aware efficient convnet design via differentiable neural architecture search

    Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Va- jda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware- aware efficient convnet design via differentiable neural architecture search. In Proceedings of the IEEE Con- ference on Com...

    2019

  8. [16]

    Binaryconnect: Training deep neural networks with binary weights during propagations

    Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. In Advances in neural information processing systems, pages 3123– 3131, 2015

    2015

  9. [17]

    Piggyback: Adapting a single network to multiple tasks by learning to mask weights

    Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggyback: Adapting a single network to multiple tasks by learning to mask weights. In ECCV, pages 67–82, 2018

    2018

  10. [18]

    Binarized neural networks

    Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Binarized neural networks. In Advances in neural information process- ing systems, pages 4107–4115, 2016

    2016

  11. [19]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2015

    2015

  12. [20]

    Improved regularization of convolutional neural networks with cutout

    Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552, 2017

    2017 arXiv

  13. [21]

    Densely connected convolutional networks

    Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR, pages 4700–4708, 2017

    2017

  14. [22]

    ImageNet: A Large-Scale Hierarchical Image Database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR, 2009

    2009

  15. [23]

    Mobilenets: Efficient convolutional neural networks for mobile vision appli- cations

    Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision appli- cations. arXiv preprint arXiv:1704.04861, 2017

    2017 arXiv

  16. [24]

    Mobilenetv2: Inverted residuals and linear bottlenecks

    Mark Sandler, Andrew Howard, Menglong Zhu, An- drey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In CVPR, pages 4510–4520, 2018

    2018

  17. [25]

    Random search and reproducibility for neural architecture search

    Liam Li and Ameet Talwalkar. Random search and reproducibility for neural architecture search. arXiv preprint arXiv:1902.07638, 2019

    1902 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.