{"id":"a27e8101-7aea-4ee2-acf2-e4dcc9c59fc6","arxiv_id":"1909.00122","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"HM-NAS reaches 2.41% test error on CIFAR-10 with 1.8M parameters and 1.8 GPU days, and 73.4% top-1 on ImageNet, by learning hierarchical masks over a weight-sharing supernet.","lead":"HM-NAS searches for neural network architectures by learning masks that prune unimportant operations, edges, and weights from an oversized network, instead of using hand-made rules. It matches or beats older search methods such as DARTS on CIFAR-10 and ImageNet while using fewer parameters and less training time.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Mask selection uses only training loss on the full training set with one learned mask per network weight, so the search phase can overfit and the reported accuracy may reflect memorized masks rather than better architectures.","rationale":"The reader's weakest assumption identifies exactly the step I consider most load-bearing: masks are chosen by minimizing training loss on the full training set, with no validation-based selection. I agree with the CONDITIONAL verdict because the concern is serious but not yet demonstrated to be fatal. The paper has independent support from ablations (Tables 5-6) and a robustness analysis, and the random-initialization ablation partially addresses the confound of inherited weights. However, that ablation does not address whether the mask-selection objective itself generalizes. I considered whether the more central problem is the use of fine-tuning instead of retraining from scratch, but the random-initialization experiment shows the architecture alone reaches 2.95%, limiting the force of that objection. The training-loss mask selection remains the least protected premise: it is high-dimensional, uses no validation signal, and is the exact step where overfitting would inflate the reported results. A concrete validation-split rerun would settle whether the concern lands; absent that, CONDITIONAL remains the right verdict.","tokens_in":11937,"tokens_out":7491,"duration_ms":109835,"concrete_test":"Re-run the CIFAR-10 pipeline with mask selection on a held-out validation split (the same 20% used for alpha/beta, or a fresh split) instead of the full training set: freeze the supernet weights, optimize M^r by minimizing L_val(P_M(alpha*, beta*, w*)), then fine-tune the resulting architecture on the full training set and compare test error to Table 2. Also record train and validation loss curves during the mask phase. If the validation-selected run is not within roughly 0.1-0.2% of 2.41%, the training-loss objective is overfitting and the central efficiency/accuracy claim needs revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is the hierarchical-mask objective in Eq. (3): M^r* = argmin_{M^r} L_train(P_M(alpha*, beta*, w*)), together with Algorithm 1's search loop, which updates masks using gradients of L_train on the entire training set. This is not a low-dimensional selection: M_w^r is a per-network-weight mask (Figure 2), so the mask phase optimizes on the order of the supernet's number of weights against only about 50k CIFAR-10 training images, with no validation split, no early stopping, and no report of train-versus-test error during mask training. The central claim that learned masks select a better architecture requires these masks to transfer from training loss to test accuracy. The ablations compare against random architecture and random initialization, but they never compare masks selected by training loss against masks selected by held-out validation loss. Notably, alpha and beta during supernet training are optimized on a 20% validation split, while the mask step — the architectural-selection step most exposed to overfitting — has no validation signal. If the masks memorize training examples, the reported 2.41% test error could be substantially attributable to the inherited supernet weights and the fine-tuning schedule, rather than to genuinely better discovered architectures.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HM-NAS, a weight-sharing neural architecture search method that relaxes hand-designed constraints in prior differentiable NAS approaches. It introduces a multi-level architecture encoding with separate operation-level and edge-level mixing weights, and a hierarchical masking scheme that learns binary masks over operations, edges, and network weights by minimizing the training loss of the masked supernet. After search, the final network is fine-tuned rather than retrained from scratch. On CIFAR-10, HM-NAS (second-order) reports 2.41% test error with 1.8M parameters and a total cost of 1.8 GPU days, outperforming DARTS in efficiency and parameter count; on ImageNet it reports 73.4% top-1 accuracy with 3.6M parameters. Ablation studies compare against single-level encoding, hand-designed heuristics, random architectures, and random initialization, and an architecture analysis shows the discovered cells have more than two incoming edges per node and multiple operations per edge.","tokens_in":12228,"tokens_out":4338,"duration_ms":70341,"significance":"If the reported results hold, HM-NAS makes a useful contribution by showing that removing hand-designed heuristics in weight-sharing NAS and replacing them with learned hierarchical masks can improve search efficiency while maintaining accuracy. The paper includes error bars on CIFAR-10, several ablations, and an analysis of the searched architectures, which are strengths. The claim of generalizing existing weight-sharing NAS approaches is credible in scope. However, the absence of released code, the single-run ImageNet result, and several underspecified hyperparameters currently limit reproducibility and make the central claims only conditionally supported.","major_comments":[{"comment":"The mask optimization objective M^r* = argmin_{M^r} L_train(P_M(α*,β*,w*)) uses only the training loss on the full training set, with no validation split, no early stopping, and no reporting of training-versus-validation error during mask training. Since M_w^r is a per-network-weight mask (Figure 2), the mask search is high-dimensional relative to the size of CIFAR-10 (≈50k training images). The central claim that learned masks select better architectures requires evidence that masks minimizing training loss also generalize. Please add an analysis comparing masks selected by training loss against masks selected by held-out validation loss, and report the train/validation curves during mask training. Without such evidence, the reported 2.41% test error could partly reflect mask overfitting rather than genuinely better architectures.","section":"§3.3, Eq. (3) and Algorithm 1"},{"comment":"The stochastic update probability p = σ(iter) for optimizing α and β is described only as 'monotonically non-increasing' with σ(0)=1, but no concrete schedule is given. This schedule controls the bilevel optimization and affects the quality of the supernet, which is the starting point for mask search. Please specify the exact schedule used in all experiments (e.g., exponential decay, step decay, the value at each epoch, and when it reaches zero). Similarly, the warm-up of 10 epochs is stated, but the mask learning rates (1e-4 for M_w, 1e-5 for M_α and M_β) and the threshold τ=5e-31 are given without sensitivity analysis; the footnote claims robustness to τ in [0, 1e-2], but no experimental evidence is shown. Without these details the method is not reproducible.","section":"§3.2 and §4.2"},{"comment":"The paper repeatedly claims that the hierarchical masks 'help correct the architecture search bias caused by bilevel optimization' of architecture parameters and network weights, but no experiment isolates or tests this claim. The ablations compare hierarchical masks against hand-designed heuristics and random architectures, but they do not compare against a supernet trained without the bilevel procedure (e.g., joint optimization or uniform sampling) to show that masks specifically compensate for bilevel bias. Please either provide direct evidence for the bias-correction mechanism or remove/weaken this claim, as it is currently unsupported.","section":"§1 and §3.3"},{"comment":"The ImageNet result (73.4% top-1 accuracy) is reported from a single run, with no error bars or multiple seeds. Given the stochasticity in random initialization, the random 100-class subset used for supernet training, and the stochastic α/β updates, the ImageNet comparison to DARTS and SNAS is currently based on one realization. Please report the variance over at least three runs, or explicitly state that the reported number is a single run and discuss the expected variability.","section":"§4.3, Table 4"}],"minor_comments":[{"comment":"The text says the searched network loses '∼1% accuracy' relative to the supernet; Table 3 shows 4.2% to 5.14%, which is 0.94 percentage points. Please use '≈0.9%' or specify that the comparison is in percentage points rather than relative error.","section":"§4.2, Table 3"},{"comment":"The random architecture baseline uses 2.1M parameters and 600 training epochs, whereas HM-NAS uses 1.8M parameters and 200 epochs. This is not an apples-to-apples comparison in either parameter count or training cost; please note this difference explicitly in the text to avoid misleading readers.","section":"§4.4, Table 6"},{"comment":"The figure caption and the text refer to 'test errors' and 'training errors' on the left and right axes, but the curves are not described in enough detail to tell which color corresponds to which metric at a glance. Please add a legend or explicit labels to the curves.","section":"Figure 3"},{"comment":"The mask initialization is given as 1e-2, but the threshold τ is 5e-31. Since all initialized masks are far above the threshold, the binarization initially keeps all connections; please state the intended initial state and whether the masks are real-valued throughout or are binarized only in the forward pass.","section":"§4.2, Training Details"},{"comment":"The related work discussion cites ProxylessNAS and FBNet, but the comparison table (Table 1) lists 'Operations' for both; please clarify that HM-NAS additionally prunes edges and weights, which is the key distinction.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for the journal and the central efficiency/accuracy claims are promising, but the mask-overfitting concern is load-bearing and the current manuscript does not rule it out. The authors should be asked to add a validation-loss-based mask comparison and report train/validation curves during mask training. Also, the lack of code release and the underspecified σ(iter) schedule are reproducibility issues that should be addressed in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"HM-NAS is a genuine extension of DARTS/SNAS: it adds edge-level mixing weights (Eq. 2) and a hierarchical mask over operations, edges, and weights, and it removes the hand-designed 'top-2 edges, one op per edge' heuristic. The CIFAR-10 numbers are good—2.41% test error with 1.8M params and 1.8 GPU days total, versus DARTS second-order at 2.76%, 3.3M, and 6 GPU days—and the ablations against single-level encoding, hand-designed heuristics, random architectures, and random initialization give the central claim real support. The ImageNet result is competitive, and the finding that learned edge importance is robust across five seeds is a nice piece of evidence.\n\nThe soft spots are real but not deal-breakers. Most importantly, the mask-selection step minimizes training loss on the full training set (Eq. 3, Algorithm 1) with no validation split and no report of train-versus-test error during mask training. Since the masks include a per-weight mask, the search phase has enough capacity to memorize, and the paper never compares against masks selected using a held-out validation loss. That is a legitimate gap. The paper's own Table 3 tempers the concern: the masked network alone (5.14%) is worse than the supernet (4.2%), and fine-tuning recovers to 2.41%, so the final result depends on the fine-tuning schedule as much as on the discovered architecture. The random-initialization ablation (2.95% vs 2.41%) shows the unmasked supernet weights help, but it does not isolate the mask-selection signal. A simple baseline—mask selection on a validation split—would settle it.\n\nOther smaller issues: no code release, ImageNet is a single run, and tau (5e-31) and the sigma schedule are described loosely. The paper's claim that hierarchical masks 'correct' bilevel-optimization bias is asserted without independent evidence; I would soften that or add a comparison.\n\nOverall, this is a solid within-subfield contribution that deserves a serious referee. It is not a paradigm shift, but it is a clean extension with careful ablations. I would send it to review, asking the authors to release code and add a validation-based mask baseline.","headline":"HM-NAS is a solid extension of weight-sharing NAS with edge-level mixing weights and hierarchical masks; the CIFAR-10 results are competitive, but the mask-selection step's use of only training loss leaves an overfitting gap that needs a validation-based baseline.","tokens_in":12717,"tokens_out":2760,"would_cite":true,"duration_ms":24113,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"HM-NAS replaces hand-coded search heuristics with learned masks and finds a 1.8M-parameter CIFAR-10 net at 2.41% error.","keywords":["neural architecture search","weight sharing","hierarchical masking","multi-level architecture encoding","model pruning","supernet","differentiable architecture search","CIFAR-10"],"falsifier":"Run the hierarchical masking phase twice on CIFAR-10 with identical supernets and random seeds, once minimizing training loss and once minimizing a held-out validation loss. If the training-loss version does not match or beat the validation-loss version in test error, the assumption that training-loss mask selection generalizes is false.","tokens_in":1772,"feed_emoji":"🧠","tokens_out":2707,"duration_ms":89197,"temperature":0.7,"pith_summary":"HM-NAS is a neural architecture search method built on weight sharing that replaces hand-designed rules for extracting architectures with learned hierarchical masks. The paper claims these masks prune redundant operations, edges, and weights of a supernet, so the final architecture can have arbitrary numbers of incoming edges per node and multiple operations per edge. On CIFAR-10 this yields 2.41% test error with 1.8M parameters in 1.8 GPU days total, and on ImageNet 73.4% top-1 accuracy with 3.6M parameters. The central idea is to treat architecture search as model pruning: masks are optimized on the training loss of the masked supernet, and the surviving unmasked weights become the starting point for fine-tuning instead of retraining. A sympathetic reader would care because it suggests that the constraints imposed by earlier differentiable NAS methods, rather than the weight-sharing idea itself, are what limit search quality and efficiency.","feed_headline":"HM-NAS: 2.41% CIFAR-10 error, 1.8M params, 1.8 GPU days","feed_subtitle":"Learned masks prune operations, edges, and weights, so the searched model fine-tunes rather than retraining.","key_machinery":"The machinery is a multi-level architecture encoding plus a hierarchical mask. In the supernet, $\\alpha$ are real-valued operation mixing weights on each edge and $\\beta$ are real-valued edge mixing weights between predecessors and intermediate nodes, so each intermediate node is a weighted sum of transformed predecessors. On top of the trained supernet, three real-valued masks $M^r_\\alpha$, $M^r_\\beta$, and $M^r_w$ are thresholded by a Heaviside step into binary masks and multiplied with the architecture parameters and weights; the real-valued masks are trained to minimize the training loss of the masked network, using a straight-through gradient approximation to backpropagate through the threshold. This mask search simultaneously decides how many incoming edges each node keeps, how many operations each edge keeps, and which weights survive, and the surviving unmasked weights become the fine-tuning initialization.","core_discovery":"The paper's central claim is that a weight-sharing supernet with multi-level architecture encoding and learned hierarchical masks can find better architectures more efficiently than existing weight-sharing NAS. Specifically, it claims that treating architecture search as a pruning problem, optimizing binary masks over operations, edges, and weights by minimizing the training loss of the masked supernet, discovers cells in which each node may have more than two incoming edges and each edge may carry several operations, something hand-designed candidate extraction rules cannot express. On CIFAR-10 this yields 2.41% test error with 1.8M parameters and a total cost of 1.8 GPU days (0.85 for the first-order variant), versus 2.76%, 3.3M parameters and 6 GPU days for the second-order version of the differentiable baseline it extends. On ImageNet the searched model reaches 73.4% top-1 accuracy with 3.6M parameters and 482M FLOPs. The paper also claims the masked weights themselves are a good starting point, so the final network only needs fine-tuning rather than a full retrain.","pith_inferences":["The mask-selection criterion, minimizing the training loss of the masked supernet, is the load-bearing premise; a natural test is to compare mask selection on a held-out validation split, which the paper does not do.","The threshold $\\tau$ is fixed and the paper notes robustness over the range [0, 1e-2]; making thresholds learnable or annealing them during mask search could improve the accuracy-efficiency tradeoff further, but this was not explored.","The observed pattern that edges into later intermediate nodes carry higher importance may be a transferable prior for cell design beyond NAS, though the paper does not test this outside its own searched cells.","Because the masks are real-valued and optimized with straight-through gradients, the approach may compose with quantization or hardware-aware pruning objectives to directly search for deployment-friendly architectures."],"forward_implications":["The final architectures can contain nodes with more than two incoming edges and edges carrying multiple operations, which previous weight-sharing approaches could not express.","The unmasked weights left by the mask search are a good initialization, so the final model is fine-tuned for 200 epochs instead of retrained for about 600, which is what makes the total cost 1.8 GPU days.","Ablations show that each component contributes: multi-level encoding beats single-level encoding, learned masks beat hand-designed heuristics, and HM-NAS beats random architecture search from the same supernet.","The same search recipe transfers to ImageNet with competitive 73.4% top-1 accuracy while using fewer parameters and FLOPs than the compared methods.","The paper states that other structural constraints, such as the number of cells, channels, nodes, and the combination operation, could be relaxed by the same encoding and masking scheme, leaving this as future work."],"supporting_citations":[{"why":"Defines the differentiable bilevel search over architecture parameters and network weights, including the hand-designed two-edge-per-node, one-operation-per-edge candidate extraction that HM-NAS generalizes.","marker":"[7]"},{"why":"Supplies a gradient-based weight-sharing baseline using single-level architecture encoding, used as the comparison for the multi-level encoding claim.","marker":"[8]"},{"why":"Formulates NAS as a model pruning problem and prunes operations; HM-NAS extends the pruning view to edges and network weights and drops retraining.","marker":"[9]"},{"why":"Provides the straight-through gradient approximation that lets the binary masks be trained end-to-end despite the Heaviside threshold.","marker":"[16]"},{"why":"Introduces the warm-start strategy for training network weights before architecture parameters, which HM-NAS adopts to stabilize supernet training.","marker":"[15]"},{"why":"Supplies evidence that random architectures are competitive and that bilevel optimization can bias search, motivating the learned-mask correction.","marker":"[11]"}],"fun_headline_variants":["HM-NAS: 2.41% error, 1.8M params, 1.8 GPU days","Hierarchical masks find flexible cells in 1.8 GPU days","NAS without heuristics: learned masks beat hand-designed rules","Mask-based NAS: fewer params, lower error, faster search"],"cache_read_input_tokens":14848,"weakest_assumption_plain":"The paper selects masks by minimizing training loss on the full training set, and it assumes that what works on the training set will also work on unseen test data without a validation split for mask selection.","fun_headline_variants_meta":{"raw":{"variants":["HM-NAS: 2.41% error, 1.8M params, 1.8 GPU days","Hierarchical masks find flexible cells in 1.8 GPU days","NAS without heuristics: learned masks beat hand-designed rules","Mask-based NAS: fewer params, lower error, faster search"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00033,"raw_usage":{"total_tokens":1854,"prompt_tokens":973,"completion_tokens":881,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":589,"completion_tokens_details":{"reasoning_tokens":798}},"tokens_in":589,"tokens_out":881,"duration_ms":35651,"temperature":1.0,"reasoning_tokens":798,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T06:00:58.530584+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the hierarchical masking phase twice on CIFAR-10 with identical supernets and random seeds, once minimizing training loss and once minimizing a held-out validation loss. If the training-loss version does not match or beat the validation-loss version in test error, the assumption that training-loss mask selection generalizes is false.","supporting_citations":[{"cited_title":"Snas: stochastic neural architecture search","cited_arxiv_id":null,"evidence_quote":"Supplies a gradient-based weight-sharing baseline using single-level architecture encoding, used as the comparison for the multi-level encoding claim."},{"cited_title":"Proxylessnas: Direct neural architecture search on target task and hardware","cited_arxiv_id":null,"evidence_quote":"Formulates NAS as a model pruning problem and prunes operations; HM-NAS extends the pruning view to edges and network weights and drops retraining."},{"cited_title":"Binaryconnect: Training deep neural networks with binary weights during propagations","cited_arxiv_id":null,"evidence_quote":"Provides the straight-through gradient approximation that lets the binary masks be trained end-to-end despite the Heaviside threshold."},{"cited_title":"Fbnet: Hardware- aware efficient convnet design via differentiable neural architecture search","cited_arxiv_id":null,"evidence_quote":"Introduces the warm-start strategy for training network weights before architecture parameters, which HM-NAS adopts to stabilize supernet training."}],"review_version":1}