{"id":"9c6f04e1-3ee3-4199-9a38-01f2626fb55d","arxiv_id":"2411.14296","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A new neural architecture search pipeline that averages multiple one-shot networks and augments the predictor data achieves a 0.9802 ROC-AUC for chip routing hotspot detection.","lead":"This paper introduces SOAP-NAS, a neural architecture search method that trains five one-shot networks, combines their outputs, and augments the data to find a better model for predicting routing hotspots in chip layouts. The resulting model, SOAPNet, reaches a detection accuracy of 0.98 ROC-AUC, slightly above prior machine learning baselines.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 0.9802 AUC claim is not tied to any stated train/test split; Table IV's 0.889 on the 74-design set suggests the headline result depends on an unspecified evaluation protocol.","rationale":"The paper's central claim is the 0.9802 ROC-AUC of SOAPNet. For that claim to be true, the final architecture must be evaluated on data not used to train the one-shot networks, train the predictor, or select the architecture. The manuscript does not describe any such split. Section IV-A describes a dataset of 7,000 placements but says nothing about partitioning; Section IV-B then reports AUC 'among all 7k evaluated placements.' This is a load-bearing gap because the entire contribution is an empirical result, and an in-sample evaluation would invalidate it. Table IV adds direct evidence of protocol sensitivity: the same method is listed as 0.9802 on the 7k set but 0.889 on the standard 74-design set. While part of this difference might come from dataset scope, the large drop—and the paper's failure to explain the relationship between the two evaluations—makes the 0.9802 figure untrustworthy until the evaluation protocol is specified. I considered the reader's weaker-assumption concern about the one-shot predictor being a biased proxy for standalone performance. That is a genuine risk, especially given the reported 0.52–0.65 correlations, but it is secondary because the paper claims to validate candidates by standalone training (Section III-D). The more immediate threat is that no independent test set is defined at all. The proposed concrete check—design-disjoint holdout evaluation—would settle whether the headline number is a real predictive performance or an artifact of the evaluation protocol. Since this is a condition that can be satisfied by additional reporting and rerunning, the existing CONDITIONAL verdict remains appropriate; the condition should now explicitly include a design-disjoint split and clarification of Table III versus Table IV. For these reasons, I do not move the verdict, but I identify a sharper condition than the reader's stated weakest assumption.","tokens_in":9201,"tokens_out":5672,"duration_ms":52412,"concrete_test":"Ask the authors to report the exact split: how many of the 7k placements are used for one-shot training, predictor training, and final evaluation, and whether any designs are held out. Then rerun SOAP-NAS with a design-disjoint split (e.g., hold out all placements from 10 of the 74 designs) and compute ROC-AUC on those held-out designs only. If the held-out AUC falls materially below 0.9802—or below the NAS-crafted baseline—the headline result should be revised to specify the in-sample protocol.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that SOAPNet achieves 0.9802 ROC-AUC on the 7k placement dataset, but the paper never states how the 7k placements are divided among one-shot training, predictor training, standalone validation, and final evaluation. Section III-D says the one-shot network is trained and then a validation set of (network, AUC) pairs is generated; Section IV-B reports 'an impressive AUC of 0.9802 among all 7k evaluated placements.' If the placements used to train the one-shot networks and the final SOAPNet overlap with the 'evaluated' placements, the reported AUC is an in-sample fit, not a predictive-accuracy estimate. The inconsistency between Table III (SOAPNet 0.9802 on 7k) and Table IV (SOAP-NAS 0.889 on the standard 74-design set) reinforces this concern: the headline number is protocol-dependent, and the protocol is not described. Until a design-disjoint holdout is specified, the claim that SOAPNet 'achieves 0.9802' is unverified. The reader's concern about predictor bias is real, but it is secondary: even an unbiased predictor cannot support a headline number if the evaluation set is not shown to be independent of training and search.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes SOAP-NAS, a neural architecture search method for DRC hotspot detection. It first selects NASBench101 among three search spaces by comparing one-shot NAS correlations. It then trains k=5 one-shot networks, queries them to build a dataset of (architecture, ROC-AUC) pairs, augments the dataset 7x by adding noise whose variance is fitted from standalone retraining, trains an XGBoost predictor, and uses the predictor to select a final architecture (SOAPNet). The paper reports that SOAPNet achieves 0.9802 ROC-AUC and 0.461 ms query time on a 7k-placement dataset, and that this is 40% closer to the ideal ROC-AUC than the previous NAS-crafted baseline. The methodology is evaluated against RouteNet, PROS, cGAN, and NAS-crafted baselines, with per-design results on 74 designs.","tokens_in":9478,"tokens_out":4931,"duration_ms":43003,"significance":"The combination of k-shot smoothing and variance-matched augmentation is a reasonable and potentially reusable idea for NAS when the search metric has high variance and differs from the training loss. The paper also provides a useful comparison of three search spaces for this EDA task and reports query-time advantages. However, the central empirical claim is not currently verifiable: the paper does not define the train/validation/test split for the 7k-placement dataset, the single reported AUC has no error bars, and one baseline is implausibly below random. These problems must be fixed before the claimed state-of-the-art result can be accepted. The paper does not ship code, data, or configuration files, which further limits reproducibility.","major_comments":[{"comment":"The headline result of Table III, an ROC-AUC of 0.9802 \"among all evaluated 7k evaluated placements,\" is not tied to any stated train/validation/test split. Section III-D describes generating a validation set of (network, AUC) pairs from the one-shot network and training candidates from scratch to check agreement, but it does not state whether the placements used to train the one-shot networks and the predictor are disjoint from the placements used to compute the Table III AUC. If the \"evaluated\" placements overlap with those used for one-shot training, predictor training, or architecture selection, the reported AUC is an in-sample measure rather than a predictive-accuracy estimate. The discrepancy with Table IV (SOAP-NAS 0.889 on the 74-design set) reinforces this concern. Please specify the exact split (e.g., design-disjoint training/validation/test) and report the final AUC on a held-out set of placements not used in any training or selection step.","section":"Section IV-B and Section III-D"},{"comment":"Table IV reports PROS with an ROC-AUC of 0.487 on all 74 designs and 0.478-0.489 on the per-design columns, which is below the random-guessing value of 0.5. This is an unexplained result for a published FCN baseline and is inconsistent with Table III, where the reproduced PROS is 0.9580. Either the reproduction protocol is broken (e.g., label misalignment or a sign error) or the comparison protocol differs between tables. The paper must explain or correct this baseline, because a below-random baseline artificially inflates the apparent improvement of SOAPNet.","section":"Table IV"},{"comment":"The key hyperparameters k=5 (number of one-shot networks) and x=7 (augmentation factor) are selected through the ablation study in Figures 6 and 7 on the same task and dataset used to report the final 0.9802 AUC. No nested validation procedure is described, so the final number is subject to selection bias. Please report the final performance with k and x fixed before seeing the test set, or provide a sensitivity analysis over k and x on a separate validation split.","section":"Section IV-D"},{"comment":"The predictor's Pearson correlation, 0.65 after smoothing and augmentation, is reported without an error bar or design-disjoint evaluation. Because the final architecture is chosen by this predictor, a correlation of 0.65 leaves substantial room for ranking errors; the paper should report the variance of the final SOAPNet AUC over multiple runs of the entire SOAP-NAS pipeline, and should state how many standalone retraining runs were used to verify the selected architecture.","section":"Section III-F"}],"minor_comments":[{"comment":"The \"40% closer to ideal\" figure is not exactly supported by Table III: (0.9802 - 0.9627)/(1 - 0.9627) is about 0.47, not 0.40. Please clarify the calculation or correct the percentage.","section":"Abstract and Section IV-C"},{"comment":"The header \"Kendal Tau\" should be \"Kendall Tau.\"","section":"Table I"},{"comment":"There is a typo, \"encoporates,\" and the reference given for TransNASBench is actually NasBench-301; the correct TransNASBench reference should be provided.","section":"Section III-A"},{"comment":"The sentence \"keeping the top performance value of networks appearing in two or more sets\" is ambiguous; please state whether the maximum, mean, or median across one-shot networks is used for the combined dataset.","section":"Section III-E"},{"comment":"Please report the number of standalone trained networks and the standard error of the mean, not just the variance, so that the reader can judge the precision of the variance estimates.","section":"Table II"},{"comment":"The caption says \"Curves do not intersect outside the viewing region,\" which is confusing; please state clearly whether the full ROC curves cross anywhere outside the cropped area.","section":"Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is an empirical conference-style contribution; the novelty is incremental but acceptable if the empirical claims hold. The main barrier is not novelty but verifiability: the 7k evaluation protocol must be fully specified, the PROS baseline must be corrected or explained, and error bars or repeated-run statistics should be provided. I would welcome a revision that addresses these points with a design-disjoint holdout evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The k-shot smoothing idea is worth a look. Training several one-shot supernets and merging their queried predictions is a sensible way to attack the variance that makes ROC-AUC a noisy search target, and I don't know of prior work doing exactly that for routability prediction. The ablation study is honest: it shows correlation climbing from k=1 to k=5 and plateauing, and the augmentation factor x=7 similarly. That is real empirical work, and the across-search-space comparison (NASBench101, ShuffleNetV2, TransNASBench) gives the paper useful scope. The reported query time of 0.461 ms is also a plus for the EDA use case.\n\nThe soft spots are serious. The largest is the evaluation protocol. The paper says the one-shot network is trained, a validation set of (network, AUC) pairs is generated, and later SOAPNet \"achieves an ROC-AUC of 0.9802 among all 7k evaluated placements\" — but it never states how those 7k placements are divided among one-shot training, predictor training, standalone validation, and final evaluation. If the placements used to train the one-shot networks and to select the final architecture overlap with the \"evaluated\" placements, 0.9802 is an in-sample fit, not a predictive accuracy. Table IV deepens the worry: SOAP-NAS scores 0.889 on the standard 74-design set, a long way from 0.9802. That gap is not necessarily damning — the 7k set is meant to be a new, harder benchmark — but without a stated design-disjoint holdout, the headline number is protocol-dependent and unverified. The authors need to say exactly which placements were held out for the reported AUC, and ideally report per-design or per-benchmark results on the 7k set the way Table IV does for the 74 designs.\n\nSecondary issues: the predictor's Pearson correlation is only 0.65 after smoothing and augmentation, which is weak even for ranking; the paper acknowledges this but does not discuss how much it undermines final architecture selection. The PROS baseline at 0.487 AUC on the 74-design set falls below random, which should have been flagged as a reproduction anomaly. There are no error bars anywhere, and k=5 and x=7 are tuned on the same task and dataset they are evaluated on. No code or data is released.\n\nIn proportion: the method is plausible, the components are not exotic, and the ablation supports the smoothing/augmentation story. The missing split description is the load-bearing flaw, not the idea. This paper should go to peer review, but with a firm request for a design-disjoint evaluation and release of artifacts. A serious referee could turn this into a solid contribution; as is, the headline claim is not yet supportable.","headline":"The k-shot smoothing idea is a genuinely fresh twist on one-shot NAS for EDA, but the paper's headline 0.9802 AUC is not yet a verified claim because the paper never specifies how the 7k placements are split between training and evaluation.","tokens_in":9995,"tokens_out":1672,"would_cite":false,"duration_ms":17166,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that smoothing multiple one-shot networks and augmenting with variance-matched noise lets a lightweight predictor select a neural architecture that detects DRC hotspots at 0.9802 ROC-AUC in 0.461 ms.","keywords":["neural architecture search","DRC hotspot detection","routability prediction","one-shot NAS","ROC-AUC","data augmentation","electronic design automation","XGBoost predictor"],"falsifier":"Retrain SOAPNet from scratch, say 10 times, on the same 7,000-placement split and compare the distribution of ROC-AUC to the reported 0.9802; then take the predictor's top-10 candidates, train each standalone, and check whether the predictor's ranking matches the standalone ranking. If the retrained SOAPNet mean drops below roughly 0.97, or if a lower-ranked candidate consistently beats the selected one, the unbiased-proxy assumption fails.","tokens_in":9033,"feed_emoji":"🔍","tokens_out":5980,"duration_ms":48840,"temperature":0.7,"pith_summary":"SOAP-NAS is a neural architecture search method built for routability prediction, where the standard NAS trick of training one shared one-shot supernetwork fails because it optimizes accuracy while the target metric, ROC-AUC, is noisier and not directly trained. The paper argues that training several one-shot networks and merging their queried scores ('smoothing'), then inflating the resulting training set with variance-matched noise ('augmentation'), lets a simple XGBoost predictor rank candidate architectures reliably. The final architecture, SOAPNet, reaches an ROC-AUC of 0.9802 on a 7,000-placement DRC hotspot dataset, which the paper reports as 40% closer to the ideal value of 1.0 than the best prior NAS-crafted model, with a query time of 0.461 ms. If correct, this closes a known gap between what one-shot NAS can search and what routability engineers actually care about.","feed_headline":"Smoothing-and-augmented NAS predicts DRC hotspots at 0.9802 AUC","feed_subtitle":"Training five one-shot nets and adding variance-matched noise closes 40% of the gap to perfect hotspot detection.","key_machinery":"The central mechanism is the 'smooth one-shot augmented predictor' pipeline. Smoothing consists of training several one-shot supernetworks, sampling the same candidate set from each, and combining the queried scores so that an architecture's score is the maximum among the one-shot models that evaluated it, which damps the effect of random weight initialization on ROC-AUC. Augmentation then fits a noise model to the variance of ROC-AUC observed by retraining 50 standalone architectures 10 times each, and adds samples from that noise distribution to multiply the dataset by a factor of 7. These two steps produce a training set that mimics standalone-training outcomes, on which the XGBoost predictor selects the final architecture rather than ranking candidates by the noisy one-shot query itself.","core_discovery":"On its own terms, the paper's claim is that a NAS pipeline called SOAP-NAS solves the metric mismatch in DRC hotspot detection: instead of querying a single one-shot network for an architecture that maximizes ROC-AUC, SOAP-NAS trains $k=5$ one-shot networks from different random initializations, queries all of them, keeps the best score for architectures appearing in multiple query sets, and then augments this combined dataset sevenfold by adding noise whose magnitude matches the measured run-to-run variance of ROC-AUC. The resulting dataset trains an XGBoost predictor whose ranking correlation with standalone-trained architectures reaches 0.65 Pearson, up from 0.52 for the raw one-shot query, and the predictor's top pick, SOAPNet, achieves 0.9802 ROC-AUC on 7,000 placements from 74 designs while answering in 0.461 ms.","pith_inferences":["The smoothing step is effectively an ensemble in score space; the same treatment could reduce rank noise in other NAS objectives that are high-variance or weakly correlated with the training loss, such as latency, energy, or robustness metrics.","The augmentation scheme assumes a single global variance for ROC-AUC, but the variance is likely architecture-dependent; a natural extension is per-candidate noise models, which could push the predictor correlation above 0.65 and possibly yield an even better final AUC.","Because the 7,000-placement dataset is compiled from public benchmarks (ISCAS'89, ITC'99, IWLS'05, ISPD'15) with a 45nm library, it could become a reusable benchmark for routability prediction; if it does, fair comparisons will need to fix the train/test design split and hardware for query-time measurements."],"forward_implications":["SOAPNet's 0.9802 ROC-AUC on the 7,000-placement dataset sets a new reported state of the art for ML-based DRC hotspot detection, improving on the best NAS-crafted baseline by closing 40% of the gap to 1.0.","With 0.461 ms per query, the architecture is fast enough to be embedded in iterative placement-and-routing loops that issue many hotspot queries per design.","The $k=5$ and $7\\times$ augmentation settings transfer across NASBench101, ShuffleNetV2, and TransNASBench search spaces, so future users of SOAP-NAS can reuse these hyperparameters without re-tuning.","End-to-end, SOAP-NAS is a one-time search: the final architecture is a fixed network that can be applied to many circuit designs, amortizing the cost of training five one-shot supernetworks."],"supporting_citations":[{"why":"Provides the RouteNet baseline, the first FCN for DRC hotspot detection, which SOAPNet is compared against.","marker":"[1]"},{"why":"Supplies the NAS-crafted baseline, the closest prior NAS-based routability predictor that SOAP-NAS improves by 40% closer to ideal.","marker":"[2]"},{"why":"Provides the PROS baseline FCN for routability optimization used in the comparison tables.","marker":"[4]"},{"why":"Supplies the conditional GAN baseline (cGAN) whose hotspot detection performance is compared in the evaluation.","marker":"[6]"},{"why":"Provides the single-path one-shot NAS method and uniform path sampling used to train the supernetworks in SOAP-NAS.","marker":"[22]"},{"why":"Defines the NASBench101 search space with 432k architectures, selected and used for the smoothing and correlation experiments.","marker":"[23]"},{"why":"Supplies the TransNASBench search space, one of the three search spaces compared in the study.","marker":"[25]"}],"fun_headline_variants":["SOAP-NAS predicts DRC hotspots 40% closer to ideal AUC","Augmented NAS yields 0.9802 AUC for DRC hotspot detection","One-shot NAS with variance-matched noise hits 0.9802 AUC","SOAP-NAS: 40% closer to perfect hotspot detection","Smooth one-shot NAS improves hotspot prediction by 40%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the variance-matched, smoothed scores used to train the predictor are an unbiased proxy for what a candidate architecture achieves when trained from scratch by itself; the paper's own numbers (raw one-shot Pearson 0.52, predictor 0.65) leave room for ranking error, and a biased proxy could make SOAPNet not the genuinely best architecture.","fun_headline_variants_meta":{"raw":{"variants":["SOAP-NAS predicts DRC hotspots 40% closer to ideal AUC","Augmented NAS yields 0.9802 AUC for DRC hotspot detection","One-shot NAS with variance-matched noise hits 0.9802 AUC","SOAP-NAS: 40% closer to perfect hotspot detection","Smooth one-shot NAS improves hotspot prediction by 40%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000883,"raw_usage":{"total_tokens":3801,"prompt_tokens":917,"completion_tokens":2884,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":533,"completion_tokens_details":{"reasoning_tokens":2789}},"tokens_in":533,"tokens_out":2884,"duration_ms":19489,"temperature":1.0,"reasoning_tokens":2789,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:19:21.768855+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain SOAPNet from scratch, say 10 times, on the same 7,000-placement split and compare the distribution of ROC-AUC to the reported 0.9802; then take the predictor's top-10 candidates, train each standalone, and check whether the predictor's ranking matches the standalone ranking. If the retrained SOAPNet mean drops below roughly 0.97, or if a lower-ranked candidate consistently beats the selected one, the unbiased-proxy assumption fails.","supporting_citations":[{"cited_title":"Routenet: Routability prediction for mixed-size designs using convolutional neural network,","cited_arxiv_id":null,"evidence_quote":"Provides the RouteNet baseline, the first FCN for DRC hotspot detection, which SOAPNet is compared against."},{"cited_title":"Automatic routability predictor development using neural architecture search,","cited_arxiv_id":null,"evidence_quote":"Supplies the NAS-crafted baseline, the closest prior NAS-based routability predictor that SOAP-NAS improves by 40% closer to ideal."},{"cited_title":"Pros: A plug-in for routability optimization applied in the state-of-the-art commercial eda tool using deep learning,","cited_arxiv_id":null,"evidence_quote":"Provides the PROS baseline FCN for routability optimization used in the comparison tables."},{"cited_title":"Painting on placement: Forecasting routing congestion using conditional generative adversarial nets,","cited_arxiv_id":null,"evidence_quote":"Supplies the conditional GAN baseline (cGAN) whose hotspot detection performance is compared in the evaluation."}],"review_version":1}