REVIEW 4 major objections 6 minor 29 references
Improving Routability Prediction via NAS Using a Smooth One-shot Augmented Predictor
T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read The paper claims that smoothing multiple one-shot networks and augmenting with variance-matched noise lets a lightweight predictor select a neural architecture that detects DRC hotspots at 0.9802 ROC-AUC in 0.461 ms.
desk verdict The k-shot smoothing idea is a genuinely fresh twist on one-shot NAS for EDA, but the paper's headline 0.9802 AUC is not yet a verified claim because the paper never specifies how the 7k placements are split between training and evaluation. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the 'smooth one-shot augmented predictor' pipeline. Smoothing consists of training several one-shot supernetworks, sampling the same candidate set from each, and combining the queried scores so that an architecture's score is the maximum among the one-shot models that evaluated it, which damps the effect of random weight initialization on ROC-AUC. Augmentation then fits a noise model to the variance of ROC-AUC observed by retraining 50 standalone architectures 10 times each, and adds samples from that noise distribution to multiply the dataset by a factor of 7. These two steps produce a training set that mimics standalone-training outcomes, on which the XGBoost predictor selects the final architecture rather than ranking candidates by the noisy one-shot query itself.
What would settle it
Retrain SOAPNet from scratch, say 10 times, on the same 7,000-placement split and compare the distribution of ROC-AUC to the reported 0.9802; then take the predictor's top-10 candidates, train each standalone, and check whether the predictor's ranking matches the standalone ranking. If the retrained SOAPNet mean drops below roughly 0.97, or if a lower-ranked candidate consistently beats the selected one, the unbiased-proxy assumption fails.
Extended reading notes
Core claim
On its own terms, the paper's claim is that a NAS pipeline called SOAP-NAS solves the metric mismatch in DRC hotspot detection: instead of querying a single one-shot network for an architecture that maximizes ROC-AUC, SOAP-NAS trains $k=5$ one-shot networks from different random initializations, queries all of them, keeps the best score for architectures appearing in multiple query sets, and then augments this combined dataset sevenfold by adding noise whose magnitude matches the measured run-to-run variance of ROC-AUC. The resulting dataset trains an XGBoost predictor whose ranking correlation with standalone-trained architectures reaches 0.65 Pearson, up from 0.52 for the raw one-shot query, and the predictor's top pick, SOAPNet, achieves 0.9802 ROC-AUC on 7,000 placements from 74 designs while answering in 0.461 ms.
Load-bearing premise
The load-bearing premise is that the variance-matched, smoothed scores used to train the predictor are an unbiased proxy for what a candidate architecture achieves when trained from scratch by itself; the paper's own numbers (raw one-shot Pearson 0.52, predictor 0.65) leave room for ranking error, and a biased proxy could make SOAPNet not the genuinely best architecture.
Editorial extensions
If this is right
- SOAPNet's 0.9802 ROC-AUC on the 7,000-placement dataset sets a new reported state of the art for ML-based DRC hotspot detection, improving on the best NAS-crafted baseline by closing 40% of the gap to 1.0.
- With 0.461 ms per query, the architecture is fast enough to be embedded in iterative placement-and-routing loops that issue many hotspot queries per design.
- The $k=5$ and $7\times$ augmentation settings transfer across NASBench101, ShuffleNetV2, and TransNASBench search spaces, so future users of SOAP-NAS can reuse these hyperparameters without re-tuning.
- End-to-end, SOAP-NAS is a one-time search: the final architecture is a fixed network that can be applied to many circuit designs, amortizing the cost of training five one-shot supernetworks.
Reading between the lines
- The smoothing step is effectively an ensemble in score space; the same treatment could reduce rank noise in other NAS objectives that are high-variance or weakly correlated with the training loss, such as latency, energy, or robustness metrics.
- The augmentation scheme assumes a single global variance for ROC-AUC, but the variance is likely architecture-dependent; a natural extension is per-candidate noise models, which could push the predictor correlation above 0.65 and possibly yield an even better final AUC.
- Because the 7,000-placement dataset is compiled from public benchmarks (ISCAS'89, ITC'99, IWLS'05, ISPD'15) with a 45nm library, it could become a reusable benchmark for routability prediction; if it does, fair comparisons will need to fix the train/test design split and hardware for query-time measurements.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper proposes SOAP-NAS, a neural architecture search method for DRC hotspot detection. It first selects NASBench101 among three search spaces by comparing one-shot NAS correlations. It then trains k=5 one-shot networks, queries them to build a dataset of (architecture, ROC-AUC) pairs, augments the dataset 7x by adding noise whose variance is fitted from standalone retraining, trains an XGBoost predictor, and uses the predictor to select a final architecture (SOAPNet). The paper reports that SOAPNet achieves 0.9802 ROC-AUC and 0.461 ms query time on a 7k-placement dataset, and that this is 40% closer to the ideal ROC-AUC than the previous NAS-crafted baseline. The methodology is evaluated against RouteNet, PROS, cGAN, and NAS-crafted baselines, with per-design results on 74 designs.
Significance. The combination of k-shot smoothing and variance-matched augmentation is a reasonable and potentially reusable idea for NAS when the search metric has high variance and differs from the training loss. The paper also provides a useful comparison of three search spaces for this EDA task and reports query-time advantages. However, the central empirical claim is not currently verifiable: the paper does not define the train/validation/test split for the 7k-placement dataset, the single reported AUC has no error bars, and one baseline is implausibly below random. These problems must be fixed before the claimed state-of-the-art result can be accepted. The paper does not ship code, data, or configuration files, which further limits reproducibility.
major comments (4)
- [Section IV-B and Section III-D] The headline result of Table III, an ROC-AUC of 0.9802 "among all evaluated 7k evaluated placements," is not tied to any stated train/validation/test split. Section III-D describes generating a validation set of (network, AUC) pairs from the one-shot network and training candidates from scratch to check agreement, but it does not state whether the placements used to train the one-shot networks and the predictor are disjoint from the placements used to compute the Table III AUC. If the "evaluated" placements overlap with those used for one-shot training, predictor training, or architecture selection, the reported AUC is an in-sample measure rather than a predictive-accuracy estimate. The discrepancy with Table IV (SOAP-NAS 0.889 on the 74-design set) reinforces this concern. Please specify the exact split (e.g., design-disjoint training/validation/test) and report the final AUC on a held-out set of placements not used in any training or selection step.
- [Table IV] Table IV reports PROS with an ROC-AUC of 0.487 on all 74 designs and 0.478-0.489 on the per-design columns, which is below the random-guessing value of 0.5. This is an unexplained result for a published FCN baseline and is inconsistent with Table III, where the reproduced PROS is 0.9580. Either the reproduction protocol is broken (e.g., label misalignment or a sign error) or the comparison protocol differs between tables. The paper must explain or correct this baseline, because a below-random baseline artificially inflates the apparent improvement of SOAPNet.
- [Section IV-D] The key hyperparameters k=5 (number of one-shot networks) and x=7 (augmentation factor) are selected through the ablation study in Figures 6 and 7 on the same task and dataset used to report the final 0.9802 AUC. No nested validation procedure is described, so the final number is subject to selection bias. Please report the final performance with k and x fixed before seeing the test set, or provide a sensitivity analysis over k and x on a separate validation split.
- [Section III-F] The predictor's Pearson correlation, 0.65 after smoothing and augmentation, is reported without an error bar or design-disjoint evaluation. Because the final architecture is chosen by this predictor, a correlation of 0.65 leaves substantial room for ranking errors; the paper should report the variance of the final SOAPNet AUC over multiple runs of the entire SOAP-NAS pipeline, and should state how many standalone retraining runs were used to verify the selected architecture.
minor comments (6)
- [Abstract and Section IV-C] The "40% closer to ideal" figure is not exactly supported by Table III: (0.9802 - 0.9627)/(1 - 0.9627) is about 0.47, not 0.40. Please clarify the calculation or correct the percentage.
- [Table I] The header "Kendal Tau" should be "Kendall Tau."
- [Section III-A] There is a typo, "encoporates," and the reference given for TransNASBench is actually NasBench-301; the correct TransNASBench reference should be provided.
- [Section III-E] The sentence "keeping the top performance value of networks appearing in two or more sets" is ambiguous; please state whether the maximum, mean, or median across one-shot networks is used for the combined dataset.
- [Table II] Please report the number of standalone trained networks and the standard error of the mean, not just the variance, so that the reader can judge the precision of the variance estimates.
- [Figure 4] The caption says "Curves do not intersect outside the viewing region," which is confusing; please state clearly whether the full ROC curves cross anywhere outside the cropped area.
Circularity Check
No significant circularity; the final ROC-AUC is a measured result from a scratch-trained architecture, not a re-statement of predictor outputs.
full rationale
The paper's derivation chain is empirical rather than definitional: one-shot networks are trained, queried to produce (network, AUC) pairs, smoothed by taking top values across k networks, augmented with noise calibrated to the measured variance of standalone-trained ROC-AUC, and then used to train an XGBoost predictor. The predictor is validated against an independent set of standalone-trained architectures (reported Pearson correlation 0.65), and the final SOAPNet architecture is trained from scratch and evaluated on the 7k placement dataset to obtain the headline ROC-AUC of 0.9802. No equation in the paper defines the final AUC as the predictor's output; the text explicitly states that candidate architectures are trained in a stand-alone fashion to check that the predicted AUC matches actual performance. The augmentation step uses the target metric's variance to add label noise, which is a data-generation choice rather than a circular reduction of the final claim. Self-citations to NAS-crafted and LissNAS appear as baselines or related work and are not load-bearing for the central derivation. The main concerns, such as the unspecified train/test split for the 7k placements and the moderate 0.65 predictor correlation, are evaluation-protocol and generalization risks, not circularity. Therefore the paper is not circular by the quoted-reduction standard.
Assumptions & free parameters
free parameters (3)
- k (number of one-shot networks) =
5
- x (data augmentation factor) =
7
- noise model variance for augmentation =
0.0144 (ROC-AUC variance from Table II)
assumptions (3)
- domain assumption One-shot network queries are an unbiased proxy for stand-alone training ROC-AUC after smoothing
- domain assumption Additive noise with measured variance describes the distribution of stand-alone AUCs
- domain assumption The 7k placement dataset from 74 designs is representative of routability prediction
Cite this review
Pith. "Pith review of Improving Routability Prediction via NAS Using a Smooth One-shot Augmented Predictor." pith.science (2026). https://pith.science/paper/FB5T6GAP
@misc{pith2026241114296,
author = {Pith},
title = {Pith review of: Improving Routability Prediction via NAS Using a Smooth One-shot Augmented Predictor},
year = {2026},
howpublished = {\url{https://pith.science/paper/FB5T6GAP}},
note = {Machine review of arXiv:2411.14296}
}
read the original abstract
Routability optimization in modern EDA tools has benefited greatly from using machine learning (ML) models. Constructing and optimizing the performance of ML models continues to be a challenge. Neural Architecture Search (NAS) serves as a tool to aid in the construction and improvement of these models. Traditional NAS techniques struggle to perform well on routability prediction as a result of two primary factors. First, the separation between the training objective and the search objective adds noise to the NAS process. Secondly, the increased variance of the search objective further complicates performing NAS. We craft a novel NAS technique, coined SOAP-NAS, to address these challenges through novel data augmentation techniques and a novel combination of one-shot and predictor-based NAS. Results show that our technique outperforms existing solutions by 40% closer to the ideal performance measured by ROC-AUC (area under the receiver operating characteristic curve) in DRC hotspot detection. SOAPNet is able to achieve an ROC-AUC of 0.9802 and a query time of only 0.461 ms.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
Routenet: Routability prediction for mixed-size designs using convolutional neural network,
Z. Xie, Y .-H. Huang, G.-Q. Fang, H. Ren, S.-Y . Fang, Y . Chen, and J. Hu, “Routenet: Routability prediction for mixed-size designs using convolutional neural network,” in 2018 IEEE/ACM International Conference on Computer-Aided Design (ICCAD) , pp. 1–8, IEEE, 2018
work page 2018
-
[2]
Automatic routability predictor development using neural architecture search,
J. Pan, C. Chang, T. Zhang, and et al, “Automatic routability predictor development using neural architecture search,” ICCAD’21, 2020
work page 2020
-
[3]
Drc hotspot prediction at sub-10nm process nodes using customized convolutional network,
R. Liang, H. Xiang, D. Pandey, L. Reddy, S. Ramji, G.-J. Nam, and J. Hu, “Drc hotspot prediction at sub-10nm process nodes using customized convolutional network,” in ISPD, pp. 135–142, 09 2020
work page 2020
-
[4]
J. Chen and et al, “Pros: A plug-in for routability optimization applied in the state-of-the-art commercial eda tool using deep learning,” in ICCAD, ICCAD ’20, 2020
work page 2020
-
[5]
High-definition routing congestion prediction for large-scale fpgas,
M. B. Alawieh and et al, “High-definition routing congestion prediction for large-scale fpgas,” in 2020 25th Asia and South Pacific Design Automation Conference (ASP-DAC), pp. 26–31, IEEE, IEEE Computer Society, 2020
work page 2020
-
[6]
Painting on placement: Forecasting routing congestion using conditional generative adversarial nets,
C. Yu and Z. Zhang, “Painting on placement: Forecasting routing congestion using conditional generative adversarial nets,” inProceedings of the 56th Annual Design Automation Conference 2019 , pp. 1–6, 2019
work page 2019
-
[7]
Lithography hotspot detection and mitigation in nanometer vlsi,
J.-R. Gao and et al, “Lithography hotspot detection and mitigation in nanometer vlsi,” in 2013 IEEE 10th International Conference on ASIC , pp. 1–4, IEEE, 2013
work page 2013
-
[8]
Fast dual graph-based hotspot detection,
A. B. Kahng and et al, “Fast dual graph-based hotspot detection,” in Photomask Technology 2006, vol. 6349, pp. 125–132, SPIE, 2006
work page 2006
Show all 29 references
-
[9]
Qplacer: Frequency-aware component placement for superconducting quantum computers,
J. Zhang, H. Wang, Q. Ding, J. Gu, R. Assouly, W. D. Oliver, S. Han, K. R. Brown, H. H. Li, and Y . Chen, “Qplacer: Frequency-aware component placement for superconducting quantum computers,” 2024
2024
-
[10]
Accurate prediction of detailed routing congestion using supervised data learning,
Z. Qi and et al, “Accurate prediction of detailed routing congestion using supervised data learning,” in 2014 IEEE 32nd international conference on computer design (ICCD) , pp. 97–103, IEEE, 2014
2014
-
[11]
Supervised-learning congestion predictor for routability-driven global routing,
Z. Zhou and et al, “Supervised-learning congestion predictor for routability-driven global routing,” in 2019 International Symposium on VLSI Design, Automation and Test (VLSI-DAT) , pp. 1–4, IEEE, 2019
2019
-
[12]
Clock-aware ultrascale fpga placement with ma- chine learning routability prediction,
C.-W. Pui and et al, “Clock-aware ultrascale fpga placement with ma- chine learning routability prediction,” in 2017 IEEE/ACM International Conference on Computer-Aided Design (ICCAD) , pp. 929–936, IEEE, 2017
2017
-
[13]
Pin accessibility prediction and optimization with deep learning-based pin pattern recognition,
T.-C. Yu and et al, “Pin accessibility prediction and optimization with deep learning-based pin pattern recognition,” in Proceedings of the 56th Annual Design Automation Conference 2019 , pp. 1–6, 2019
2019
-
[14]
Lhnn: Lattice hypergraph neural network for vlsi congestion prediction,
B. Wang, G. Shen, D. Li, J. Hao, W. Liu, Y . Huang, H. Wu, Y . Lin, G. Chen, and P. A. Heng, “Lhnn: Lattice hypergraph neural network for vlsi congestion prediction,” 2022
2022
-
[15]
Lay-net: Graft- ing netlist knowledge on layout-based congestion prediction,
S. Zheng, L. Zou, P. Xu, S. Liu, B. Yu, and M. Wong, “Lay-net: Graft- ing netlist knowledge on layout-based congestion prediction,” in 2023 42nd IEEE/ACM International Conference on Computer-Aided Design, ICCAD 2023 - Proceedings , IEEE/ACM International Conference on Computer...
2023
-
[16]
A deep learning framework to predict routabil- ity for fpga circuit placement,
A. Al-Hyari and et al, “A deep learning framework to predict routabil- ity for fpga circuit placement,” ACM Transactions on Reconfigurable Technology and Systems (TRETS) , vol. 14, no. 3, pp. 1–28, 2021
2021
-
[17]
Lissnas: Locality-based iterative search space shrinkage for neural architecture search,
B. Gopal, A. Sridhar, and et al, “Lissnas: Locality-based iterative search space shrinkage for neural architecture search,” IJCAI’23, 2023
2023
-
[18]
Neural architecture search with reinforcement learning,
B. Zoph and Q. Le, “Neural architecture search with reinforcement learning,” in International Conference on Learning Representations , 2017
2017
-
[19]
Large-scale evolution of image classifiers,
E. Real and et al, “Large-scale evolution of image classifiers,” in International Conference on Machine Learning, pp. 2902–2911, PMLR, 2017
2017
-
[20]
Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures,
J. Bergstra, D. Yamins, and D. Cox, “Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures,” in Proceedings of the 30th International Conference on Machine Learning (S. Dasgupta and D. McAllester, eds.), vol. 28 of Pro...
2013
-
[21]
Neural predictor for neural architecture search,
W. Wen and et al, “Neural predictor for neural architecture search,” in European Conference on Computer Vision, pp. 660–676, Springer, 2020
2020
-
[22]
Single path one-shot neural architecture search with uniform sampling,
Z. Guo and et al, “Single path one-shot neural architecture search with uniform sampling,” CoRR, vol. abs/1904.00420, 2019
1904 arXiv
-
[23]
Nas-bench-101: Towards reproducible neural archi- tecture search,
C. Ying and et al, “Nas-bench-101: Towards reproducible neural archi- tecture search,” CoRR, vol. abs/1902.09635, 2019
1902 arXiv
-
[24]
Shufflenet: An extremely efficient convolutional neural network for mobile devices,
X. Zhang and et al, “Shufflenet: An extremely efficient convolutional neural network for mobile devices,” in Proceedings of the IEEE confer- ence on computer vision and pattern recognition , pp. 6848–6856, 2018
2018
-
[25]
Nas-bench-301 and the case for surrogate bench- marks for neural architecture search,
J. Siems and et al, “Nas-bench-301 and the case for surrogate bench- marks for neural architecture search,” CoRR, vol. abs/2008.09777, 2020
2008 arXiv
-
[26]
Combinational profiles of sequential benchmark cir- cuits,
F. Brglez and et al, “Combinational profiles of sequential benchmark cir- cuits,” in 1989 IEEE International Symposium on Circuits and Systems (ISCAS), pp. 1929–1934, IEEE, 1989
1989
-
[27]
Rt-level itc’99 benchmarks and first atpg results,
F. Corno and et al, “Rt-level itc’99 benchmarks and first atpg results,” IEEE Design & Test of computers , vol. 17, no. 3, pp. 44–53, 2000
2000
-
[28]
Iwls 2005 benchmarks,
C. Albrecht, “Iwls 2005 benchmarks,” in International Workshop for Logic Synthesis (IWLS) , vol. 9, IWLS, 2005
2005
-
[29]
Ispd 2015 benchmarks with fence regions and routing blockages for detailed-routing-driven placement,
I. S. Bustany and et al, “Ispd 2015 benchmarks with fence regions and routing blockages for detailed-routing-driven placement,” inProceedings of the 2015 Symposium on International Symposium on Physical Design, pp. 157–164, 2015
2015
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.