Pith. sign in

REVIEW 4 major objections 6 minor 29 references

Improving Routability Prediction via NAS Using a Smooth One-shot Augmented Predictor

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Pith's one-line read The paper claims that smoothing multiple one-shot networks and augmenting with variance-matched noise lets a lightweight predictor select a neural architecture that detects DRC hotspots at 0.9802 ROC-AUC in 0.461 ms.

desk verdict The k-shot smoothing idea is a genuinely fresh twist on one-shot NAS for EDA, but the paper's headline 0.9802 AUC is not yet a verified claim because the paper never specifies how the 7k placements are split between training and evaluation. read the letter →

arxiv 2411.14296 v1 pith:FB5T6GAP submitted 2024-11-21 cs.LG

classification cs.LG
keywords neuralarchitecturesearchDRChotspotdetectionroutabilitypredictionone-shotNASROC-AUCdataaugmentationelectronicdesignautomationXGBoostpredictor
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SOAP-NAS is a neural architecture search method built for routability prediction, where the standard NAS trick of training one shared one-shot supernetwork fails because it optimizes accuracy while the target metric, ROC-AUC, is noisier and not directly trained. The paper argues that training several one-shot networks and merging their queried scores ('smoothing'), then inflating the resulting training set with variance-matched noise ('augmentation'), lets a simple XGBoost predictor rank candidate architectures reliably. The final architecture, SOAPNet, reaches an ROC-AUC of 0.9802 on a 7,000-placement DRC hotspot dataset, which the paper reports as 40% closer to the ideal value of 1.0 than the best prior NAS-crafted model, with a query time of 0.461 ms. If correct, this closes a known gap between what one-shot NAS can search and what routability engineers actually care about.

What carries the argument

The central mechanism is the 'smooth one-shot augmented predictor' pipeline. Smoothing consists of training several one-shot supernetworks, sampling the same candidate set from each, and combining the queried scores so that an architecture's score is the maximum among the one-shot models that evaluated it, which damps the effect of random weight initialization on ROC-AUC. Augmentation then fits a noise model to the variance of ROC-AUC observed by retraining 50 standalone architectures 10 times each, and adds samples from that noise distribution to multiply the dataset by a factor of 7. These two steps produce a training set that mimics standalone-training outcomes, on which the XGBoost predictor selects the final architecture rather than ranking candidates by the noisy one-shot query itself.

What would settle it

Retrain SOAPNet from scratch, say 10 times, on the same 7,000-placement split and compare the distribution of ROC-AUC to the reported 0.9802; then take the predictor's top-10 candidates, train each standalone, and check whether the predictor's ranking matches the standalone ranking. If the retrained SOAPNet mean drops below roughly 0.97, or if a lower-ranked candidate consistently beats the selected one, the unbiased-proxy assumption fails.

Watch

Extended reading notes

Core claim

On its own terms, the paper's claim is that a NAS pipeline called SOAP-NAS solves the metric mismatch in DRC hotspot detection: instead of querying a single one-shot network for an architecture that maximizes ROC-AUC, SOAP-NAS trains $k=5$ one-shot networks from different random initializations, queries all of them, keeps the best score for architectures appearing in multiple query sets, and then augments this combined dataset sevenfold by adding noise whose magnitude matches the measured run-to-run variance of ROC-AUC. The resulting dataset trains an XGBoost predictor whose ranking correlation with standalone-trained architectures reaches 0.65 Pearson, up from 0.52 for the raw one-shot query, and the predictor's top pick, SOAPNet, achieves 0.9802 ROC-AUC on 7,000 placements from 74 designs while answering in 0.461 ms.

Load-bearing premise

The load-bearing premise is that the variance-matched, smoothed scores used to train the predictor are an unbiased proxy for what a candidate architecture achieves when trained from scratch by itself; the paper's own numbers (raw one-shot Pearson 0.52, predictor 0.65) leave room for ranking error, and a biased proxy could make SOAPNet not the genuinely best architecture.

Editorial extensions

If this is right

  • SOAPNet's 0.9802 ROC-AUC on the 7,000-placement dataset sets a new reported state of the art for ML-based DRC hotspot detection, improving on the best NAS-crafted baseline by closing 40% of the gap to 1.0.
  • With 0.461 ms per query, the architecture is fast enough to be embedded in iterative placement-and-routing loops that issue many hotspot queries per design.
  • The $k=5$ and $7\times$ augmentation settings transfer across NASBench101, ShuffleNetV2, and TransNASBench search spaces, so future users of SOAP-NAS can reuse these hyperparameters without re-tuning.
  • End-to-end, SOAP-NAS is a one-time search: the final architecture is a fixed network that can be applied to many circuit designs, amortizing the cost of training five one-shot supernetworks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The smoothing step is effectively an ensemble in score space; the same treatment could reduce rank noise in other NAS objectives that are high-variance or weakly correlated with the training loss, such as latency, energy, or robustness metrics.
  • The augmentation scheme assumes a single global variance for ROC-AUC, but the variance is likely architecture-dependent; a natural extension is per-candidate noise models, which could push the predictor correlation above 0.65 and possibly yield an even better final AUC.
  • Because the 7,000-placement dataset is compiled from public benchmarks (ISCAS'89, ITC'99, IWLS'05, ISPD'15) with a 45nm library, it could become a reusable benchmark for routability prediction; if it does, fair comparisons will need to fix the train/test design split and hardware for query-time measurements.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper proposes SOAP-NAS, a neural architecture search method for DRC hotspot detection. It first selects NASBench101 among three search spaces by comparing one-shot NAS correlations. It then trains k=5 one-shot networks, queries them to build a dataset of (architecture, ROC-AUC) pairs, augments the dataset 7x by adding noise whose variance is fitted from standalone retraining, trains an XGBoost predictor, and uses the predictor to select a final architecture (SOAPNet). The paper reports that SOAPNet achieves 0.9802 ROC-AUC and 0.461 ms query time on a 7k-placement dataset, and that this is 40% closer to the ideal ROC-AUC than the previous NAS-crafted baseline. The methodology is evaluated against RouteNet, PROS, cGAN, and NAS-crafted baselines, with per-design results on 74 designs.

Significance. The combination of k-shot smoothing and variance-matched augmentation is a reasonable and potentially reusable idea for NAS when the search metric has high variance and differs from the training loss. The paper also provides a useful comparison of three search spaces for this EDA task and reports query-time advantages. However, the central empirical claim is not currently verifiable: the paper does not define the train/validation/test split for the 7k-placement dataset, the single reported AUC has no error bars, and one baseline is implausibly below random. These problems must be fixed before the claimed state-of-the-art result can be accepted. The paper does not ship code, data, or configuration files, which further limits reproducibility.

major comments (4)
  1. [Section IV-B and Section III-D] The headline result of Table III, an ROC-AUC of 0.9802 "among all evaluated 7k evaluated placements," is not tied to any stated train/validation/test split. Section III-D describes generating a validation set of (network, AUC) pairs from the one-shot network and training candidates from scratch to check agreement, but it does not state whether the placements used to train the one-shot networks and the predictor are disjoint from the placements used to compute the Table III AUC. If the "evaluated" placements overlap with those used for one-shot training, predictor training, or architecture selection, the reported AUC is an in-sample measure rather than a predictive-accuracy estimate. The discrepancy with Table IV (SOAP-NAS 0.889 on the 74-design set) reinforces this concern. Please specify the exact split (e.g., design-disjoint training/validation/test) and report the final AUC on a held-out set of placements not used in any training or selection step.
  2. [Table IV] Table IV reports PROS with an ROC-AUC of 0.487 on all 74 designs and 0.478-0.489 on the per-design columns, which is below the random-guessing value of 0.5. This is an unexplained result for a published FCN baseline and is inconsistent with Table III, where the reproduced PROS is 0.9580. Either the reproduction protocol is broken (e.g., label misalignment or a sign error) or the comparison protocol differs between tables. The paper must explain or correct this baseline, because a below-random baseline artificially inflates the apparent improvement of SOAPNet.
  3. [Section IV-D] The key hyperparameters k=5 (number of one-shot networks) and x=7 (augmentation factor) are selected through the ablation study in Figures 6 and 7 on the same task and dataset used to report the final 0.9802 AUC. No nested validation procedure is described, so the final number is subject to selection bias. Please report the final performance with k and x fixed before seeing the test set, or provide a sensitivity analysis over k and x on a separate validation split.
  4. [Section III-F] The predictor's Pearson correlation, 0.65 after smoothing and augmentation, is reported without an error bar or design-disjoint evaluation. Because the final architecture is chosen by this predictor, a correlation of 0.65 leaves substantial room for ranking errors; the paper should report the variance of the final SOAPNet AUC over multiple runs of the entire SOAP-NAS pipeline, and should state how many standalone retraining runs were used to verify the selected architecture.
minor comments (6)
  1. [Abstract and Section IV-C] The "40% closer to ideal" figure is not exactly supported by Table III: (0.9802 - 0.9627)/(1 - 0.9627) is about 0.47, not 0.40. Please clarify the calculation or correct the percentage.
  2. [Table I] The header "Kendal Tau" should be "Kendall Tau."
  3. [Section III-A] There is a typo, "encoporates," and the reference given for TransNASBench is actually NasBench-301; the correct TransNASBench reference should be provided.
  4. [Section III-E] The sentence "keeping the top performance value of networks appearing in two or more sets" is ambiguous; please state whether the maximum, mean, or median across one-shot networks is used for the combined dataset.
  5. [Table II] Please report the number of standalone trained networks and the standard error of the mean, not just the variance, so that the reader can judge the precision of the variance estimates.
  6. [Figure 4] The caption says "Curves do not intersect outside the viewing region," which is confusing; please state clearly whether the full ROC curves cross anywhere outside the cropped area.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the final ROC-AUC is a measured result from a scratch-trained architecture, not a re-statement of predictor outputs.

full rationale

The paper's derivation chain is empirical rather than definitional: one-shot networks are trained, queried to produce (network, AUC) pairs, smoothed by taking top values across k networks, augmented with noise calibrated to the measured variance of standalone-trained ROC-AUC, and then used to train an XGBoost predictor. The predictor is validated against an independent set of standalone-trained architectures (reported Pearson correlation 0.65), and the final SOAPNet architecture is trained from scratch and evaluated on the 7k placement dataset to obtain the headline ROC-AUC of 0.9802. No equation in the paper defines the final AUC as the predictor's output; the text explicitly states that candidate architectures are trained in a stand-alone fashion to check that the predicted AUC matches actual performance. The augmentation step uses the target metric's variance to add label noise, which is a data-generation choice rather than a circular reduction of the final claim. Self-citations to NAS-crafted and LissNAS appear as baselines or related work and are not load-bearing for the central derivation. The main concerns, such as the unspecified train/test split for the 7k placements and the moderate 0.65 predictor correlation, are evaluation-protocol and generalization risks, not circularity. Therefore the paper is not circular by the quoted-reduction standard.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim depends on the choice of k and x, which are tuned on the same benchmark, and on the noise model estimated from a small sample. The one-shot-to-standalone transfer assumption is the most load-bearing, as the paper's own correlation numbers are low.

free parameters (3)
  • k (number of one-shot networks) = 5
    Selected by ablation on the same task; Figure 6/7 show plateau at 5. This directly affects the smoothed dataset and final architecture.
  • x (data augmentation factor) = 7
    Selected by ablation; Figure 6/7 show optimal at 7. Controls the size and noise level of predictor training data.
  • noise model variance for augmentation = 0.0144 (ROC-AUC variance from Table II)
    Estimated from 50 architectures trained 10 times; used to generate synthetic augmented samples for predictor training.
assumptions (3)
  • domain assumption One-shot network queries are an unbiased proxy for stand-alone training ROC-AUC after smoothing
    The predictor-based NAS relies on the one-shot network's ability to rank architectures. The paper reports low Pearson correlation (0.52 max) before smoothing, and only 0.65 after, so this is a strong assumption.
  • domain assumption Additive noise with measured variance describes the distribution of stand-alone AUCs
    Used to augment predictor training data; if the true distribution is not captured by a single variance, the augmented data is mis-specified.
  • domain assumption The 7k placement dataset from 74 designs is representative of routability prediction
    The paper proposes it as a new standard; no external validation on unseen benchmarks is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improving Routability Prediction via NAS Using a Smooth One-shot Augmented Predictor." pith.science (2026). https://pith.science/paper/FB5T6GAP

@misc{pith2026241114296,
  author       = {Pith},
  title        = {Pith review of: Improving Routability Prediction via NAS Using a Smooth One-shot Augmented Predictor},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FB5T6GAP}},
  note         = {Machine review of arXiv:2411.14296}
}
read the original abstract

Routability optimization in modern EDA tools has benefited greatly from using machine learning (ML) models. Constructing and optimizing the performance of ML models continues to be a challenge. Neural Architecture Search (NAS) serves as a tool to aid in the construction and improvement of these models. Traditional NAS techniques struggle to perform well on routability prediction as a result of two primary factors. First, the separation between the training objective and the search objective adds noise to the NAS process. Secondly, the increased variance of the search objective further complicates performing NAS. We craft a novel NAS technique, coined SOAP-NAS, to address these challenges through novel data augmentation techniques and a novel combination of one-shot and predictor-based NAS. Results show that our technique outperforms existing solutions by 40% closer to the ideal performance measured by ROC-AUC (area under the receiver operating characteristic curve) in DRC hotspot detection. SOAPNet is able to achieve an ROC-AUC of 0.9802 and a query time of only 0.461 ms.

Figures

Figures reproduced from arXiv: 2411.14296 by the authors.

Figure 1
Figure 1. Several one-shot networks are independently trained to [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Histogram of queried AUC and ACC and loss values. While ROC-AUC generally mirrors accuracy, there are cases where this relation breaks down. This results in degraded performance when querying the one-shot network for a different metric than the metric used during training. Secondly, ROC-AUC as a metric has higher variance than accuracy. The variance might be due to the fact that AUC is not directly linked to accurac… view at source ↗
Figure 3
Figure 3. Overview of SOAP-NAS the impact of randomness in training and initialization of the networks. While this gets us closer to the desired ideal result, there is still noise introduced in each of the one-shot networks’ training procedure. In addition, the final dataset size is not large enough to effectively train a predictor-based NAS method. F. Augmenting To efficiently generate enough training data for the predic￾tor… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: ROC curves show the superiority of SOAPNet com [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Left column shows examples of hot spot labels at [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: The effect of smoothing and augmenting data across 3 [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Importance of smoothing and augmenting data on [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 25 canonical work pages

  1. [1]

    Routenet: Routability prediction for mixed-size designs using convolutional neural network,

    Z. Xie, Y .-H. Huang, G.-Q. Fang, H. Ren, S.-Y . Fang, Y . Chen, and J. Hu, “Routenet: Routability prediction for mixed-size designs using convolutional neural network,” in 2018 IEEE/ACM International Conference on Computer-Aided Design (ICCAD) , pp. 1–8, IEEE, 2018

  2. [2]

    Automatic routability predictor development using neural architecture search,

    J. Pan, C. Chang, T. Zhang, and et al, “Automatic routability predictor development using neural architecture search,” ICCAD’21, 2020

  3. [3]

    Drc hotspot prediction at sub-10nm process nodes using customized convolutional network,

    R. Liang, H. Xiang, D. Pandey, L. Reddy, S. Ramji, G.-J. Nam, and J. Hu, “Drc hotspot prediction at sub-10nm process nodes using customized convolutional network,” in ISPD, pp. 135–142, 09 2020

  4. [4]

    Pros: A plug-in for routability optimization applied in the state-of-the-art commercial eda tool using deep learning,

    J. Chen and et al, “Pros: A plug-in for routability optimization applied in the state-of-the-art commercial eda tool using deep learning,” in ICCAD, ICCAD ’20, 2020

  5. [5]

    High-definition routing congestion prediction for large-scale fpgas,

    M. B. Alawieh and et al, “High-definition routing congestion prediction for large-scale fpgas,” in 2020 25th Asia and South Pacific Design Automation Conference (ASP-DAC), pp. 26–31, IEEE, IEEE Computer Society, 2020

  6. [6]

    Painting on placement: Forecasting routing congestion using conditional generative adversarial nets,

    C. Yu and Z. Zhang, “Painting on placement: Forecasting routing congestion using conditional generative adversarial nets,” inProceedings of the 56th Annual Design Automation Conference 2019 , pp. 1–6, 2019

  7. [7]

    Lithography hotspot detection and mitigation in nanometer vlsi,

    J.-R. Gao and et al, “Lithography hotspot detection and mitigation in nanometer vlsi,” in 2013 IEEE 10th International Conference on ASIC , pp. 1–4, IEEE, 2013

  8. [8]

    Fast dual graph-based hotspot detection,

    A. B. Kahng and et al, “Fast dual graph-based hotspot detection,” in Photomask Technology 2006, vol. 6349, pp. 125–132, SPIE, 2006

Show all 29 references
  1. [9]

    Qplacer: Frequency-aware component placement for superconducting quantum computers,

    J. Zhang, H. Wang, Q. Ding, J. Gu, R. Assouly, W. D. Oliver, S. Han, K. R. Brown, H. H. Li, and Y . Chen, “Qplacer: Frequency-aware component placement for superconducting quantum computers,” 2024

  2. [10]

    Accurate prediction of detailed routing congestion using supervised data learning,

    Z. Qi and et al, “Accurate prediction of detailed routing congestion using supervised data learning,” in 2014 IEEE 32nd international conference on computer design (ICCD) , pp. 97–103, IEEE, 2014

  3. [11]

    Supervised-learning congestion predictor for routability-driven global routing,

    Z. Zhou and et al, “Supervised-learning congestion predictor for routability-driven global routing,” in 2019 International Symposium on VLSI Design, Automation and Test (VLSI-DAT) , pp. 1–4, IEEE, 2019

  4. [12]

    Clock-aware ultrascale fpga placement with ma- chine learning routability prediction,

    C.-W. Pui and et al, “Clock-aware ultrascale fpga placement with ma- chine learning routability prediction,” in 2017 IEEE/ACM International Conference on Computer-Aided Design (ICCAD) , pp. 929–936, IEEE, 2017

  5. [13]

    Pin accessibility prediction and optimization with deep learning-based pin pattern recognition,

    T.-C. Yu and et al, “Pin accessibility prediction and optimization with deep learning-based pin pattern recognition,” in Proceedings of the 56th Annual Design Automation Conference 2019 , pp. 1–6, 2019

  6. [14]

    Lhnn: Lattice hypergraph neural network for vlsi congestion prediction,

    B. Wang, G. Shen, D. Li, J. Hao, W. Liu, Y . Huang, H. Wu, Y . Lin, G. Chen, and P. A. Heng, “Lhnn: Lattice hypergraph neural network for vlsi congestion prediction,” 2022

  7. [15]

    Lay-net: Graft- ing netlist knowledge on layout-based congestion prediction,

    S. Zheng, L. Zou, P. Xu, S. Liu, B. Yu, and M. Wong, “Lay-net: Graft- ing netlist knowledge on layout-based congestion prediction,” in 2023 42nd IEEE/ACM International Conference on Computer-Aided Design, ICCAD 2023 - Proceedings , IEEE/ACM International Conference on Computer...

  8. [16]

    A deep learning framework to predict routabil- ity for fpga circuit placement,

    A. Al-Hyari and et al, “A deep learning framework to predict routabil- ity for fpga circuit placement,” ACM Transactions on Reconfigurable Technology and Systems (TRETS) , vol. 14, no. 3, pp. 1–28, 2021

  9. [17]

    Lissnas: Locality-based iterative search space shrinkage for neural architecture search,

    B. Gopal, A. Sridhar, and et al, “Lissnas: Locality-based iterative search space shrinkage for neural architecture search,” IJCAI’23, 2023

  10. [18]

    Neural architecture search with reinforcement learning,

    B. Zoph and Q. Le, “Neural architecture search with reinforcement learning,” in International Conference on Learning Representations , 2017

  11. [19]

    Large-scale evolution of image classifiers,

    E. Real and et al, “Large-scale evolution of image classifiers,” in International Conference on Machine Learning, pp. 2902–2911, PMLR, 2017

  12. [20]

    Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures,

    J. Bergstra, D. Yamins, and D. Cox, “Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures,” in Proceedings of the 30th International Conference on Machine Learning (S. Dasgupta and D. McAllester, eds.), vol. 28 of Pro...

  13. [21]

    Neural predictor for neural architecture search,

    W. Wen and et al, “Neural predictor for neural architecture search,” in European Conference on Computer Vision, pp. 660–676, Springer, 2020

  14. [22]

    Single path one-shot neural architecture search with uniform sampling,

    Z. Guo and et al, “Single path one-shot neural architecture search with uniform sampling,” CoRR, vol. abs/1904.00420, 2019

  15. [23]

    Nas-bench-101: Towards reproducible neural archi- tecture search,

    C. Ying and et al, “Nas-bench-101: Towards reproducible neural archi- tecture search,” CoRR, vol. abs/1902.09635, 2019

  16. [24]

    Shufflenet: An extremely efficient convolutional neural network for mobile devices,

    X. Zhang and et al, “Shufflenet: An extremely efficient convolutional neural network for mobile devices,” in Proceedings of the IEEE confer- ence on computer vision and pattern recognition , pp. 6848–6856, 2018

  17. [25]

    Nas-bench-301 and the case for surrogate bench- marks for neural architecture search,

    J. Siems and et al, “Nas-bench-301 and the case for surrogate bench- marks for neural architecture search,” CoRR, vol. abs/2008.09777, 2020

  18. [26]

    Combinational profiles of sequential benchmark cir- cuits,

    F. Brglez and et al, “Combinational profiles of sequential benchmark cir- cuits,” in 1989 IEEE International Symposium on Circuits and Systems (ISCAS), pp. 1929–1934, IEEE, 1989

  19. [27]

    Rt-level itc’99 benchmarks and first atpg results,

    F. Corno and et al, “Rt-level itc’99 benchmarks and first atpg results,” IEEE Design & Test of computers , vol. 17, no. 3, pp. 44–53, 2000

  20. [28]

    Iwls 2005 benchmarks,

    C. Albrecht, “Iwls 2005 benchmarks,” in International Workshop for Logic Synthesis (IWLS) , vol. 9, IWLS, 2005

  21. [29]

    Ispd 2015 benchmarks with fence regions and routing blockages for detailed-routing-driven placement,

    I. S. Bustany and et al, “Ispd 2015 benchmarks with fence regions and routing blockages for detailed-routing-driven placement,” inProceedings of the 2015 Symposium on International Symposium on Physical Design, pp. 157–164, 2015

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.