Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

ESM: A Framework for Building Effective Surrogate Models for Hardware-Aware Neural Architecture Search

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Counting feature combos lifts NAS latency accuracy to 97.6%

desk verdict FCC encoding is a simple, useful baseline that mostly works; the framework's convergence claim is compromised by test-set leakage until a validation split is shown. read the letter →

arxiv 2508.01505 v1 pith:6TFWZSK5 submitted 2025-08-02 cs.LG

Azaz-Ur-Rehman Nasir , Samroz Ahmad Shoaib , Muhammad Abdullah Hanif , Muhammad Shafique This is my paper · ORCID
classification cs.LG
keywords neuralarchitecturesearchhardware-awareNASlatencypredictionsurrogatemodelsencodingfeaturecombinationcountbalancedsamplingmultilayerperceptron
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that for hardware-aware neural architecture search over a fixed supernet, the limiting factor in latency prediction is how architectures are encoded, not how many samples you train on. It introduces Feature Combination Count (FCC) encoding, which represents a model by counting how many blocks use each combination of kernel size and width-expansion ratio. Combined with a simple MLP, this encoding reaches about 97.6% average latency-prediction accuracy on a ResNet search space with 8,000 training samples, while the statistical encoding baseline reaches about 85.8%. The paper wraps this encoding in ESM, a framework with balanced sampling and iterative dataset extension that stops once user-defined per-bin accuracy thresholds are met, and shows balanced sampling converges in roughly 500 samples versus 4,000 for random sampling.

What carries the argument

FCC encoding: for every possible combination of the block-level features (e.g., kernel size $k$ and width-expansion ratio $e$), count how many blocks in the model realize that combination; the vector of these counts is the model's representation. The framework's other load-bearing piece is the train-evaluate-extend loop: sample architectures (random or balanced), measure latency on the target hardware with outlier-filtered multiple inferences, train a three-layer MLP, evaluate accuracy bin-wise along the depth dimension, and if any bin fails a user threshold, add weighted new samples to the failing bins and retrain. Balanced sampling removes the central-limit bias of random depth sampling by stratifying over depth bins.

What would settle it

Run the ESM loop exactly as described but with a separate validation set for the bin-wise accuracy that triggers dataset extension, then measure final accuracy on a held-out test set. If the held-out accuracy is substantially below the reported 97.6% for ResNet on RTX 4090, the claimed advantage of balanced sampling and early convergence is an artifact of test-set selection.

Watch

Extended reading notes

Core claim

The central claim is that latency-prediction accuracy for layer/block-wise supernet search spaces is set by the representational adequacy of the architecture encoding, and that a count-based encoding can capture enough structure without being sparse (one-hot) or too dense (statistical mean and standard deviation). On the paper's own numbers, FCC with an MLP yields 97.6% and 97.8% average accuracy on the ResNet supernet for 8,000 and 20,000 samples, versus 85.8% and 83.1% for statistical encoding. The same encoding also beats lookup-table surrogates, including a bias-corrected lookup table. The ESM framework's balanced sampling, which draws equal numbers of architectures from bins along the depth dimension, converges after 3 iterations (500 samples) where random sampling takes 37 iterations (4,000 samples).

Load-bearing premise

The reported accuracies assume the 4,000-sample test set is an unbiased measure of generalization, even though the framework's dataset-extension step uses bin-wise accuracy computed on that same test set to decide which architectures to add.

Editorial extensions

If this is right

  • Hardware-aware NAS can get accurate latency estimates from a few hundred measured architectures instead of tens of thousands.
  • Surrogate accuracy is comparable across GPU, CPU, and embedded devices once the encoding is expressive enough, per the paper's device experiments.
  • Bin-wise accuracy thresholds give the user a per-depth guarantee instead of a single overall number.
  • The ESM loop can be stopped early by design, making predictor generation cost proportional to the difficulty of the search space.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: FCC should transfer to any search space whose block-level features can be enumerated in advance; for spaces with continuous or unbounded features the count-based vector would need discretization or hashing.
  • Editorial extension: Because the paper's accuracy numbers are measured against a test set that also guides dataset extension, a separate validation hold-out is the natural next test; if accuracy falls, the convergence claim is partly an artifact of selection on the test set.
  • Editorial extension: The same framework could target energy or memory footprint rather than latency, since the encoding and binning logic do not depend on the measured quantity.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces ESM, a framework for building latency-prediction surrogate models for hardware-aware NAS over fixed macro-architecture search spaces. It proposes Feature Combination Count (FCC) encoding for block-wise architectures, a train-evaluate-extend loop with random or balanced sampling, bin-wise accuracy thresholds, and dataset-quality control via reference models. Experiments compare FCC/FC encoding against statistical encoding and lookup-table baselines on ResNet, DenseNet, and MobileNetV3 supernets across GPU/CPU/embedded devices, and compare random vs. balanced sampling convergence. The central claims are that FCC+MLP outperforms the considered baselines and that balanced sampling with ESM converges with far fewer training samples.

Significance. If the evaluation-protocol issues are resolved, the paper would make a useful practical contribution: FCC is a simple encoding whose dimension is independent of the number of blocks, and the iterative extension loop directly targets a real bottleneck in HW-NAS, namely data-collection cost. The paper's strengths are the breadth of the empirical study (three supernets, four devices, two sampling strategies), the inclusion of lookup-table baselines and bias correction, and the concrete dataset-quality-control step. However, because the core quantitative claims rest on an undefined accuracy metric and on an evaluation loop that appears to use the test set for data-extension decisions, the significance cannot be fully assessed until these points are fixed.

major comments (4)
  1. [§III.A, §II.E (Algorithm 1), Fig. 5] The ESM evaluation protocol appears circular as described. Section III.A states that, unless otherwise noted, the predictor is 'tested on 4,000 samples,' and Section II.E/Algorithm 1 uses bin-wise accuracies AccBins to decide which bins to extend (via w1/w2 weighting) and whether to stop (via AccTH). If the same 4,000 samples are used both to trigger dataset extension and to produce the reported accuracy, then Fig. 11's convergence comparison (balanced at ~500 samples vs. random at ~4,000 samples) measures convergence toward the test set rather than unbiased generalization. The authors must either demonstrate that a separate validation set was used for all extension decisions and report final accuracy on an untouched test set, or re-run the experiment with a proper validation/test split. Without this, the framework claims in Sections I.B and III.D are not supported.
  2. [§III.B, Fig. 9] The 'average accuracy' metric is never defined. Since latency prediction is a regression task, values such as 97.6%, 85.8%, and 83.9% in Fig. 9 require an explicit formula (e.g., 1 − mean absolute percentage error, percentage of predictions within a tolerance, or R²). Without this definition the quantitative comparisons in Figs. 8–11 cannot be reproduced or interpreted, and the headline claims about FCC outperforming statistical encoding are unverifiable. Please add the metric definition where it is first used and use it consistently.
  3. [§III.C, Fig. 10(c)] The claim that the proposed encodings outperform statistical encoding is not supported on all reported conditions. For DenseNet (Fig. 10(c)), FCC gives 99%, 94%, 94% accuracy on RTX 4090, AMD CPU, and RTX 3080Ti, while statistical encoding gives 91%, 96%, 95%; statistical encoding is better on two of the three devices. In addition, for MobileNetV3 (Fig. 10(b)), FC matches or exceeds FCC on all devices, so the advantage is not exclusive to FCC. The paper should soften the 'most devices' claim to reflect this device dependence or provide an aggregate/statistical comparison across all networks and devices.
  4. [§III.D, Fig. 11] The convergence comparison between random and balanced sampling is reported without any indication of the number of runs or variance. Because the initial sample NI, the NStep additions, and MLP training are stochastic, a single trajectory is not sufficient to support the statement that balanced sampling 'converged after 3 iterations with 500 samples' while random sampling 'took 37 iterations to converge with 4000 samples.' Please report results over multiple seeds (e.g., mean and standard deviation) or otherwise demonstrate the stability of the comparison.
minor comments (5)
  1. [§III.A, §III.C, Fig. 10] Target device names are inconsistent: Section III.A lists 'NVIDIA RTX 3080 Max-Q' but Section III.C and Fig. 10 use 'RTX 3080Ti'; please align the naming throughout.
  2. [Fig. 10] Fig. 10 shows Raspberry Pi 4 results only for ResNet, although the text says the framework was evaluated on four devices; please clarify why Pi 4 is omitted for MobileNetV3 and DenseNet.
  3. [§I.A] The first paragraph of Section I.A contains a typo, 'O F A architecture spaces,' which should read 'OFA architecture spaces.'
  4. [§II.E, Algorithm 1] Algorithm 1's loop over 'BelowAccTH and AboveAccTH' is syntactically confusing: it appears to assign NBTH and NATH once, but the intended computation should be stated more clearly.
  5. [§II.C] Section II.C would benefit from a formal definition of FCC encoding (the set of feature combinations and the resulting vector length); the current description is example-based and hard to reproduce exactly.

Circularity Check

1 steps flagged · score 4.0 of 10

FCC-vs-statistical encoding comparison is independent and non-circular; however, Fig. 11's balanced-sampling convergence claim is partially circular because dataset extension and stopping are driven by bin accuracies computed on the same 4,000-sample test set used for the reported accuracy.

  1. fitted input called prediction [Section II.D-II.E (Algorithm 1), Section III.A, Fig. 11]
    "Unless mentioned otherwise, the predictor is trained on 8,000 samples and tested on 4,000 samples. ... If all the bin's accuracy, Acci, is greater than the threshold, AccTH, the predictor meets the evaluation criteria. If not, all the accuracies of the bins, AccBins, are noted. The framework proceeds to dataset extension, after which the predictor is retrained and re-evaluated."

    Algorithm 1's balanced branch separates bins into BelowAccTH and AboveAccTH using AccBins and allocates NStep samples to below-threshold bins; training stops only when every Acc_i >= AccTH. Under the paper's default protocol (Section III.A), AccBins are computed on the same 4,000 test samples later reported as final accuracy in Fig. 11. Hence the extension policy and stopping rule are fit to the test set, so the reported convergence (balanced ~500 vs random ~4,000 samples) measures test-set fitting, not unbiased generalization. This is adaptive-data-analysis circularity rather than an equation-level identity; it does not affect the fixed-size encoding comparisons in Figs. 8-10, which do not use the extension loop.

full rationale

The paper's main encoding claims (Section III.B, Figs. 8-9) are one-shot predictions on a held-out 4,000-sample test set and do not involve the ESM extension loop, so they are not circular. The FCC encoder is a hand-designed feature count, not defined in terms of the predicted latencies. The only load-bearing circularity is in the ESM convergence evaluation (Section II.D-II.E, Algorithm 1, Fig. 11): bin-wise accuracies AccBins decide which bins receive additional samples (weights w1/w2) and when training stops (Acc_i >= AccTH), and the paper's default protocol reports test accuracy on the same 4,000 samples. Thus the balanced-sampling advantage is an adaptive-data-analysis artifact: the model is being fitted to the test set's bin errors, and the final accuracy is not an unbiased generalization estimate. This does not invalidate the fixed-size encoding comparisons, so the overall circularity is partial. No self-citation chain or imported uniqueness theorem is load-bearing.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claims rely on a handful of hand-chosen design choices (encoding, sampling, thresholds, MLP shape) and on the assumption that profiling on a specific device gives stable ground-truth latency. The FCC encoding is asserted to be information-preserving enough; no proof of optimality. The ESM loop's use of the test set for dataset extension is the key methodological choice.

free parameters (3)
  • Framework user inputs (NI, NStep, w1, w2, NBins, AccTH) = Not reported for most experiments; Fig. 11 uses NI=300, NStep=100; AccTH around 98% from plot
    These hand-chosen controls determine when the iterative framework stops and how data are extended; the convergence comparison depends on them.
  • MLP architecture and training hyperparameters = 3 fully-connected layers of hidden size 64, LR=0.01, weight decay 1e-4, MSE loss, Adam
    Chosen by hand with no sensitivity analysis; the reported accuracy could shift with different MLP settings.
  • Latency profiling protocol constants = 150 inference runs, discard slowest and fastest 20%, average middle 60%, 3% variance threshold
    Arbitrary but plausible choices for obtaining stable ground-truth latency; they affect dataset quality and collection cost.
assumptions (4)
  • domain assumption A fixed macro-architecture with mutable blocks (OFA-style supernets) is a representative architecture space for hardware-aware NAS.
    All experiments use ResNet, MobileNetV3, and DenseNet supernets; the framework's generality to other search spaces is assumed (Sec. I.A, III.A).
  • domain assumption Profiled latency after trimming is a reliable ground truth.
    The 3% variance quality gate is author-defined and not validated against an external measurement standard (Sec. II.C.3).
  • domain assumption An MLP with 3x64 units can learn latency from the proposed encodings.
    No theoretical justification is given; the claim rests entirely on empirical fit (Sec. II.D).
  • ad hoc to paper Using test-set bin accuracy to guide dataset extension does not bias the final evaluation.
    Algorithm 1 extends data in bins that fail on the same 4,000-sample test set, so the stopping rule and reported accuracy are not fully independent (Sec. II.E, Fig. 5).

how reviews work

0 comments
Cite this review

Pith. "Pith review of ESM: A Framework for Building Effective Surrogate Models for Hardware-Aware Neural Architecture Search." pith.science (2026). https://pith.science/paper/6TFWZSK5

@misc{pith2026250801505,
  author       = {Pith},
  title        = {Pith review of: ESM: A Framework for Building Effective Surrogate Models for Hardware-Aware Neural Architecture Search},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6TFWZSK5}},
  note         = {Machine review of arXiv:2508.01505}
}
read the original abstract

Hardware-aware Neural Architecture Search (NAS) is one of the most promising techniques for designing efficient Deep Neural Networks (DNNs) for resource-constrained devices. Surrogate models play a crucial role in hardware-aware NAS as they enable efficient prediction of performance characteristics (e.g., inference latency and energy consumption) of different candidate models on the target hardware device. In this paper, we focus on building hardware-aware latency prediction models. We study different types of surrogate models and highlight their strengths and weaknesses. We perform a systematic analysis to understand the impact of different factors that can influence the prediction accuracy of these models, aiming to assess the importance of each stage involved in the model designing process and identify methods and policies necessary for designing/training an effective estimation model, specifically for GPU-powered devices. Based on the insights gained from the analysis, we present a holistic framework that enables reliable dataset generation and efficient model generation, considering the overall costs of different stages of the model generation pipeline.

Figures

Figures reproduced from arXiv: 2508.01505 by the authors.

Figure 1
Figure 1. A general overview of hardware-aware NAS which highlights the [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Impact of inaccuracy in latency prediction on neural architecture search [PITH_FULL_IMAGE:figures/full_fig_p001_2.png] view at source ↗
Figure 3
Figure 3. Actual vs. predicted latency for 4k ResNet model configurations using [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: (a) Comparison between the time required to accurately compute the [PITH_FULL_IMAGE:figures/full_fig_p002_4.png]
Figure 5
Figure 5. Figure 5: Overview of our ESM Framework. 0 0.01 0.02 1 17 33 49 65 81 97 113 129 145 161 177 193 209 225 241 257 Latency [sec] Iterations Less than 3% Error Range [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 6
Figure 6. Figure 6: Variance Plot of Reference models. Most of reference model instances [PITH_FULL_IMAGE:figures/full_fig_p004_6.png]
Figure 7
Figure 7. Figure 7: (a) General representation of ResNet, MobileNetV3, and DenseNet supernets. (b) General representation of encoding, illustrating concatenation of unit-level [PITH_FULL_IMAGE:figures/full_fig_p005_7.png]
Figure 8
Figure 8. Figure 8: Scatter plots illustrating a comparison between FCC-based MLP with Statistical Encoding-based MLP and Lookup-table for ResNet(Top-row) and [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]
Figure 9
Figure 9. Figure 9: Average accuracies’ comparison for the cases presented in [PITH_FULL_IMAGE:figures/full_fig_p006_9.png]
Figure 11
Figure 11. Figure 11: Comparison between Random and Balanced Sampling Strategy for [PITH_FULL_IMAGE:figures/full_fig_p006_11.png]
Figure 10
Figure 10. Figure 10: Showcasing effectiveness of the proposed encoding schemes across different target devices: NVIDIA RTX 4090, AMD Ryzen Threadripper 5975, NVIDIA RTX 3080 Max-Q, and Raspberry Pi 4. The three plots are for (a) ResNet, (b) MobileNetV3, and (c) DenseNet. D. Random and Bal…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. DIY hybrid SSVEP-P300 LED stimuli for BCI platform using EMOTIV EEG headset

    eess.SP 2025-08 unverdicted novelty 4.0 of 10

    A hybrid SSVEP-P300 LED stimulator with hardware event timestamps was built and used for real-time four-direction robot control.

Reference graph

Works this paper leans on

19 extracted references · 13 canonical work pages · cited by 1 Pith paper

  1. [1]

    Deep learning for computer vision: A brief review,

    A. V oulodimos, N. Doulamis, A. Doulamis, and E. Protopapadakis, “Deep learning for computer vision: A brief review,” Computational intelligence and neuroscience, vol. 2018, no. 1, p. 7068349, 2018

    work page 2018

  2. [2]

    A survey of the usages of deep learning for natural language processing,

    D. W. Otter, J. R. Medina, and J. K. Kalita, “A survey of the usages of deep learning for natural language processing,” IEEE transactions on neural networks and learning systems , vol. 32, no. 2, pp. 604–624, 2020

    2020

  3. [3]

    A survey of large language models,

    W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y . Hou, Y . Min, B. Zhang, J. Zhang, Z. Dong et al. , “A survey of large language models,” arXiv preprint arXiv:2303.18223, 2023

    arXiv 2023

  4. [4]

    A survey of accelerator architectures for deep neural networks,

    Y . Chen, Y . Xie, L. Song, F. Chen, and T. Tang, “A survey of accelerator architectures for deep neural networks,” Engineering, vol. 6, no. 3, pp. 264–274, 2020

    work page 2020

  5. [5]

    Once-for-all: Train one network and specialize it for efficient deployment,

    H. Cai, C. Gan, T. Wang, Z. Zhang, and S. Han, “Once-for-all: Train one network and specialize it for efficient deployment,” arXiv preprint arXiv:1908.09791, 2019

    arXiv 1908

  6. [6]

    Hardware-aware neural architecture search: Survey and taxonomy

    H. Benmeziane, K. El Maghraoui, H. Ouarnoughi, S. Niar, M. Wistuba, and N. Wang, “Hardware-aware neural architecture search: Survey and taxonomy.” in IJCAI, 2021, pp. 4322–4329

    work page 2021

  7. [7]

    Netadapt: Platform-aware neural network adaptation for mobile applications,

    T.-J. Yang, A. Howard, B. Chen, X. Zhang, A. Go, M. Sandler, V . Sze, and H. Adam, “Netadapt: Platform-aware neural network adaptation for mobile applications,” in Proceedings of the European conference on computer vision (ECCV) , 2018, pp. 285–300

    work page 2018

  8. [8]

    Proxylessnas: Direct neural architecture search on target task and hardware,

    H. Cai, L. Zhu, and S. Han, “Proxylessnas: Direct neural architecture search on target task and hardware,” arXiv preprint arXiv:1812.00332 , 2018

    arXiv 2018

Show all 19 references
  1. [9]

    Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search,

    B. Wu, X. Dai, P. Zhang, Y . Wang, F. Sun, Y . Wu, Y . Tian, P. Vajda, Y . Jia, and K. Keutzer, “Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition ...

    2019

  2. [10]

    Apq: Joint search for network architecture, pruning and quantization policy,

    T. Wang, K. Wang, H. Cai, J. Lin, Z. Liu, H. Wang, Y . Lin, and S. Han, “Apq: Joint search for network architecture, pruning and quantization policy,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020, pp. 2078–2087

    2020

  3. [11]

    Hat: Hardware-aware transformers for efficient natural language processing,

    H. Wang, Z. Wu, Z. Liu, H. Cai, L. Zhu, C. Gan, and S. Han, “Hat: Hardware-aware transformers for efficient natural language processing,” arXiv preprint arXiv:2005.14187 , 2020

    2005 arXiv

  4. [12]

    Efficientnet: Rethinking model scaling for convolu- tional neural networks,

    M. Tan and Q. Le, “Efficientnet: Rethinking model scaling for convolu- tional neural networks,” in International conference on machine learning . PMLR, 2019, pp. 6105–6114

    2019

  5. [13]

    Shufflenet: An extremely efficient convolutional neural network for mobile devices,

    X. Zhang, X. Zhou, M. Lin, and J. Sun, “Shufflenet: An extremely efficient convolutional neural network for mobile devices,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2018, pp. 6848–6856

    2018

  6. [14]

    Efficient sampling for predictor-based neural architecture search,

    L. Mauch, S. Tiedemann, J. A. Garcia, B. N. Cong, K. Yoshiyama, F. Cardinaux, and T. Kemp, “Efficient sampling for predictor-based neural architecture search,” arXiv preprint arXiv:2011.12043 , 2020

    2011 arXiv

  7. [15]

    Fbnetv3: Joint architecture-recipe search using predictor pretraining,

    X. Dai, A. Wan, P. Zhang, B. Wu, Z. He, Z. Wei, K. Chen, Y . Tian, M. Yu, P. Vajda et al. , “Fbnetv3: Joint architecture-recipe search using predictor pretraining,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021, pp. 16 276–16 285

    2021

  8. [16]

    Curious: Efficient neural archi- tecture search based on a performance predictor and evolutionary search,

    S. Hassantabar, X. Dai, and N. K. Jha, “Curious: Efficient neural archi- tecture search based on a performance predictor and evolutionary search,” IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, vol. 41, no. 11, pp. 4975–4990, 2022

    2022

  9. [17]

    Inference latency prediction at the edge,

    Z. Li, M. Paolieri, and L. Golubchik, “Inference latency prediction at the edge,” arXiv preprint arXiv:2210.02620 , 2022

    2022 arXiv

  10. [18]

    Predicting latency of neural network inference,

    D. M. Mendoza and S. Wang, “Predicting latency of neural network inference,” 2020

    2020

  11. [19]

    A generic graph- based neural architecture encoding scheme for predictor-based nas,

    X. Ning, Y . Zheng, T. Zhao, Y . Wang, and H. Yang, “A generic graph- based neural architecture encoding scheme for predictor-based nas,” in European Conference on Computer Vision . Springer, 2020, pp. 189–204

    2020

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.