Pith. sign in

REVIEW 23 cited by

Once-for-All: Train One Network and Specialize it for Efficient Deployment

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 1908.09791 v5 pith:F7ITQ424 submitted 2019-08-26 cs.LG cs.CVstat.ML

Han Cai , Chuang Gan , Tianzhe Wang , Zhekai Zhang , Song Han This is my paper · ORCID
classification cs.LGcs.CVstat.ML
keywords manyaccuracydevicesnetworktrainconstraintslatencyonce-for-all
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

We address the challenging problem of efficient inference across many devices and resource constraints, especially on edge devices. Conventional approaches either manually design or use neural architecture search (NAS) to find a specialized neural network and train it from scratch for each case, which is computationally prohibitive (causing $CO_2$ emission as much as 5 cars' lifetime) thus unscalable. In this work, we propose to train a once-for-all (OFA) network that supports diverse architectural settings by decoupling training and search, to reduce the cost. We can quickly get a specialized sub-network by selecting from the OFA network without additional training. To efficiently train OFA networks, we also propose a novel progressive shrinking algorithm, a generalized pruning method that reduces the model size across many more dimensions than pruning (depth, width, kernel size, and resolution). It can obtain a surprisingly large number of sub-networks ($> 10^{19}$) that can fit different hardware platforms and latency constraints while maintaining the same level of accuracy as training independently. On diverse edge devices, OFA consistently outperforms state-of-the-art (SOTA) NAS methods (up to 4.0% ImageNet top1 accuracy improvement over MobileNetV3, or same accuracy but 1.5x faster than MobileNetV3, 2.6x faster than EfficientNet w.r.t measured latency) while reducing many orders of magnitude GPU hours and $CO_2$ emission. In particular, OFA achieves a new SOTA 80.0% ImageNet top-1 accuracy under the mobile setting ($<$600M MACs). OFA is the winning solution for the 3rd Low Power Computer Vision Challenge (LPCVC), DSP classification track and the 4th LPCVC, both classification track and detection track. Code and 50 pre-trained models (for many devices & many latency constraints) are released at https://github.com/mit-han-lab/once-for-all.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 23 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Elastic ViTs from Pretrained Models without Retraining

    cs.CV 2025-10 conditional novelty 6.0 of 10

    A single-shot, label-free, retraining-free structured pruning method generates elastic ViTs at any sparsity by reweighting gradient-based importance scores with block correlations learned by an evolutionary strategy.

  2. NN-Former: Rethinking Graph Structure in Neural Architecture Representation

    cs.LG 2025-07 conditional novelty 6.0 of 10

    NN-Former improves neural accuracy and latency prediction by using attention masks over sibling nodes in the architecture graph.

  3. Language Embedding Meets Dynamic Graph: A New Exploration for Neural Architecture Representation Learning

    cs.LG 2025-06 conditional novelty 6.0 of 10

    LeDG-Former improves neural architecture latency prediction by combining BERT-based language embeddings of architectures and hardware with dynamic graph self-attention, and reports SOTA on NNLQP.

  4. FlexGS: Train Once, Deploy Everywhere with Many-in-One Flexible 3D Gaussian Splatting

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A single 3DGS model with a learned Gaussian selector and transform field renders at any requested compression ratio without fine-tuning.

  5. Heterogeneous Resource Allocation with Multi-task Learning for Wireless Networks

    cs.LG 2025-02 conditional novelty 6.0 of 10

    A DNN with a learned binary router jointly solves fourteen wireless power allocation tasks of varying dimensionality and objectives, nearing single-task performance.

  6. EfficientLLM: Scalable Pruning-Aware Pretraining for Architecture-Agnostic Edge Language Models

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Pruning-aware pretraining iteratively deletes low-saliency parameter groups from a larger pretrained LLM during continued training, yielding sub-1B EfficientLLM models that outperform direct-pretraining baselines.

  7. Training-free Neural Architecture Search through Variance of Knowledge of Deep Network Weights

    cs.CV 2025-02 conditional novelty 6.0 of 10

    VKDNW scores untrained image classifiers by the entropy of eigenvalue deciles of a Fisher information matrix sampled from one weight per layer, and this ranking outperforms prior zero-cost NAS proxies.

  8. SuperSFL: Resource-Heterogeneous Federated Split Learning with Weight-Sharing Super-Networks

    cs.DC 2026-01 conditional novelty 5.0 of 10

    A weight-sharing super-network plus locally supervised gradient fusion makes split-federated learning converge in fewer communication rounds than fixed-split baselines.

  9. Animal Re-Identification on Microcontrollers

    cs.CV 2025-12 conditional novelty 5.0 of 10

    A width-and-depth-scaled MobileNetV2 quantized to INT8 performs competitive 64x64 animal re-identification on an Arduino Nano 33 BLE Sense.

  10. ESM: A Framework for Building Effective Surrogate Models for Hardware-Aware Neural Architecture Search

    cs.LG 2025-08 conditional novelty 5.0 of 10

    A new encoding and iterative training-set expansion method for latency prediction in hardware-aware neural architecture search, reporting accuracy gains on GPU, CPU, and embedded targets.

  11. Coflex: Enhancing HW-NAS with Sparse Gaussian Processes for Efficient and Scalable DNN Accelerator Design

    cs.LG 2025-07 conditional novelty 5.0 of 10

    Coflex applies sparse Gaussian processes to multi-objective hardware-aware NAS, claiming near-linear scaling and superior Pareto fronts for DNN accelerator co-design.

  12. Loss Functions for Predictor-based Neural Architecture Search

    cs.LG 2025-06 conditional novelty 5.0 of 10

    Weighted losses identify top architectures best with enough training data, ranking losses win with very few data, and switching between them (PWLNAS) gives small consistent gains in predictor-based NAS.

  13. Adaptive Routing for Efficient Diffusion Transformer-Based PNI Prediction

    cs.CV 2026-07 conditional novelty 4.5 of 10

    DiT-RC predicts cholangiocarcinoma PNI from tumor-centered 3D MRI with AUC 0.731 at 257.57 GFLOPs by routing attention heads, tokens, and MLP width in a diffusion classifier.

  14. SAR-NAS: Lightweight SAR Object Detection with Neural Architecture Search

    cs.CV 2025-09 conditional novelty 4.0 of 10

    Applying a one-shot evolutionary NAS to YOLOv10 channel widths yields SAR-NAS-S and SAR-NAS-N, which modestly improve mAP on SARDet-100K over YOLOv10 baselines while cutting parameters and FLOPs.

  15. DANCE: Resource-Efficient Neural Architecture Search with Data-Aware and Continuous Adaptation

    cs.LG 2025-07 reject novelty 4.0 of 10

    DANCE reformulates NAS as continuous evolution via learned stochastic gates over feature dimensions, but its headline 'consistently outperforms' claim fails on CIFAR-10 and is undermined by weak baselines.

  16. EfficientQuant: An Efficient Post-Training Quantization for CNN-Transformer Hybrid Models on Edge Devices

    cs.CV 2025-06 reject novelty 4.0 of 10

    EfficientQuant applies uniform weight quantization to CNN blocks and logarithmic activation quantization to transformer blocks in hybrid models, reporting latency reductions of 2.5x to 8.7x with modest accuracy loss.

  17. Greening AI-enabled Systems with Software Engineering: A Research Agenda for Environmentally Sustainable AI Practices

    cs.SE 2025-06 accept novelty 4.0 of 10

    A 29-participant workshop synthesized a research agenda for reducing AI's environmental footprint through software engineering, covering measurement, benchmarking, architecture, empirical methods, and education.

  18. Searching Neural Architectures for Sensor Nodes on IoT Gateways

    cs.LG 2025-05 conditional novelty 4.0 of 10

    GatewayNAS adapts the hardware-aware neural architecture search space to the time and energy budget of an IoT gateway, producing tiny CNNs for sensor nodes without cloud data transfer.

  19. Fine-Tuning Federated Learning-Based Intrusion Detection Systems for Transportation IoT

    cs.LG 2025-02 reject novelty 4.0 of 10

    A server-edge fine-tuning federated learning framework claims up to 42% memory savings and 75% training time savings with 99.2% intrusion detection accuracy on NSL-KDD, though the evaluation design uses the test set f...

  20. Synergistic Effects of Knowledge Distillation and Structured Pruning for Self-Supervised Speech Models

    eess.AS 2025-02 conditional novelty 4.0 of 10

    Combining knowledge distillation with l0 or low-rank pruning improves compressed RNN-T ASR, and joint pruning with fine-tuning gives 8.9% and 13.4% relative WER gains over baseline.

  21. Streaming Chain

    cs.CR 2026-04 conditional novelty 3.5 of 10

    Prune then INT8 QAT then KD yields a better measured accuracy–size–latency trade-off on CIFAR CNNs than any single stage, and stage order itself changes accuracy.

  22. SHeRL-FL: When Representation Learning Meets Split Learning in Hierarchical Federated Learning

    cs.LG 2025-08 unverdicted novelty 2.0 of 10

    The submitted body is an unrelated survey, not the SHeRL-FL method claimed in the metadata.

  23. Frugal Machine Learning for Energy-efficient, and Resource-aware Artificial Intelligence

    cs.LG 2025-06 conditional novelty 1.0 of 10

    A survey paper that defines and categorizes Frugal Machine Learning methods but introduces no new techniques or empirical results.

Pith tools