Improved Techniques for Training Adaptive Deep Networks

Gao Huang; Hao Li; Hong Zhang; Ruigang Yang; Xiaojuan Qi

REVIEW 2 major objections 3 minor 40 references

Improved Techniques for Training Adaptive Deep Networks

T0 review · 2 major / 3 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Adaptive networks with multiple exits can be trained markedly more effectively by rebalancing gradients and letting classifiers teach each other, yielding consistent accuracy gains at equal compute.

desk verdict Gradient Equilibrium is a real, match-schedule improvement; the ISC/OFA gains are likely overstated by an unequally trained baseline. read the letter →

arxiv 1908.06294 v1 pith:SGXVOQE3 submitted 2019-08-17 cs.CV

Hao Li , Hong Zhang , Xiaojuan Qi , Ruigang Yang , Gao Huang This is my paper · ORCID

classification cs.CV

keywords adaptiveinferenceearlyexitingmulti-exitnetworksgradientequilibriumknowledgedistillationinlinesubnetworkcollaborationMSDNetaccuracy-computetradeoff

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Adaptive (early-exit) networks decide per input how much computation to spend, but most work on them has gone into architecture design rather than training. This paper argues that the standard way of training a multi-exit network, simply summing the cross-entropy losses of all classifiers, is suboptimal because overlapping subnetworks create gradient imbalance and the classifiers do not collaborate. It proposes three training techniques: Gradient Equilibrium (GE), which rescales gradients along backward paths to keep their variance bounded; Inline Subnetwork Collaboration (ISC), which feeds an early classifier's logits into later classifiers as a forward prior; and One-for-all Knowledge Distillation (OFA), which uses the deepest classifier as a teacher for all earlier exits. On CIFAR-10, CIFAR-100, and ImageNet, an MSDNet trained with all three beats the same MSDNet trained with the standard cumulative loss at equal average compute budgets. The significance is that efficiency gains could be obtained by changing how adaptive networks are trained, independently of architectural innovations.

What carries the argument

The central object is the multi-exit adaptive network, instantiated as MSDNet, where several classifiers share convolutional blocks and samples exit early when the predicted confidence exceeds a budget-dependent threshold. GE is the load-bearing training mechanism: it inserts forward-identity, backward-rescaling modules $R(x;s)$ at branches, so the gradient contributed by each downstream classifier is divided by the number of contributing exits, keeping the total backward gradient variance bounded independently of $k$. ISC and OFA add knowledge-transfer paths, identity logit shortcuts forward and KL-divergence distillation backward, to make the exits learn collaboratively rather than as independent heads.

What would settle it

Train the same MSDNet architecture under two schedules: the paper's full GE+ISC+OFA two-phase procedure, and a control that uses the identical two-phase schedule but fine-tunes the classifier heads with plain cross-entropy in phase II. If the control matches or nearly matches the full method's accuracy at equal average compute budgets on CIFAR-100, the central claim that ISC and OFA are responsible for the gains would be refuted. Additionally, measure per-block gradient variance during phase I with and without GE; GE predicts variance bounded by $2\max_l \mathrm{Var}(X_l)$, so unbounded growth without GE and near-flat variance with GE would support the mechanism.

Watch

Extended reading notes

Core claim

The paper's central claim is that the training procedure, not the architecture, is the bottleneck for adaptive inference networks with multiple intermediate classifiers. For a $k$-exit network, summing $k$ cross-entropy losses makes the variance of the gradient at an early block grow with the number of downstream exits; GE renormalizes branch gradients with factors $\frac{1}{k-i+1}$ and $\frac{k-i}{k-i+1}$, yielding gradient variance bounded by $2\max_l \mathrm{Var}(X_l)$, which stabilizes training. ISC adds identity shortcuts from each classifier's logits to the next classifier's input, effectively turning each later classifier into a residual learner over earlier predictions while discarding backward gradients through the shortcut. OFA supervises every intermediate exit with both its own cross-entropy loss and a KL-divergence term against the soft predictions of the last exit. Together these techniques improve every individual classifier's top-1 accuracy, with larger gains at deeper exits for ISC and at shallower exits for OFA, and shift the accuracy-versus-average-compute curve upward on all three datasets.

Load-bearing premise

The reported gains assume a fair comparison with the baseline: the full method includes a second phase that fine-tunes only the classifier heads with OFA and ISC, and the baseline is not described as receiving a matched fine-tuning phase without those additions, so extra training alone could account for part of the improvement.

Editorial extensions

If this is right

Training a multi-exit network with GE instead of a plain sum or average of classifier losses reduces gradient variance at shared early blocks, which should make deeper adaptive networks with more exits trainable.
ISC makes later classifiers behave as residual predictors over earlier predictions, so deeper exits improve even when their own features are weak.
OFA transfers the last classifier's knowledge to every earlier exit, improving early-exit accuracy and thereby improving performance at low average compute budgets.
The techniques are model-agnostic training recipes that should transfer to other multi-exit adaptive architectures without architectural change.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the improvements are not an artifact of the extra fine-tuning phase, the same combination of gradient re-scaling and self-distillation should transfer to any shared-trunk multi-head network, not just adaptive classifiers.
The variance bound suggests GE matters most when the number of exits is large; an untested prediction is that the per-exit accuracy gain grows with $k$, and that GE reduces sensitivity to the loss weights $\lambda_i$.
The identity shortcut used by ISC is a form of residual learning on logits; one could test whether a learned lightweight transform of previous logits improves deep exits further.
Because OFA uses only the network's own last classifier, it needs no external teacher and would combine with future architectures; that also means part of the early-exit accuracy may be credited to distillation rather than to the early features themselves.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Dear colleague,

Short version: the Gradient Equilibrium (GE) piece is a solid, simple trick that appears to improve MSDNet training under a matched schedule (Figure 6), but the paper's main "all three techniques help" claim is undercut by a schedule mismatch in the ablations. The full method trains 300 epochs then fine-tunes classifier heads for 180 more with OFA/ISC; the baseline and GE-only rows get only the 300 epochs. So the per-module gains for ISC and OFA in Tables 2 and 3 are confounded with extra fine-tuning.

That's the thing to know. Now the fair half: the paper is clearly written, the GE idea is novel and principled (re-scaling gradients at branch points to keep variance bounded), and the GE-only ablation (line 2 in Tables 2/3 and Figure 6) is a clean comparison: same schedule, same architecture, only the gradient rescaling differs, and it improves every exit on CIFAR-100 and ImageNet. The OFA/ISC concepts—using early-exit logits as a prior, and distilling the final exit into earlier ones—are reasonable extensions of existing distillation and are presented without overfitting to a particular architecture. Code is provided.

The soft spots are real but not fatal. The missing schedule-matched control for phase II is the main one; head-only fine-tuning for 180 epochs can plausibly improve calibration and accuracy on its own, so "ISC improves E-5 by 1.4%" (Table 2) is not established. Second, the alpha in Eq. 7 is never stated; you cannot reproduce the OFA curves without it. Third, the confidence ranking plot (Figure 7) is nice but not quantitative. The variance bound in Eq. 6 is loose (they get 2 max instead of max) but still valid. The "model-agnostic" claim in the intro is too strong given they only test on MSDNet, though they do acknowledge future work for other adaptive architectures.

Who is this for? Anyone working on multi-exit adaptive inference or training strategies for branched networks. The GE trick alone is worth a close look; the ISC/OFA results should be treated as preliminary until a schedule-matched baseline is run. If this crosses a desk, it deserves a serious referee—a couple of extra experiments would settle the main concern. I'd send it to review rather than desk-reject, and would cite the GE component in related work with a note about the phase-II issue.

Recommendation: engage with it, but ask for the control.

Referee Report

2 major / 3 minor

Summary. The paper proposes three training techniques for multi-exit adaptive networks, evaluated on MSDNet: Gradient Equilibrium (GE), which rescales gradients along shared backbone paths to stabilize training; Inline Subnetwork Collaboration (ISC), which feeds earlier-exit logits into later classifiers; and One-for-all Knowledge Distillation (OFA), which distills the deepest classifier's soft predictions into all earlier classifiers. Experiments on CIFAR-10, CIFAR-100, and ImageNet report that the combined method improves the accuracy-efficiency tradeoff of the MSDNet baseline under budgeted batch classification, and ablation tables report gains for each technique individually.

Significance. If the claims hold, the paper contributes simple, model-agnostic training modifications that could be useful across the multi-exit adaptive-inference literature. The GE idea of bounding gradient variance along shared paths is principled as a training stabilization mechanism and is not used circularly: it is motivated by a variance calculation, not fitted to the results. The ISC and OFA techniques are easy to implement and the paper includes per-exit ablations on two datasets plus a code release, which supports reproducibility. The main significance is limited by the incomplete training-protocol specification for the ablations: the comparison underlying the ISC and OFA claims is not clearly schedule-matched, so the magnitude of the attributed improvements is uncertain until this is resolved.

major comments (2)

[Section 4, 'Training Details'; Tables 2 and 3] The ablations for ISC and OFA are confounded with extra training. The paper states that phase I trains from scratch for 300 epochs and that phase II fine-tunes only the last layer of each classifier with OFA and ISC for 180 epochs, but it does not state that the baseline MSDNet (the first row of Tables 2 and 3, and the 'Baseline' curves) receives a schedule-matched 180-epoch head-only fine-tuning phase without ISC/OFA. Without that control, the per-module gains in Tables 2 and 3 and the curves in Figure 4 could partly reflect the additional fine-tuning epochs rather than the proposed modules. Please report a baseline that receives the identical two-phase schedule with OFA and ISC removed, and specify the training schedule used for every row in Tables 2 and 3.
[Section 3.3, Eq. (7); Section 4, 'Training Details'] The OFA loss in Eq. (7) contains an unstated hyperparameter alpha, but the value of alpha is never reported in the training details or the ablation description. Since alpha controls the tradeoff between cross-entropy and distillation and can change the results substantially, please report the alpha value used for all experiments and state whether it is tuned separately per dataset or fixed across all runs.

minor comments (3)

[Section 3.1, Eq. (6)] The inequality chain in Eq. (6) writes the final bound as 2 max Var(X_l), but the preceding line gives (n max + n(n-1) max)/n^2 = (2 - 1/n) max, so the penultimate displayed step should read '<=' followed by (2 - 1/n) max rather than 2 max. The stated bound is still valid, but the derivation as printed skips a step.
[Section 4, 'Training Details' and Figure 6 caption] The caption of Figure 6 says 'three different depth of networks with 4, 6, 8 exits,' while Section 4.1 reports CIFAR-100 MSDNets with {4, 5, 6, 8} exits and the caption text also mentions 4, 6, 8. Please clarify whether the 5-exit configuration was used in Figure 6 and keep the notation consistent.
[Throughout] There are several typos and wording issues, including 'vise verse' in the Introduction, 'dacay' in Section 4, 'avaliable' in the code statement, and 'Quantitive' in Section 4.3. These should be corrected in a revision.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the training gains are measured against an independently trained MSDNet baseline, and no prediction is constructed from the proposed equations or from fitted parameters.

full rationale

The paper's central claim—that GE, ISC, and OFA improve adaptive MSDNet training—is an empirical comparison against a re-implemented MSDNet baseline, not a quantity derived from the proposed equations. GE re-scales gradients to bound variance in Eq. (6), but the scaling factors are set by the variance argument and are not fitted to the reported accuracy; the improvements in Fig. 6 and Tables 2–3 are measured on held-out data. ISC and OFA modify the training objective (Eq. 7), and their benefits are read directly from the tables, not inferred from the same objective's definition. The citation of MSDNet [14], co-authored by one of the present authors, supplies the architecture and evaluation protocol, but it is not used as proof of the new techniques, nor does it forbid alternative explanations. The only notable weakness is experimental: phase II in Section 4 gives the full model an extra 180-epoch head-only fine-tune that is not schedule-matched in the baseline, which is a potential confound for the per-module ISC/OFA rows; however, confounding is not circularity because the baseline is not constructed from the proposed quantities and no fitted parameter is relabeled as a prediction.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central empirical claim rests on standard ML assumptions (representative validation set, reliable teacher) and a heuristic variance model. One free hyperparameter (α) is not reported. No invented entities.

free parameters (1)

α (OFA weighting)
In Eq. (7), the OFA loss mixes cross-entropy and KL divergence with weight α. The paper never states its value, so the exact loss is not reproducible.

assumptions (4)

domain assumption Gradients from different classifiers have bounded covariance, so the variance of the summed gradient is bounded by a constant times the maximum per-classifier variance.
Used in Eq. (6) to argue GE bounds gradient variance; this is a probabilistic assumption about the loss landscape, not a proven property.
domain assumption The last classifier is the most accurate and can serve as a reliable teacher for earlier classifiers.
Stated in Section 3.3 as intuition; empirically true in the reported tables but not guaranteed for all adaptive networks.
domain assumption Fine-tuning only the last layer in phase II preserves the backbone features learned in phase I.
Assumed by the two-phase training scheme; the paper does not analyze whether backbone drift matters.
domain assumption Confidence thresholds tuned on a held-out validation set transfer to the test distribution.
Standard practice inherited from MSDNet; the threshold search uses validation, then results are reported on test.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Improved Techniques for Training Adaptive Deep Networks." pith.science (2026). https://pith.science/paper/SGXVOQE3

@misc{pith2026190806294,
  author       = {Pith},
  title        = {Pith review of: Improved Techniques for Training Adaptive Deep Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SGXVOQE3}},
  note         = {Machine review of arXiv:1908.06294}
}

read the original abstract

Adaptive inference is a promising technique to improve the computational efficiency of deep models at test time. In contrast to static models which use the same computation graph for all instances, adaptive networks can dynamically adjust their structure conditioned on each input. While existing research on adaptive inference mainly focuses on designing more advanced architectures, this paper investigates how to train such networks more effectively. Specifically, we consider a typical adaptive deep network with multiple intermediate classifiers. We present three techniques to improve its training efficacy from two aspects: 1) a Gradient Equilibrium algorithm to resolve the conflict of learning of different classifiers; 2) an Inline Subnetwork Collaboration approach and a One-for-all Knowledge Distillation algorithm to enhance the collaboration among classifiers. On multiple datasets (CIFAR-10, CIFAR-100 and ImageNet), we show that the proposed approach consistently leads to further improved efficiency on top of state-of-the-art adaptive deep networks.

Figures

Figures reproduced from arXiv: 1908.06294 by the authors.

**Figure 2.** Accuracy (top-1) of budgeted batch classification as a function of average computational budget per image on the CIFAR-10. and all the models are trained for 90 epochs both in phase I and II with learning rate drops after 30 and 60 epochs. Adaptive inference with MSDNet. Following [14], we evaluate our model in the adaptive inference setting. For a given input image, we forward it through the intermediate classifier… view at source ↗

**Figure 3.** Accuracy (top-1) of budgeted batch classification as a function of average computational budget per image on the CIFAR-100. 1 2 3 4 5 6 average budget (in MUL-ADD) ×107 68 69 70 71 72 73 74 75 accuracy (%) GE on CIFAR-100 Baseline Gradient Equilibrium (a) 1 2 3 4 5 6 average budget (in MUL-ADD) ×107 68 69 70 71 72 73 74 75 accuracy (%) ISC on CIFAR-100 Baseline Inline Subnetwork Collaboration (b) 1 2 3 4 5 6 average… view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Ablation results produced by integrating [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

**Figure 5.** Figure 5: Top-1 accuracy of budgeted batch classification as a function of average computational budget per image on ImageNet. layers. Besides, compared with ResNets and DenseNets, adaptive inference based approaches (MSDNet with and without our training strategies) perform sign…

**Figure 6.** Figure 6: Validation accuracy on CIFAR-100 at different epochs. We plot the results of three different depth of networks with 4, 6, 8 exits respectively. The losses of the models trained with GE are consistently lower than the baseline models. Gradient Equilibrium. To further ev…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

40 extracted references · 33 canonical work pages

[1]

Do deep nets really need to be deep? In NIPS, 2014

Jimmy Ba and Rich Caruana. Do deep nets really need to be deep? In NIPS, 2014. 2

work page 2014
[2]

Adaptive neural networks for fast test-time pre- diction

Tolga Bolukbasi, Joseph Wang, Ofer Dekel, and Venkatesh Saligrama. Adaptive neural networks for fast test-time pre- diction. In ICML, 2017. 1, 2

work page 2017
[3]

Model compression

Cristian Bucilua, Rich Caruana, and Alexandru Niculescu- Mizil. Model compression. In ACM SIGKDD, 2006. 2

work page 2006
[4]

Compressing convolutional neural networks in the frequency domain

Wenlin Chen, James Wilson, Stephen Tyree, Kilian Q Wein- berger, and Yixin Chen. Compressing convolutional neural networks in the frequency domain. In ACM SIGKDD, 2016. 1, 2

work page 2016
[5]

Imagenet: A large-scale hierarchical image database

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009. 5

2009
[6]

Spatially Adaptive Computation Time for Residual Networks

Michael Figurnov, Maxwell D Collins, Yukun Zhu, Li Zhang, Jonathan Huang, Dmitry Vetrov, and Ruslan Salakhutdinov. Spatially adaptive computation time for residual networks. arXiv preprint arXiv:1612.02297, 2016. 1, 2, 8

work page Pith review arXiv 2016
[7]

Adaptive computation time for recurrent neural networks

Alex Graves. Adaptive computation time for recurrent neural networks. arXiv preprint arXiv:1603.08983, 2016. 1

arXiv 2016
[8]

Deep com- pression: Compressing deep neural networks with pruning, trained quantization and huffman coding

Song Han, Huizi Mao, and William J Dally. Deep com- pression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In ICLR, 2016. 1, 2

work page 2016

Show all 40 references

[9]

Mask r-cnn

Kaiming He, Georgia Gkioxari, Piotr Doll ´ar, and Ross Gir- shick. Mask r-cnn. In ICCV, 2017. 1

2017
[10]

Deep residual learning for image recognition

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR,
[11]

Amc: Automl for model compression and accel- eration on mobile devices

Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. Amc: Automl for model compression and accel- eration on mobile devices. In ECCV, 2018. 2

2018
[12]

Distilling the knowledge in a neural network

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. In NIPS Deep Learning Workshop, 2014. 2, 4

2014
[13]

Mobilenets: Efﬁcient convolu- tional neural networks for mobile vision applications

Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco An- dreetto, and Hartwig Adam. Mobilenets: Efﬁcient convolu- tional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017. 2, 6

2017 arXiv
[14]

Multi-scale dense networks for resource efﬁcient image classiﬁcation

Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Laurens van der Maaten, and Kilian Q Weinberger. Multi-scale dense networks for resource efﬁcient image classiﬁcation. InICLR,
[15]

Condensenet: An efﬁcient densenet using learned group convolutions

Gao Huang, Shichen Liu, Laurens Van der Maaten, and Kil- ian Q Weinberger. Condensenet: An efﬁcient densenet using learned group convolutions. In CVPR, 2018. 1, 2

2018
[16]

Densely connected convolutional networks

Gao Huang, Zhuang Liu, Kilian Q Weinberger, and Laurens van der Maaten. Densely connected convolutional networks. In CVPR, 2017. 1, 5, 7

2017
[17]

Binarized neural networks

Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El- Yaniv, and Yoshua Bengio. Binarized neural networks. In NIPS, 2016. 1, 2

2016
[18]

Incorporating side information by adaptive convolution

Di Kang, Debarun Dhar, and Antoni Chan. Incorporating side information by adaptive convolution. In NIPS, 2017. 2

2017
[19]

Pixel-wise attentional gating for parsimonious pixel labeling

Shu Kong and Charless Fowlkes. Pixel-wise attentional gating for parsimonious pixel labeling. arXiv preprint arXiv:1805.01556, 2018. 2

2018 arXiv
[20]

Learning multiple layers of features from tiny images

Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech Report, 2009. 5

2009
[21]

Imagenet classiﬁcation with deep convolutional neural net- works

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classiﬁcation with deep convolutional neural net- works. In NIPS, 2012. 1

2012
[22]

Knowledge distillation by on-the-ﬂy native ensemble

Xu Lan, Xiatian Zhu, and Shaogang Gong. Knowledge distillation by on-the-ﬂy native ensemble. arXiv preprint arXiv:1806.04606, 2018. 2

2018 arXiv
[23]

Pruning ﬁlters for efﬁcient convnets

Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning ﬁlters for efﬁcient convnets. In ICLR, 2017. 1, 2

2017
[24]

Dynamic computational time for visual attention

Zhichao Li, Yi Yang, Xiao Liu, Feng Zhou, Shilei Wen, and Wei Xu. Dynamic computational time for visual attention. In ICCV, 2017. 2

2017
[25]

Runtime neural pruning

Ji Lin, Yongming Rao, Jiwen Lu, and Jie Zhou. Runtime neural pruning. In NIPS, 2017. 2

2017
[26]

Learning efﬁcient convolutional networks through network slimming

Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efﬁcient convolutional networks through network slimming. InICCV,
[27]

Fully convolutional networks for semantic segmentation

Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR, 2015. 1

2015
[28]

Shufﬂenet v2: Practical guidelines for efﬁcient cnn architec- ture design

Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufﬂenet v2: Practical guidelines for efﬁcient cnn architec- ture design. In ECCV, 2018. 1, 2

2018
[29]

Recurrent segmentation for variable com- putational budgets

Lane McIntosh, Niru Maheswaranathan, David Sussillo, and Jonathon Shlens. Recurrent segmentation for variable com- putational budgets. In CVPR Workshops, 2018. 2

2018
[30]

Mobilenetv2: Inverted residuals and linear bottlenecks

Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zh- moginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018. 1, 2

2018
[31]

Collaborative learning for deep neural networks

Guocong Song and Wei Chai. Collaborative learning for deep neural networks. In NIPS, 2018. 1

2018
[32]

Going deeper with convolutions

Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, 2015. 1, 7

2015
[33]

Branchynet: Fast inference via early exiting from deep neu- ral networks

Surat Teerapittayanon, Bradley McDanel, and HT Kung. Branchynet: Fast inference via early exiting from deep neu- ral networks. In ICPR, 2016. 1

2016
[34]

Hydranets: Specialized dynamic archi- tectures for efﬁcient inference

Ravi Teja Mullapudi, William R Mark, Noam Shazeer, and Kayvon Fatahalian. Hydranets: Specialized dynamic archi- tectures for efﬁcient inference. In CVPR, 2018. 1, 7

2018
[35]

Convolutional networks with adaptive inference graphs

Andreas Veit and Serge Belongie. Convolutional networks with adaptive inference graphs. In ECCV, 2018. 1, 2

2018
[36]

Skipnet: Learning dynamic routing in convolutional networks

Xin Wang, Fisher Yu, Zi-Yi Dou, Trevor Darrell, and Joseph E Gonzalez. Skipnet: Learning dynamic routing in convolutional networks. In ECCV, 2018. 2

2018
[37]

Blockdrop: Dynamic inference paths in residual networks

Zuxuan Wu, Tushar Nagarajan, Abhishek Kumar, Steven Rennie, Larry S Davis, Kristen Grauman, and Rogerio Feris. Blockdrop: Dynamic inference paths in residual networks. In CVPR, 2018. 2

2018
[38]

Depth-adaptive com- putational policies for efﬁcient visual tracking

Chris Ying and Katerina Fragkiadaki. Depth-adaptive com- putational policies for efﬁcient visual tracking. In EMM- CVPR, 2017. 2

2017
[39]

Shufﬂenet: An extremely efﬁcient convolutional neural net- work for mobile devices

Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufﬂenet: An extremely efﬁcient convolutional neural net- work for mobile devices. In CVPR, 2018. 2, 6

2018
[40]

Learning transferable architectures for scalable image recognition

Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In CVPR, 2018. 1, 2, 6

2018

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Do deep nets really need to be deep? In NIPS, 2014

Jimmy Ba and Rich Caruana. Do deep nets really need to be deep? In NIPS, 2014. 2

work page 2014

[2] [2]

Adaptive neural networks for fast test-time pre- diction

Tolga Bolukbasi, Joseph Wang, Ofer Dekel, and Venkatesh Saligrama. Adaptive neural networks for fast test-time pre- diction. In ICML, 2017. 1, 2

work page 2017

[3] [3]

Model compression

Cristian Bucilua, Rich Caruana, and Alexandru Niculescu- Mizil. Model compression. In ACM SIGKDD, 2006. 2

work page 2006

[4] [4]

Compressing convolutional neural networks in the frequency domain

Wenlin Chen, James Wilson, Stephen Tyree, Kilian Q Wein- berger, and Yixin Chen. Compressing convolutional neural networks in the frequency domain. In ACM SIGKDD, 2016. 1, 2

work page 2016

[5] [5]

Imagenet: A large-scale hierarchical image database

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009. 5

2009

[6] [6]

Spatially Adaptive Computation Time for Residual Networks

Michael Figurnov, Maxwell D Collins, Yukun Zhu, Li Zhang, Jonathan Huang, Dmitry Vetrov, and Ruslan Salakhutdinov. Spatially adaptive computation time for residual networks. arXiv preprint arXiv:1612.02297, 2016. 1, 2, 8

work page Pith review arXiv 2016

[7] [7]

Adaptive computation time for recurrent neural networks

Alex Graves. Adaptive computation time for recurrent neural networks. arXiv preprint arXiv:1603.08983, 2016. 1

arXiv 2016

[8] [8]

Deep com- pression: Compressing deep neural networks with pruning, trained quantization and huffman coding

Song Han, Huizi Mao, and William J Dally. Deep com- pression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In ICLR, 2016. 1, 2

work page 2016