Pith. sign in

REVIEW 4 major objections 4 minor 15 references

Optimal Depth of Neural Networks

T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Optimal depth of a residual network is the solution to an optimal stopping problem, and the expected optimal stopping depth is provably finite.

desk verdict A clean optimal-stopping framing undercut by an overclaimed theorem and an empirical table that cannot be right. read the letter →

arxiv 2506.16862 v1 pith:ZZKBYBJU submitted 2025-06-20 cs.LG math.OC

Qian Qi This is my paper · ORCID
classification cs.LGmath.OC MSC 62L1568T0760G40
keywords OptimalStoppingDeepLearningTheoryResidualNetworksSequentialDecisionProcessesRegularizationHamilton-Jacobi-BellmanEquationEarlyExiting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the depth of a residual network is not something to be fixed by architecture search: it is an optimal stopping problem. At each layer the forward pass can either halt and predict, collecting reward $g(h_l)-c\cdot l$, or continue to the next residual block $h_{l+1}=h_l+f_l(h_l)$. The main theoretical claim is that, under a plausible diminishing-returns condition on the residual functions, the expected optimal stopping depth is finite even in an infinite-horizon network, giving a principled account of why 'deeper is better' fails. The paper converts this into a regularizer $L_{\rm depth}=\sum_l w_l\,\mathbb{E}\|f_l(h_l)\|_2^2$ and an adaptive inference rule, and reports on ImageNet that the regularized ResNet-50 dominates static and entropy-based early-exit baselines on accuracy versus compute. The same framework is extended to Transformers and to a continuous-depth free-boundary formulation.

What carries the argument

The machinery is the optimal-stopping reformulation of the ResNet update. The state is the hidden representation $h_l$, the dynamics are $h_{l+1}=h_l+f_l(h_l)$, and the reward for stopping is $Y_l=g(h_l)-c\cdot l$; the Snell envelope computes the value of continuing at each layer. The load-bearing objects are the residual functions $f_l$ themselves: bounding their expected norm (Proposition 4.3) is how the paper argues utility gains become sublinear in depth. The practical fork is the regularizer $L_{\rm depth}$ and the one-step lookahead stop rule, which replaces the intractable Snell envelope with the condition $g(h_{l+1})-g(h_l)\le c$.

What would settle it

Reproduce the ResNet-50 ImageNet training of Section 10 with the stated hyperparameters ($\beta=0.5$, $\lambda=1.0$, $w_l=(l/L)^2$), then measure average stopping depth and Top-1 accuracy under the ASTI rule at $c=0.001$. The paper reports 4.11 G-FLOPs and 73.1% accuracy for this configuration; if the measured average stopping depth is not essentially the full depth, the row cannot be reproduced as reported, and if it is full depth, the configuration provides no computational saving over the full model.

Watch

Extended reading notes

Core claim

The paper's central claim is that the forward pass of a trained ResNet is a Markov process whose optimal input-dependent depth is the solution of an optimal stopping problem with reward $Y_l=g(h_l)-c\cdot l$. The value function is the Snell envelope, and the optimal policy is to stop at the first layer where the immediate reward meets or exceeds the expected value of continuing. The paper proves in Theorem 4.4 that, in the infinite-horizon limit, the expected optimal stopping time $\mathbb{E}[\tau^*]$ is finite once residual outputs are small enough on average that expected utility gain falls below the per-layer cost; the bound is obtained by sandwiching the value between the reward at a fixed layer $L_0$ and $g_{\max}-c\mathbb{E}[\tau^*]$. It then turns this normative model into a training objective by penalizing squared residual norms with layer-dependent weights, and into an inference algorithm by approximating the Snell-envelope rule with a one-step lookahead: stop when $g(h_{l+1})-g(h_l)\le c$ and predict from $h_{l+1}$.

Load-bearing premise

The paper's practical claims rest on the ImageNet measurements in Table 1 being genuine experimental results; if the 'final corrected simulation' entries, particularly ASTI (c=0.001) with full-network compute and higher accuracy than the full-depth model, are not real trained-model outcomes, the evidence for the regularizer's efficiency gains collapses.

Editorial extensions

If this is right

  • If the finiteness theorem is right, a trained residual network has a finite optimal inference depth per input, so always running to the last layer is provably suboptimal once the per-layer cost is positive.
  • Training with $L_{\rm depth}$ and an increasing weight schedule should drive deep residual blocks toward the identity map, producing the measured decay in average residual norms that Figure 1 reports.
  • At inference the single parameter $c$ slides the model along the accuracy-computation frontier, so one model can serve low-cost and high-accuracy regimes without retraining.
  • The same reward structure and regularizer apply to Transformers because a Transformer block is also a residual update $h_{l+1}=h_l+\mathrm{Block}_l(h_l)$.
  • In the continuous-depth limit the stopping set and continuation set are separated by a free boundary that solves an HJB variational inequality, connecting early-exit depth to free-boundary theory.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The proof of Theorem 4.4 as written never invokes Proposition 4.3: the finite bound on $\mathbb{E}[\tau^*]$ follows already from boundedness of $g$ and positivity of $c$, so the diminishing-returns condition named in the abstract is not actually the load-bearing part of the theorem.
  • A clean way to separate the regularizer from the stopping rule would be to run ASTI on a vanilla pretrained ResNet: since vanilla residuals do not decay with depth, the one-step lookahead rule should rarely stop early, which would show how much of the reported efficiency comes from $L_{\rm depth}$ rather than from the rule itself.
  • The paper reports that the ASTI-trained full network is more accurate than the vanilla baseline (72.2% vs 69.8%); an ablation removing intermediate supervision and $L_{\rm depth}$ separately would determine whether this accuracy gain is due to depth regularization or to the auxiliary heads.
  • The practical rule uses a one-step lookahead approximation of the Snell envelope; comparing ASTI's decisions with exact backward induction on a small dataset would quantify the approximation error the paper leaves implicit.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper frames the forward pass of a ResNet as an infinite-horizon optimal stopping problem: at each layer the network either stops and predicts from the current representation or continues at cost c, with reward g(h_l) - c*l. It proves a finite expected stopping depth under a proposed diminishing-returns condition on the residual functions, derives an L_depth regularizer intended to induce that condition, and proposes the ASTI inference algorithm based on a one-step lookahead approximation. The paper also sketches extensions to Transformers and to the continuous-depth free-boundary setting, and reports ImageNet/ResNet-50 experiments as validation. The central theoretical claim is that diminishing returns imply finite optimal expected depth; the central practical claim is that ASTI achieves a superior accuracy-efficiency trade-off.

Significance. If the advertised results held, framing neural depth as optimal stopping would be a useful conceptual contribution and the regularizer might have practical value. The paper is clearly structured, the optimal-stopping setup is standard and mostly well stated, and the HJB connection in Section 7 is reasonable as a research direction. However, the main theorem does not actually use the diminishing-returns condition, the ASTI stopping rule is derived from an unvalidated approximation, and the empirical validation is circular with respect to the regularizer and internally inconsistent in Table 1. These are load-bearing issues: the abstract's primary theoretical contribution and the paper's main efficiency claims are not supported as written.

major comments (4)
  1. [Section 4.2, Theorem 4.4; Appendix A.3] The proof of Theorem 4.4 does not invoke Proposition 4.3 or the diminishing-returns hypothesis at any point. The derivation only uses boundedness of g (Assumption A1) and positivity of c: after establishing V = E[g(h_{tau*})] - c E[tau*], Step 4 bounds E[g(h_{tau*})] by g_max and rearranges to obtain E[tau*] <= (g_max - E[g(h_L0)])/c + L0. This argument would go through identically even if the residual norms E[||f_l(h_l)||] were large at every layer. Consequently the abstract's claim that finite expected optimal depth is proved 'under a plausible condition of diminishing returns' is not accurate; the advertised primary theoretical contribution is not a theorem about diminishing returns and should be restated or strengthened.
  2. [Section 9.1, Eq. (17)] The practical stopping rule is obtained by replacing the continuation value E[U_{l+1}|h_l] with E[Y_{l+1}|h_l] and then, because the transition is deterministic for a fixed input, with the realized value Y_{l+1}. This conflates a conditional expectation over the data distribution with the actual next-state reward on a single input, and no bound or experiment shows that the replacement is a faithful approximation of the Snell envelope. Since ASTI's status as a 'theoretically grounded' algorithm rests on this step, the authors should either provide a quantitative approximation guarantee or present the rule as a heuristic and validate it against a computed or estimated optimal policy.
  3. [Table 1 and Section 10.2.2] The ASTI (c = 0.001) row reports an average of 4.11 G-FLOPs, identical to the full-depth ASTI row, while its Top-1 accuracy is 73.1%, 0.9 points higher than the full-depth ASTI model's 72.2%. Because both rows use the same trained weights and c = 0.001 is a very small threshold, essentially full-depth inference should reproduce the full-depth outputs and accuracy; if a nontrivial fraction of inputs exits early, the average FLOPs should decrease noticeably. The table note 'Data from final corrected simulation' also leaves it unclear whether any of these numbers come from actual training and inference runs. This table is the main support for the paper's accuracy-efficiency claim, so it needs to be replaced with real measured results, including exit rates, standard deviations, and training details.
  4. [Section 10.2.1, Figure 1; Eq. (11)] Figure 1 plots E[||f_l(h_l)||^2], which is exactly the quantity penalized by L_depth in Eq. (11). Observing that this quantity decays after training with L_depth is therefore a check that the optimizer minimized the regularizer, not independent evidence of the theoretically predicted diminishing-returns behavior. The 'smoking gun' language is not supported unless the comparison also reports quantities not directly in the training objective, such as the induced expected utility increments or the behavior of a model trained with intermediate supervision but without L_depth.
minor comments (4)
  1. [Section 5.1 and Section 10.1.2] The paper proposes a polynomial schedule w_l = (l+1)^alpha in Section 5.1 but reports using w_l = (l/L)^2 in Section 10.1.2; please state the exact schedule used for all reported results and clarify whether alpha was tuned.
  2. [Section 10.1.2] The training protocol omits important details such as the number of epochs, batch size, learning rate schedule, data augmentation, weight decay, and number of seeds; the reported accuracies also lack error bars, which is necessary to support the claimed Pareto-frontier superiority.
  3. [Section 9.1, Remark 9.1] The discussion of 'look-then-leap' stopping is confusing: the algorithm decides at layer l after computing h_{l+1} and then returns a prediction from h_{l+1}, so the formal stopping time is l+1, not a decision made before observing h_{l+1}. Clarify the timing and the filtration used for the decision.
  4. [Section 10.2.2] The claim that ASTI (c = 0.005) 'dramatically outperforms' the best entropy-based baseline (70.6% vs. 64.3%) should be accompanied by the threshold-setting details for the entropy method; otherwise the comparison is not reproducible.

Circularity Check

1 steps flagged · score 6.0 of 10

Figure 1 'confirms' a behavior that Ldepth enforces by construction: the plotted E[||f_l||^2] is exactly the quantity penalized with depth-increasing weights, so the empirical validation of diminishing returns is circular.

  1. self definitional [Section 10.2.1 / Figure 1; Section 5 Eq. (11); Section 5.1]
    "Figure 1 provides "smoking gun" evidence for the efficacy of our proposed regularizer. It plots the average L2-norm of the residual function's output, E[||fl(hl)||^2]... In stark contrast, the model trained with our Ldepth regularizer (blue, solid) exhibits precisely the behavior predicted by our theory: the residual norms decay smoothly and monotonically with depth. This confirms that the regularizer successfully encourages the network to learn functions that satisfy the diminishing returns condition (Proposition 4.3)."

    Ldepth is defined (Eq. 11) as a sum over layers of w_l times E[||f_l(h_l)||^2], with depth-increasing weights (Section 5.1 chooses w_l=(l/L)^2), and the paper states this regularizer 'directly encourages the condition in Proposition 4.3.' Figure 1 then plots exactly E[||f_l(h_l)||^2] for a model trained with that regularizer. The observed decay with depth is therefore a check that the training objective was minimized, not an independent empirical confirmation of a theoretical prediction. The 'diminishing returns condition' (small expected residual norms at depth) is enforced by construction, so the claimed validation is self-definitional rather than evidence for the theory.

full rationale

The one genuinely circular step is the empirical validation of diminishing returns: Ldepth is a depth-weighted penalty on E[||f_l(h_l)||^2], and Figure 1 reports that this exact loss term decays with depth. That is an optimization outcome, not an independent prediction. Separate issues are not circular but are serious: Theorem 4.4's proof never uses Proposition 4.3's diminishing-returns condition; finiteness follows from boundedness of g and positive linear cost, a standard optimal-stopping bound, so the advertised dependence on diminishing returns is an overclaim. Table 1 is also internally inconsistent and is explicitly sourced to 'Data from final corrected simulation': the ASTI (c=0.001) row reports full-depth FLOPs yet a 0.9-point accuracy gain over the full-depth ASTI model, which cannot arise if early exits are as rare as the FLOPs imply. These are evidentiary/consistency problems rather than circularity. The self-citation (Qi 2025) is introductory and not load-bearing, and no uniqueness theorem is imported. Overall, the central theoretical result is independent, but a central empirical claim—that Ldepth produces the theoretically predicted diminishing-return structure—reduces by construction to the regularizer's own objective, warranting a partial-circularity score of 6.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The framework rests on standard optimal stopping theory plus a set of assumptions. The diminishing returns condition is advertised as the key condition but is not used in the main theorem; the one-step lookahead approximation is unvalidated; and the empirical validation directly penalizes the quantity it reports.

free parameters (4)
  • lambda (L_depth strength) = 1.0
    Selected by a preliminary hyperparameter sweep on a held-out validation set (Section 10.1.2).
  • beta (intermediate supervision weight) = 0.5
    Selected by the same sweep (Section 10.1.2).
  • weight schedule exponent = quadratic (l/L)^2
    Chosen to make the depth penalty increase with depth (Sections 5.1 and 10.1.2).
  • inference cost c = 0.001 and 0.005
    User-defined hyperparameter controlling the accuracy-efficiency trade-off at test time (Section 9.2.2, Table 1).
assumptions (5)
  • domain assumption The utility function g is bounded (A1).
    Assumed in Section 4.1 to bound the value function. It is natural for accuracy-like metrics but is the only assumption actually used in the proof of Theorem 4.4.
  • domain assumption Hidden state trajectories stay in a compact set and g is Lipschitz on it (A2).
    Assumed in Section 4.1; the paper notes it can be enforced architecturally, but no enforcement is reported in the ImageNet experiments.
  • ad hoc to paper Diminishing returns condition: E[||f_l(h_l)||] <= delta for l >= L0 (hypothesis of Proposition 4.3).
    Stated as a plausible condition in Sections 4.1 and 4.2. The regularizer is designed to enforce it, and the proof of Theorem 4.4 never uses it.
  • ad hoc to paper One-step lookahead approximation E[U_{l+1}|h_l] ≈ E[Y_{l+1}|h_l] in Section 9.1.
    Assumed without error bound to derive the ASTI stopping rule. All practical efficiency claims depend on this approximation.
  • domain assumption The utility function g(h_l) = max softmax(head(h_l)) is a valid Lipschitz utility.
    Adopted in Section 9.2.1 to instantiate the theory; no comparison against alternative utility choices is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Optimal Depth of Neural Networks." pith.science (2026). https://pith.science/paper/ZZKBYBJU

@misc{pith2026250616862,
  author       = {Pith},
  title        = {Pith review of: Optimal Depth of Neural Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZZKBYBJU}},
  note         = {Machine review of arXiv:2506.16862}
}
abstract

Determining the optimal depth of a neural network is a fundamental yet challenging problem, typically resolved through resource-intensive experimentation. This paper introduces a formal theoretical framework to address this question by recasting the forward pass of a deep network, specifically a Residual Network (ResNet), as an optimal stopping problem. We model the layer-by-layer evolution of hidden representations as a sequential decision process where, at each layer, a choice is made between halting computation to make a prediction or continuing to a deeper layer for a potentially more refined representation. This formulation captures the intrinsic trade-off between accuracy and computational cost. Our primary theoretical contribution is a proof that, under a plausible condition of diminishing returns on the residual functions, the expected optimal stopping depth is provably finite, even in an infinite-horizon setting. We leverage this insight to propose a novel and practical regularization term, $\mathcal{L}_{\rm depth}$, that encourages the network to learn representations amenable to efficient, early exiting. We demonstrate the generality of our framework by extending it to the Transformer architecture and exploring its connection to continuous-depth models via free-boundary problems. Empirical validation on ImageNet confirms that our regularizer successfully induces the theoretically predicted behavior, leading to significant gains in computational efficiency without compromising, and in some cases improving, final model accuracy.

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 13 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION format.url url empty "" url if FUNCTION article output.bibitem format.authors "author" output.check author format.key output output.year.check new.block format.title "title" output.check new.block crossref missing format.jour.vol output format.article.crossref output.nonnull format.pages output if ne...

  2. [2]

    Ishmael M

    binproceedings [author] Belghazi , M. Ishmael M. I. , Baratin , Aristide A. , Rajesh , Sai S. , Ozair , Sherjil S. , Bengio , Yoshua Y. , Courville , Aaron A. Hjelm , R. Devon R. D. ( 2018 ). Mutual Information Neural Estimation . In Proceedings of the 35th International Conference on Machine Learning (ICML) . Proceedings of Machine Learning Research 80 5...

    work page 2018

  3. [3]

    binproceedings [author] Chen , Ricky T. Q. R. T. Q. , Rubanova , Yulia Y. , Bettencourt , Jesse J. Duvenaud , David K. D. K. ( 2018 ). Neural Ordinary Differential Equations . In Advances in Neural Information Processing Systems (NeurIPS) 31 . binproceedings

    work page 2018

  4. [4]

    , Zhang , Xiangyu X

    binproceedings [author] He , Kaiming K. , Zhang , Xiangyu X. , Ren , Shaoqing S. Sun , Jian J. ( 2016 ). Deep Residual Learning for Image Recognition . In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 770--778 . binproceedings

    work page 2016

  5. [5]

    , Sun , Yu Y

    binproceedings [author] Huang , Gao G. , Sun , Yu Y. , Liu , Zhuang Z. , Sedra , Daniel D. Weinberger , Kilian Q. K. Q. ( 2016 ). Deep Networks with Stochastic Depth . In European Conference on Computer Vision (ECCV) 646--661 . binproceedings

    work page 2016

  6. [6]

    , Gabriel , Franck F

    binproceedings [author] Jacot , Arthur A. , Gabriel , Franck F. Hongler , Cl \'e ment C. ( 2018 ). Neural Tangent Kernel: Convergence and Generalization in Neural Networks . In Advances in Neural Information Processing Systems (NeurIPS) 31 . binproceedings

    work page 2018

  7. [7]

    , Lin , Ting T

    barticle [author] Li , Qianxiao Q. , Lin , Ting T. Shen , Zuowei Z. ( 2022 ). Deep learning via dynamical systems: An approximation perspective . Journal of the European Mathematical Society 25 1671--1709 . barticle

    work page 2022

  8. [8]

    , Xu , Zheng Z

    binproceedings [author] Li , Hao H. , Xu , Zheng Z. , Taylor , Gavin G. , Studer , Christoph C. Goldstein , Tom T. ( 2018 ). Visualizing the Loss Landscape of Neural Nets . In Advances in Neural Information Processing Systems (NeurIPS) 31 . binproceedings

    work page 2018

Show all 15 references
  1. [9]

    Shiryaev , Albert A

    bbook [author] Peskir , Goran G. Shiryaev , Albert A. ( 2006 ). Optimal Stopping and Free-Boundary Problems . Birkh \"a user , Basel . bbook

    2006

  2. [10]

    ( 2025 )

    binproceedings [author] Qi , Qian Q. ( 2025 ). Universal Approximation Theorem of Deep Q-Networks . In International Conference on Machine Learning . PMLR . binproceedings

    2025

  3. [11]

    , Debut , Lysandre L

    barticle [author] Sanh , Victor V. , Debut , Lysandre L. , Chaumond , Julien J. Wolf , Thomas T. ( 2019 ). DistilBERT, a distilled version of BERT : smaller, faster, cheaper and lighter . arXiv preprint arXiv:1910.01108 . barticle

    2019 arXiv

  4. [12]

    Spiliopoulos , Konstantinos K

    barticle [author] Sirignano , Justin J. Spiliopoulos , Konstantinos K. ( 2022 ). Mean field analysis of deep neural networks . Mathematics of Operations Research 47 120--152 . barticle

    2022

  5. [13]

    , McDanel , Bradley B

    binproceedings [author] Teerapittayanon , Surat S. , McDanel , Bradley B. Kung , H. T. H. T. ( 2016 ). BranchyNet: A Trainable Deep Network for Scalable Inference . In Proceedings of the 33rd International Conference on Machine Learning (ICML) . Proceedings of Machine Learning...

    2016

  6. [14]

    , Shazeer , Noam N

    binproceedings [author] Vaswani , Ashish A. , Shazeer , Noam N. , Parmar , Niki N. , Uszkoreit , Jakob J. , Jones , Llion L. , Gomez , Aidan N. A. N. , Kaiser , ukasz . Polosukhin , Illia I. ( 2017 ). Attention Is All You Need . In Advances in Neural Information Processing Sys...

    2017

  7. [15]

    , Han , Jiequn J

    barticle [author] Weinan , E E. , Han , Jiequn J. Li , Qianxiao Q. ( 2019 ). A mean-field optimal control formulation of deep learning . Research in the Mathematical Sciences 6 1--41 . barticle

    2019

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.