{"id":"a661ff29-e358-4b53-b842-644d132d27b3","arxiv_id":"2506.16862","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"The paper casts ResNet depth as an optimal stopping problem, proves finiteness of optimal depth under bounded utility, and proposes a residual-norm regularizer for early exiting.","lead":"This paper models a neural network's depth selection as an optimal stopping problem, balancing prediction accuracy against compute cost. It proposes a regularizer that pushes residual networks to learn representations that can exit early, with tests on ImageNet.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1's ASTI (c=0.001) row is internally inconsistent: a 0.9-point accuracy gain over full-depth inference with average FLOPs identical to full depth cannot arise from early exits, and the note admits the data are simulated.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: Table 1 is internally inconsistent and appears to come from a simulation rather than genuine training. My analysis sharpens this into a quantitative impossibility argument using the paper's own FLOPs numbers. The ASTI (c=0.001) row cannot be a real operating point of the same model: the average FLOPs equal the full network to two decimals, bounding the early-exit fraction so tightly that the reported accuracy gain of 0.9 points is unattainable. Separately, the theoretical contribution is weaker than claimed—Theorem 4.4's proof uses only boundedness of g and the positive cost c, not the diminishing-returns condition (Proposition 4.3) that the abstract credits—but the theorem itself is true, so this is a presentation issue rather than a mathematical failure. The empirical inconsistency, combined with the explicit 'final corrected simulation' note, is sufficient to reject the paper's central efficiency claim. The reader's verdict of REJECT is therefore appropriate, and no change to that verdict is needed.","tokens_in":18916,"tokens_out":4519,"duration_ms":46179,"concrete_test":"Recompute the Table 1 rows from raw per-sample stopping depths and predictions. For the ASTI (c=0.001) row, check whether any early-exit fraction can reconcile the reported average 4.11 G-FLOPs with the 73.1% Top-1 accuracy given the full-depth accuracy of 72.2%. A simple bound using the paper's own Static-Exit FLOPs shows that any exit at layer 40 or earlier saves at least 0.81 G-FLOPs, so an average saving below 0.005 G-FLOPs caps the early-exit fraction below 0.62%, making the reported +0.9-point gain impossible. Releasing the per-sample stopping layer distribution and per-layer head accuracies would settle whether the table is genuine.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central practical claim—that ASTI achieves a superior accuracy-efficiency trade-off—rests entirely on Table 1. The table note states 'Data from final corrected simulation,' which is a red flag that the numbers are not measured from training runs. More concretely, the ASTI (c=0.001) row reports an average of 4.11 G-FLOPs, identical to the full network, yet a Top-1 accuracy of 73.1%, which is 0.9 percentage points higher than the full-depth ASTI model's 72.2%. Since c=0.001 is a very small stopping threshold, early exits are rare; the reported FLOPs imply the early-exit fraction is below 0.62% (using the paper's own Static-Exit data: exiting at layer 40 saves 0.81 G-FLOPs, so a saving below 0.005 G-FLOPs permits at most 0.62% early exits). The maximum possible accuracy gain from such a small fraction is below 0.62 percentage points, so the reported 0.9-point gain is mathematically impossible unless the early-exit heads dramatically outperform the final head on a tiny subset, which would still be bounded by the fraction. Thus the empirical validation is not merely weak; it is internally inconsistent. Additionally, the 'smoking gun' Figure 1 plots E[||f_l(h_l)||^2], the exact quantity penalized by L_depth, so it is not independent evidence of the theory. The theoretical theorem is true but standard, and the diminishing-returns condition is not used in its proof; nevertheless, the load-bearing problem for the paper's acceptance is the unsupported empirical centerpiece.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper frames the forward pass of a ResNet as an infinite-horizon optimal stopping problem: at each layer the network either stops and predicts from the current representation or continues at cost c, with reward g(h_l) - c*l. It proves a finite expected stopping depth under a proposed diminishing-returns condition on the residual functions, derives an L_depth regularizer intended to induce that condition, and proposes the ASTI inference algorithm based on a one-step lookahead approximation. The paper also sketches extensions to Transformers and to the continuous-depth free-boundary setting, and reports ImageNet/ResNet-50 experiments as validation. The central theoretical claim is that diminishing returns imply finite optimal expected depth; the central practical claim is that ASTI achieves a superior accuracy-efficiency trade-off.","tokens_in":19329,"tokens_out":7872,"duration_ms":87237,"significance":"If the advertised results held, framing neural depth as optimal stopping would be a useful conceptual contribution and the regularizer might have practical value. The paper is clearly structured, the optimal-stopping setup is standard and mostly well stated, and the HJB connection in Section 7 is reasonable as a research direction. However, the main theorem does not actually use the diminishing-returns condition, the ASTI stopping rule is derived from an unvalidated approximation, and the empirical validation is circular with respect to the regularizer and internally inconsistent in Table 1. These are load-bearing issues: the abstract's primary theoretical contribution and the paper's main efficiency claims are not supported as written.","major_comments":[{"comment":"The proof of Theorem 4.4 does not invoke Proposition 4.3 or the diminishing-returns hypothesis at any point. The derivation only uses boundedness of g (Assumption A1) and positivity of c: after establishing V = E[g(h_{tau*})] - c E[tau*], Step 4 bounds E[g(h_{tau*})] by g_max and rearranges to obtain E[tau*] <= (g_max - E[g(h_L0)])/c + L0. This argument would go through identically even if the residual norms E[||f_l(h_l)||] were large at every layer. Consequently the abstract's claim that finite expected optimal depth is proved 'under a plausible condition of diminishing returns' is not accurate; the advertised primary theoretical contribution is not a theorem about diminishing returns and should be restated or strengthened.","section":"Section 4.2, Theorem 4.4; Appendix A.3"},{"comment":"The practical stopping rule is obtained by replacing the continuation value E[U_{l+1}|h_l] with E[Y_{l+1}|h_l] and then, because the transition is deterministic for a fixed input, with the realized value Y_{l+1}. This conflates a conditional expectation over the data distribution with the actual next-state reward on a single input, and no bound or experiment shows that the replacement is a faithful approximation of the Snell envelope. Since ASTI's status as a 'theoretically grounded' algorithm rests on this step, the authors should either provide a quantitative approximation guarantee or present the rule as a heuristic and validate it against a computed or estimated optimal policy.","section":"Section 9.1, Eq. (17)"},{"comment":"The ASTI (c = 0.001) row reports an average of 4.11 G-FLOPs, identical to the full-depth ASTI row, while its Top-1 accuracy is 73.1%, 0.9 points higher than the full-depth ASTI model's 72.2%. Because both rows use the same trained weights and c = 0.001 is a very small threshold, essentially full-depth inference should reproduce the full-depth outputs and accuracy; if a nontrivial fraction of inputs exits early, the average FLOPs should decrease noticeably. The table note 'Data from final corrected simulation' also leaves it unclear whether any of these numbers come from actual training and inference runs. This table is the main support for the paper's accuracy-efficiency claim, so it needs to be replaced with real measured results, including exit rates, standard deviations, and training details.","section":"Table 1 and Section 10.2.2"},{"comment":"Figure 1 plots E[||f_l(h_l)||^2], which is exactly the quantity penalized by L_depth in Eq. (11). Observing that this quantity decays after training with L_depth is therefore a check that the optimizer minimized the regularizer, not independent evidence of the theoretically predicted diminishing-returns behavior. The 'smoking gun' language is not supported unless the comparison also reports quantities not directly in the training objective, such as the induced expected utility increments or the behavior of a model trained with intermediate supervision but without L_depth.","section":"Section 10.2.1, Figure 1; Eq. (11)"}],"minor_comments":[{"comment":"The paper proposes a polynomial schedule w_l = (l+1)^alpha in Section 5.1 but reports using w_l = (l/L)^2 in Section 10.1.2; please state the exact schedule used for all reported results and clarify whether alpha was tuned.","section":"Section 5.1 and Section 10.1.2"},{"comment":"The training protocol omits important details such as the number of epochs, batch size, learning rate schedule, data augmentation, weight decay, and number of seeds; the reported accuracies also lack error bars, which is necessary to support the claimed Pareto-frontier superiority.","section":"Section 10.1.2"},{"comment":"The discussion of 'look-then-leap' stopping is confusing: the algorithm decides at layer l after computing h_{l+1} and then returns a prediction from h_{l+1}, so the formal stopping time is l+1, not a decision made before observing h_{l+1}. Clarify the timing and the filtration used for the decision.","section":"Section 9.1, Remark 9.1"},{"comment":"The claim that ASTI (c = 0.005) 'dramatically outperforms' the best entropy-based baseline (70.6% vs. 64.3%) should be accompanied by the threshold-setting details for the entropy method; otherwise the comparison is not reproducible.","section":"Section 10.2.2"}],"recommendation":"reject","confidential_remarks":"The most serious issue for the editor is provenance of the empirical numbers in Table 1. The note 'Data from final corrected simulation,' the lack of any training details, and the internally inconsistent ASTI (c = 0.001) row together suggest that the empirical validation may not come from actual experimental runs. I would ask the authors to provide code, checkpoints, and training logs before any resubmission is considered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nTwo things to know about arXiv:2506.16862. First, the main theoretical claim is technically true but much weaker than advertised. Theorem 4.4 proves that in an infinite-horizon optimal stopping problem with bounded utility and positive per-layer cost, the optimal expected stopping time is finite. That is a standard fact, and the proof never uses the paper's \"diminishing returns\" condition (Proposition 4.3). So the \"primary theoretical contribution\" reduces to a textbook exercise. Second, the empirical centerpiece is not credible. The ASTI (c=0.001) row in Table 1 reports the same average FLOPs as the full network (4.11 G-FLOPs) yet a Top-1 accuracy of 73.1%, which is 0.9 points higher than the full-depth ASTI model's 72.2%. Given that the early-exit fraction at that FLOP count is at most a fraction of a percent (using the paper's own Static-Exit data), that accuracy gain is mathematically impossible unless the early-exit heads are perfect on a tiny subset and the final head is much worse on that subset, which would still be bounded by the exit fraction. The table note \"Data from final corrected simulation\" is a red flag, and no code or data are provided to check.\n\nWhat the paper does well: the setup is clean. Recasting the ResNet forward pass as a Markov decision process with reward g(h_l) - c·l is a nice expository move, and the dynamic-programming derivations are correct. The L_depth regularizer—penalizing squared residual norms with weights increasing in depth—is a reasonable heuristic for encouraging early-exit-friendly representations. The extensions to Transformers and the continuous limit are sketchy but suggest the framework generalizes.\n\nSoft spots beyond the theorem and Table 1. Figure 1, offered as \"smoking gun\" evidence, plots exactly the quantity L_depth penalizes, so it cannot independently validate the theory. The ASTI inference rule is a one-step lookahead approximation with no error analysis or comparison to optimal stopping thresholds. The baselines (Static-Exit, Entropy-Exit) are weak; no BranchyNet, MSDNet, or any recent adaptive-depth method is compared. No error bars anywhere.\n\nOverall: the theoretical scaffolding is sound but elementary, and the load-bearing empirical evidence does not hold up. The paper as it stands does not support its claims. Would I cite it? No. Would I bring it to reading group? Probably not. But I would not desk-reject it out of hand. A serious referee could check whether the ImageNet numbers are genuine and whether the regularizer has any independent benefit. If the numbers are simulated, that is a deal-breaker. If they are real, the paper needs significant revision: redo the experiments with proper baselines, report variation, and either fix the c=0.001 row or drop it.","headline":"A clean optimal-stopping framing undercut by an overclaimed theorem and an empirical table that cannot be right.","tokens_in":19765,"tokens_out":4565,"would_cite":false,"duration_ms":44660,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62L15","68T07","60G40"],"pacs":[],"model":"deepseek-v4-flash","headline":"Optimal depth of a residual network is the solution to an optimal stopping problem, and the expected optimal stopping depth is provably finite.","keywords":["Optimal Stopping","Deep Learning Theory","Residual Networks","Sequential Decision Processes","Regularization","Hamilton-Jacobi-Bellman Equation","Early Exiting"],"falsifier":"Reproduce the ResNet-50 ImageNet training of Section 10 with the stated hyperparameters ($\\beta=0.5$, $\\lambda=1.0$, $w_l=(l/L)^2$), then measure average stopping depth and Top-1 accuracy under the ASTI rule at $c=0.001$. The paper reports 4.11 G-FLOPs and 73.1% accuracy for this configuration; if the measured average stopping depth is not essentially the full depth, the row cannot be reproduced as reported, and if it is full depth, the configuration provides no computational saving over the full model.","tokens_in":18692,"feed_emoji":"⏱️","tokens_out":11722,"duration_ms":111592,"temperature":0.7,"pith_summary":"This paper argues that the depth of a residual network is not something to be fixed by architecture search: it is an optimal stopping problem. At each layer the forward pass can either halt and predict, collecting reward $g(h_l)-c\\cdot l$, or continue to the next residual block $h_{l+1}=h_l+f_l(h_l)$. The main theoretical claim is that, under a plausible diminishing-returns condition on the residual functions, the expected optimal stopping depth is finite even in an infinite-horizon network, giving a principled account of why 'deeper is better' fails. The paper converts this into a regularizer $L_{\\rm depth}=\\sum_l w_l\\,\\mathbb{E}\\|f_l(h_l)\\|_2^2$ and an adaptive inference rule, and reports on ImageNet that the regularized ResNet-50 dominates static and entropy-based early-exit baselines on accuracy versus compute. The same framework is extended to Transformers and to a continuous-depth free-boundary formulation.","feed_headline":"Optimal network depth is finite—and this paper shows when","feed_subtitle":"A proof that expected optimal stopping depth is finite, plus a regularizer that makes ResNets stop early on ImageNet.","key_machinery":"The machinery is the optimal-stopping reformulation of the ResNet update. The state is the hidden representation $h_l$, the dynamics are $h_{l+1}=h_l+f_l(h_l)$, and the reward for stopping is $Y_l=g(h_l)-c\\cdot l$; the Snell envelope computes the value of continuing at each layer. The load-bearing objects are the residual functions $f_l$ themselves: bounding their expected norm (Proposition 4.3) is how the paper argues utility gains become sublinear in depth. The practical fork is the regularizer $L_{\\rm depth}$ and the one-step lookahead stop rule, which replaces the intractable Snell envelope with the condition $g(h_{l+1})-g(h_l)\\le c$.","core_discovery":"The paper's central claim is that the forward pass of a trained ResNet is a Markov process whose optimal input-dependent depth is the solution of an optimal stopping problem with reward $Y_l=g(h_l)-c\\cdot l$. The value function is the Snell envelope, and the optimal policy is to stop at the first layer where the immediate reward meets or exceeds the expected value of continuing. The paper proves in Theorem 4.4 that, in the infinite-horizon limit, the expected optimal stopping time $\\mathbb{E}[\\tau^*]$ is finite once residual outputs are small enough on average that expected utility gain falls below the per-layer cost; the bound is obtained by sandwiching the value between the reward at a fixed layer $L_0$ and $g_{\\max}-c\\mathbb{E}[\\tau^*]$. It then turns this normative model into a training objective by penalizing squared residual norms with layer-dependent weights, and into an inference algorithm by approximating the Snell-envelope rule with a one-step lookahead: stop when $g(h_{l+1})-g(h_l)\\le c$ and predict from $h_{l+1}$.","pith_inferences":["The proof of Theorem 4.4 as written never invokes Proposition 4.3: the finite bound on $\\mathbb{E}[\\tau^*]$ follows already from boundedness of $g$ and positivity of $c$, so the diminishing-returns condition named in the abstract is not actually the load-bearing part of the theorem.","A clean way to separate the regularizer from the stopping rule would be to run ASTI on a vanilla pretrained ResNet: since vanilla residuals do not decay with depth, the one-step lookahead rule should rarely stop early, which would show how much of the reported efficiency comes from $L_{\\rm depth}$ rather than from the rule itself.","The paper reports that the ASTI-trained full network is more accurate than the vanilla baseline (72.2% vs 69.8%); an ablation removing intermediate supervision and $L_{\\rm depth}$ separately would determine whether this accuracy gain is due to depth regularization or to the auxiliary heads.","The practical rule uses a one-step lookahead approximation of the Snell envelope; comparing ASTI's decisions with exact backward induction on a small dataset would quantify the approximation error the paper leaves implicit."],"forward_implications":["If the finiteness theorem is right, a trained residual network has a finite optimal inference depth per input, so always running to the last layer is provably suboptimal once the per-layer cost is positive.","Training with $L_{\\rm depth}$ and an increasing weight schedule should drive deep residual blocks toward the identity map, producing the measured decay in average residual norms that Figure 1 reports.","At inference the single parameter $c$ slides the model along the accuracy-computation frontier, so one model can serve low-cost and high-accuracy regimes without retraining.","The same reward structure and regularizer apply to Transformers because a Transformer block is also a residual update $h_{l+1}=h_l+\\mathrm{Block}_l(h_l)$.","In the continuous-depth limit the stopping set and continuation set are separated by a free boundary that solves an HJB variational inequality, connecting early-exit depth to free-boundary theory."],"supporting_citations":[{"why":"Defines the residual network update $h_{l+1}=h_l+f_l(h_l)$ that serves as the state dynamics of the optimal stopping model.","marker":"He et al. (2016)"},{"why":"Supplies the Snell-envelope and optimal stopping theory behind Proposition 4.2 and Theorem 4.4.","marker":"Peskir and Shiryaev (2006)"},{"why":"Provides the neural-ODE continuous-depth limit that motivates the free-boundary and HJB analysis.","marker":"Chen et al. (2018)"},{"why":"Defines the Transformer block whose residual structure is used to extend the framework in Section 8.","marker":"Vaswani et al. (2017)"},{"why":"Introduces BranchyNet, the early-exiting baseline ASTI is compared against.","marker":"Teerapittayanon, McDanel and Kung (2016)"},{"why":"Introduces Stochastic Depth, the related dynamic-depth method positioned as a baseline.","marker":"Huang et al. (2016)"}],"fun_headline_variants":["Optimal depth is finite: proof plus a practical regularizer","When to stop? Theory says finite, ResNets can early exit","Proven: ResNets have a finite optimal depth, and a way to use it","Early exit from deep nets: optimal stopping finds the depth"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's practical claims rest on the ImageNet measurements in Table 1 being genuine experimental results; if the 'final corrected simulation' entries, particularly ASTI (c=0.001) with full-network compute and higher accuracy than the full-depth model, are not real trained-model outcomes, the evidence for the regularizer's efficiency gains collapses.","fun_headline_variants_meta":{"raw":{"variants":["Optimal depth is finite: proof plus a practical regularizer","When to stop? Theory says finite, ResNets can early exit","Proven: ResNets have a finite optimal depth, and a way to use it","Early exit from deep nets: optimal stopping finds the depth"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000223,"raw_usage":{"total_tokens":1485,"prompt_tokens":1002,"completion_tokens":483,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":618,"completion_tokens_details":{"reasoning_tokens":405}},"tokens_in":618,"tokens_out":483,"duration_ms":5601,"temperature":1.0,"reasoning_tokens":405,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:17:16.010184+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Reproduce the ResNet-50 ImageNet training of Section 10 with the stated hyperparameters ($\\beta=0.5$, $\\lambda=1.0$, $w_l=(l/L)^2$), then measure average stopping depth and Top-1 accuracy under the ASTI rule at $c=0.001$. The paper reports 4.11 G-FLOPs and 73.1% accuracy for this configuration; if the measured average stopping depth is not essentially the full depth, the row cannot be reproduced as reported, and if it is full depth, the configuration provides no computational saving over the full model.","supporting_citations":[{"cited_title":", Sun , Yu Y","cited_arxiv_id":null,"evidence_quote":"Introduces Stochastic Depth, the related dynamic-depth method positioned as a baseline."},{"cited_title":"Shiryaev , Albert A","cited_arxiv_id":null,"evidence_quote":"Supplies the Snell-envelope and optimal stopping theory behind Proposition 4.2 and Theorem 4.4."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the neural-ODE continuous-depth limit that motivates the free-boundary and HJB analysis."},{"cited_title":", Shazeer , Noam N","cited_arxiv_id":null,"evidence_quote":"Defines the Transformer block whose residual structure is used to extend the framework in Section 8."},{"cited_title":", McDanel , Bradley B","cited_arxiv_id":null,"evidence_quote":"Introduces BranchyNet, the early-exiting baseline ASTI is compared against."}],"review_version":2}