{"id":"2c76f42a-68c9-45ba-ae4a-8d213282ba2e","arxiv_id":"2506.14386","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Variable-depth, residual-like networks generalize better than fixed-depth networks of matched average nonlinear depth, even when optimization differences are minimized.","lead":"This paper argues that residual connections do more than make deep networks easier to train: they create a genuinely different space of functions. The authors show that variable-depth, ResNet-like sub-networks extracted from a trained network keep more accuracy than fixed-depth ones of the same average depth.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim that variable-depth networks beat fixed-depth networks beyond optimization rests on the untested §4.5 assumption that the channel-wise and layer-wise post-training procedures are optimization-equivalent; if layer-wise extraction is merely harder to optimize, the observed gap is…","rationale":"The reader's weakest assumption correctly identifies the optimization-equivalence claim in §4.5 as the hinge of the empirical argument. I agree: this is the point where the paper's central conclusion would break if the assumption fails. The theoretical result that residual and feedforward function spaces differ is secondary; it only establishes that a reparametrization is impossible under equal shape, not that variable-depth architectures generalize better. The experiments are careful in many respects—starting from a converged network, matching NAPL, and including parameter-count controls (B.3–B.5)—which is why I would not reject the paper. But those controls do not directly test optimization difficulty: they address parameter count and post-hoc flexibility, not whether the layer-wise procedure is systematically harder to optimize. The proposed check (longer training, stronger optimizer, restarts, plus training-loss reporting) would settle this. If the gap persists under a much stronger fixed-depth optimization effort, the empirical claim is substantially supported; if it closes, the paper's headline overstates the evidence. Since the reader already conditioned acceptance on this and related points, I recommend no change to the CONDITIONAL verdict.","tokens_in":20495,"tokens_out":7634,"duration_ms":76597,"concrete_test":"Re-run the layer-wise (fixed-depth) extraction with an order-of-magnitude larger optimization budget: e.g., 3× more post-training epochs, AdamW with cosine decay, and 5 independent restarts of the slope parameters, at the same NAPL values as Figure 1. Report both final training loss and validation accuracy for both methods. If layer-wise reaches channel-wise validation accuracy at equal or lower training loss, the gap is an optimization artifact; if layer-wise remains below channel-wise even when its training loss is no higher, the inductive-bias interpretation survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The empirical core of the paper is a post-training comparison: starting from a converged RepVGG feedforward network, the channel-wise and layer-wise linearization procedures are meant to differ only in the shape of the extracted network. But they also differ in optimization difficulty. Channel-wise linearization has one slope parameter per channel; layer-wise has one per layer. At a matched normalized average path length (NAPL), the layer-wise variant must fully linearize entire layers, which is a substantially more rigid constraint on the function and on the loss landscape. The paper's §4.5 acknowledges that there is no guarantee the extracted networks reach the best possible test accuracy for their target shape, and asserts that because the post-training phase is short and starts from convergence, optimization is unlikely to make a difference. No evidence is given for this assertion: no training-loss curves, no convergence diagnostics, no multiple restarts, no optimizer/length ablation. The controls in Appendix B.4 and B.5 are not decisive: B.4 releases the layer-wise constraint only after the layer-wise solution has been found, so it cannot reveal whether a better layer-wise basin exists, and B.5 adds parameters but keeps the same rigid per-layer constraint during optimization. Consequently, the central claim—that the gap reflects an inductive bias of variable-depth architectures rather than an optimization artifact—is not yet established. This is the load-bearing assumption: if a more thorough layer-wise optimization closes the gap, the headline conclusion is reduced to a statement about ease of optimization, not about the function class.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that residual connections confer performance advantages beyond optimization by placing networks in a different function space: variable-depth networks (containing both short and long paths) rather than fixed-depth feedforward networks. It first gives an analytical argument (Section 3 and Appendix A) that residual and feedforward blocks of the same width and depth are not reparametrizations of each other, since a residual block can represent the identity while a feedforward block with a non-injective nonlinearity cannot. The main empirical contribution is a post-training linearization experiment: starting from a pretrained RepVGG-A2 feedforward network, the authors regularize PReLU slopes toward 1 either channel-wise (allowing some channels to become linear and creating variable-depth subnetworks) or layer-wise (forcing whole layers to become linear and creating fixed-depth subnetworks). They compare test accuracy at matched normalized average path length (NAPL) on ImageNet, CIFAR-10, and CIFAR-100, and find that the channel-wise (variable-depth) networks outperform the layer-wise (fixed-depth) ones at low NAPL. This gap largely disappears when the starting architecture is a ResNet, which the authors interpret as evidence that the advantage is due to variable-depth structure rather than to other artifacts.","tokens_in":20811,"tokens_out":6977,"duration_ms":70473,"significance":"If the central claim holds, the paper would reframe the role of skip connections in deep learning: rather than merely improving trainability, residual connections impose an inductive bias aligned with natural data. The experimental protocol is creative and includes several thoughtful controls, such as parameter-count comparisons (Appendices B.3-B.5), alternative nonlinearity measures (Appendix B.2), and a ResNet starting-point control (Section 4.3). The analytical argument in Section 3 is simple but correct for the stated restricted setting. The paper also ships code and training details in the appendix, which supports reproducibility. The main risk is that the interpretation rests on an untested assumption about the equivalence of optimization difficulty between the two extraction procedures; if that assumption fails, the observed gap could be an optimization artifact rather than evidence for an inductive bias.","major_comments":[{"comment":"The central claim that the observed gap reflects an inductive bias rather than an optimization artifact depends on the assumption, stated in Section 4.5, that the short post-training phases for channel-wise and layer-wise linearization are unlikely to differ in their final generalization because both start from a converged network. This assumption is load-bearing and currently unsupported. The layer-wise procedure imposes a much more rigid constraint (one slope per layer vs. one per channel), so at matched NAPL it may simply be harder to optimize; the loss landscapes of the two procedures are not the same. The controls in Appendix B.4 and B.5 do not resolve this: B.4 releases the layer-wise constraint only after the layer-wise solution has been found, so it cannot reveal whether a better layer-wise basin exists, and B.5 adds parameters while keeping the same rigid per-layer constraint during optimization. Please provide convergence diagnostics (training loss and validation curves for both procedures), multiple restarts, and ideally a within-optimization control in which the layer-wise constraint is relaxed during training (e.g., a run initialized from the channel-wise solution but trained under the layer-wise objective, or vice versa). Without such evidence, the interpretation as an inductive bias is not uniquely supported.","section":"Section 4.5 and Appendices B.4-B.5"},{"comment":"The ImageNet results, which are the headline comparison, are reported from a single run without error bars. Given that the paper makes a strong claim about a consistent performance gap, the ImageNet panel should be supported by at least a few independent seeds or by a clear statement of run-to-run variability, for example by reporting the spread observed in the CIFAR-100 repeated runs. As it stands, the ImageNet curve could be consistent with run-to-run noise at some NAPL values, particularly where the gap is small.","section":"Figure 1 (left)"},{"comment":"The analytical argument establishes that the identity function is representable by a residual block but not by a feedforward block with the same width, depth, and a non-injective elementwise nonlinearity. This is a valid existence proof, but it does not by itself predict the magnitude or even the sign of a generalization gap on natural data. The paper already phrases this as making the gap 'plausible,' which is appropriate, but the discussion in Sections 5 and 6 should more clearly separate this possibility argument from the empirical evidence, so that readers do not mistake the theory for a direct explanation of the measured accuracy differences.","section":"Section 3 and Appendix A.1"}],"minor_comments":[{"comment":"The proof says that applying Lemma 1 to the first layer 'follows immediately,' but it should explicitly state that composing a non-injective first layer with arbitrary subsequent layers preserves non-injectivity; this is a trivial but necessary step.","section":"Appendix A.1, Proposition 2 proof"},{"comment":"The phrase 'starting for networks with NAPL under 12' should be 'starting from networks with NAPL under 12'.","section":"Section 4.2"},{"comment":"There are typos: 'vey similar' in Section 4 and 'stating from a ResNet56' in the Figure 2 caption should be 'very similar' and 'starting from a ResNet56', respectively.","section":"Section 4 and Figure 2 caption"},{"comment":"Several references are duplicated with identical titles and URLs: Saxe et al. 2014a/2014b, Yang et al. 2019a/2019b, and Mingard et al. 2021a/2021b. These should be disambiguated or merged.","section":"References"},{"comment":"The text refers to 'Figure 4.4' when discussing the path-length histograms, but the figure is numbered as Figure 3 in the paper; the cross-reference is incorrect.","section":"Section 4.4"},{"comment":"The checklist states that full code and launching scripts are provided, but no URL or repository link appears in the manuscript; for a journal submission, please include a link or a statement of when the code will be made publicly available.","section":"NeurIPS Checklist, item 5"}],"recommendation":"major_revision","confidential_remarks":"The paper is a promising and well-structured contribution, but the central empirical claim rests on an untested optimization-equivalence assumption that could be addressed with additional experiments. If the authors can provide convergence diagnostics, multiple restarts, and a within-optimization control, the paper would be much stronger. The ImageNet single-run issue is also worth addressing, though the CIFAR-100 repeated runs provide some support. The theoretical argument is correct but narrow; the authors should be careful not to overstate its reach. The reference list and checklist contain several typos and duplicates, suggesting the manuscript needs a careful proofreading pass before resubmission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is a serious empirical paper with a genuinely nice experimental idea, but its headline conclusion—that ResNets win due to an inductive bias beyond optimization—is only as strong as an untested equivalence assumption about the extraction procedures. I'd send it to review, but the authors should either close that gap or soften the claim.\n\nWhat's new: instead of training ResNets and feedforward nets from scratch (where any gap can be blamed on trainability), they start from a trained RepVGG feedforward net and use two partial-linearization schemes to extract a variable-depth or a fixed-depth network at matched normalized average path length. Channel-wise PReLU regularization yields variable-depth sub-networks; layer-wise regularization yields fixed-depth ones. The observed performance gap favoring variable-depth shapes is replicated on CIFAR-10/100 and ImageNet, with controls for parameter count (B.5) and post-hoc freeing of layer-wise constraints (B.4). The CIFAR-100 runs have error bars; ImageNet is single-run. The paper is honest about limitations and provides code and hyperparameters. That's a useful experimental template.\n\nThe soft spot is exactly what Section 4.5 acknowledges: the two procedures differ not only in the shape they impose but in optimization difficulty. Layer-wise linearization forces entire layers to become linear at matched NAPL, a much more rigid constraint. The authors assert that a short post-training phase starting from convergence makes optimization unlikely to matter, but they give no training curves, no restarts, no comparison of achieved training loss. The appendix controls don't settle it: B.4 releases the layer-wise constraint only after the layer-wise solution is found, and B.5 adds parameters without changing the rigid per-layer constraint during training. So the central interpretation—inductive bias rather than optimization artifact—is not yet established. This is not fatal; the paper frames its results as \"another indication.\" But the abstract and conclusion state it more strongly than the evidence supports.\n\nOne point where I part ways with the reader: the proof of Proposition 2 is fine. The argument that feedforward networks with non-injective nonlinearities are non-injective, and therefore cannot represent the identity in a residual network, is valid. Novelty is moderate given Veit et al.'s ensemble view and Huang et al.'s NTK analysis, but the post-training comparison is new.\n\nWho should read this: people working on architecture analysis, inductive bias, and post-hoc model simplification. It deserves a serious referee—major revision—with the main request being direct evidence about the optimization landscape of the two extraction procedures.","headline":"A clever post-training comparison suggests variable-depth nets have a real advantage, but the headline claim of inductive bias beyond optimization rests on an untested assumption about the two extraction procedures; worth a serious review with requests for more evidence.","tokens_in":21315,"tokens_out":3793,"would_cite":true,"duration_ms":38116,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Residual connections put networks in a different function space, and the paper argues this is why they keep beating feedforward nets even when training is no longer the bottleneck.","keywords":["residual connections","trainability","inductive bias","function space","variable-depth networks","partial linearization","normalized average path length","generalization gap"],"falsifier":"Take the same pretrained feedforward model, run the channel-wise and layer-wise linearization many times with different random seeds and a much longer post-training phase, or with a solver that provably finds the best linear subset of units, and compare test accuracy at matched NAPL; if the layer-wise extraction ever catches up with the channel-wise one, the variable-depth advantage is an optimization artifact rather than an inductive bias.","tokens_in":20310,"feed_emoji":"🧠","tokens_out":7524,"duration_ms":68927,"temperature":0.7,"pith_summary":"The paper sets out to show that residual connections are not just a training aid. Its central claim is that a residual network lives in a different function space from a feedforward network of the same width and depth, and that this difference carries an inductive bias that matches natural data. On the evidence side, the authors take a pretrained feedforward network, then in a short post-training phase linearize some of its nonlinear units either per channel (producing variable-depth networks with a mixture of long and short paths, like a ResNet) or per layer (producing fixed-depth networks). They find that the variable-depth extraction beats the fixed-depth extraction at matched average nonlinear depth, even though both start from the same converged model and differ only in the shape constraint. The upshot is that the long-standing performance gap between residual and feedforward networks may never be fully closed by better initialization or schedules.","feed_headline":"Variable-depth nets beat fixed-depth nets once training is equalized","feed_subtitle":"A post-training experiment isolates generalization from optimization and finds residual-style paths still win.","key_machinery":"The load-bearing object is the partial-linearization procedure used to compare function spaces. Starting from a pretrained feedforward network, ReLU units are replaced by PReLU units with slopes $\\alpha_i$, and a sparsity penalty $\\sum_i |1-\\alpha_i|^{0.5}$ pushes some slopes toward 1. Linearizing per channel yields variable-depth networks (each channel may become linear independently, so paths of different lengths emerge), while linearizing per layer yields fixed-depth networks; normalized average path length (NAPL) is used to match the resulting nonlinear depth. The analytical counterpart is the impossibility result for equivalent re-parametrization, which establishes that the two shapes genuinely span different function classes. The empirical machinery controls for trainability by extracting both shapes from the same converged network, so the only difference is the shape constraint itself.","core_discovery":"The discovery is that, once trainability is factored out, variable-depth networks still generalize better than fixed-depth networks. The paper proves a small negative result first: with a non-injective nonlinearity and square weight matrices, a residual block $R(x)=\\phi(Wx+b)+x$ cannot be re-parametrized as an equivalent feedforward block $F(x)=\\phi(Wx+b)$ of the same width and depth, because the feedforward function class cannot represent the identity map. It then shows experimentally that when a pretrained fixed-depth network is molded post-training into either a variable-depth or fixed-depth shape, the variable-depth shape wins on ImageNet, CIFAR-10, and CIFAR-100 for low normalized average path length, and the gap widens as networks get shallower. The authors argue this is an emergent property of optimization rather than an artifact: the extracted variable-depth networks display path-length histograms similar to those of standard ResNets, and the advantage mostly disappears when the same procedure starts from a pretrained residual network.","pith_inferences":["Beyond the paper: one could test the inductive-bias story directly by training from scratch architectures that are deliberately shaped as variable-depth networks without residual skip connections, for example by pruning channels to linearity during training; the paper's account predicts they would retain a generalization edge over fixed-depth networks of the same NAPL.","Beyond the paper: the channel-wise extraction result suggests model-compression techniques that convert nonlinear units to linear ones per channel may preserve accuracy better than layer-wise folding, because they retain variable-depth structure; that is a testable engineering consequence.","Beyond the paper: if the inductive bias is truly aligned with natural data, synthetic data with controlled scale structure, such as mixtures of low- and high-frequency components, should show a tunable version of the gap, with the break-even NAPL shifting with the depth of the data's own hierarchy."],"forward_implications":["If the claim is right, no amount of initialization tuning, learning-rate scheduling, or gradient shaping should fully close the gap between deep feedforward networks and ResNets, because the gap is not only an optimization effect.","Architectures that contain a mixture of long and short paths, with residual connections being one way to get them, should be preferred for natural data with multi-scale structure, and the advantage should grow as the target depth shrinks.","The advantage should transfer to new datasets roughly in proportion to dataset difficulty: the paper observes a lower break-even NAPL on easier datasets, so harder datasets should show a wider regime where variable-depth beats fixed-depth.","Post-training linearization gives a practical way to probe a network's effective depth and to compare architectures while holding optimization roughly constant."],"supporting_citations":[{"why":"Introduces residual connections and documents the performance gap that the paper seeks to explain.","marker":"[He et al., 2016]"},{"why":"Supplies the ensemble-of-paths view that residual networks are mixtures of long and short paths, used to interpret the extracted path-length histograms.","marker":"[Veit et al., 2016]"},{"why":"Shows that numerically healthy deep feedforward networks still trail residual networks, motivating the beyond-trainability hypothesis.","marker":"[Martens et al., 2021]"},{"why":"Further narrows but does not eliminate the gap with tailored rectifiers, supporting the claim that trainability fixes are not enough.","marker":"[Zhang et al., 2022]"},{"why":"Provides the channel-wise partial linearization method that the experiment builds on.","marker":"[Ali Mehmeti-Göpel and Disselhoff, 2023]"},{"why":"Provides the layer-wise linearization counterpart used to produce fixed-depth networks.","marker":"[Dror et al., 2021]"},{"why":"RepVGG is the pretrained feedforward base architecture for the main comparison.","marker":"[Ding et al., 2021]"},{"why":"Shows that warm-up and gradient-norm normalization fix exploding and shattering gradients yet a gap remains, supporting the beyond-trainability interpretation.","marker":"[Ali Mehmeti-Göpel and Wand, 2024]"}],"fun_headline_variants":["ResNets' real edge: variable depth, not training ease","Variable-depth nets beat fixed ones even after training equalized","Why ResNets win: a structural advantage, not optimization","Equal training, yet variable-depth nets generalize better","Residual nets' depth flexibility outlasts feedforward training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the short post-training phase used to linearize networks does not systematically favor one extraction method: after the network has converged, the layer-wise and channel-wise procedures are equally likely to reach the best possible network of their respective shape, so the measured gap is a property of the shape rather than of optimization.","fun_headline_variants_meta":{"raw":{"variants":["ResNets' real edge: variable depth, not training ease","Variable-depth nets beat fixed ones even after training equalized","Why ResNets win: a structural advantage, not optimization","Equal training, yet variable-depth nets generalize better","Residual nets' depth flexibility outlasts feedforward training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000718,"raw_usage":{"total_tokens":3201,"prompt_tokens":900,"completion_tokens":2301,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":516,"completion_tokens_details":{"reasoning_tokens":2218}},"tokens_in":516,"tokens_out":2301,"duration_ms":16705,"temperature":1.0,"reasoning_tokens":2218,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:52:21.875908+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same pretrained feedforward model, run the channel-wise and layer-wise linearization many times with different random seeds and a much longer post-training phase, or with a solver that provably finds the best linear subset of units, and compare test accuracy at matched NAPL; if the layer-wise extraction ever catches up with the channel-wise one, the variable-depth advantage is an optimization artifact rather than an inductive bias.","supporting_citations":[{"cited_title":"Wilber, and Serge J","cited_arxiv_id":null,"evidence_quote":"Supplies the ensemble-of-paths view that residual networks are mixtures of long and short paths, used to interpret the extracted path-length histograms."},{"cited_title":"Deep learning without shortcuts: Shaping the kernel with tailored rectifiers","cited_arxiv_id":null,"evidence_quote":"Further narrows but does not eliminate the gap with tailored rectifiers, supporting the claim that trainability fixes are not enough."}],"review_version":1}