{"id":"a532c944-7f7e-4833-b9dd-fbb1f769f946","arxiv_id":"2412.05144","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"During training, the effective feature rank of a network's last hidden layer grows in a staircase pattern alongside loss drops, and initializing for high rank accelerates training.","lead":"The paper introduces epsilon-rank, a count of how many independent features a neural network's final hidden layer is using, and reports that this count rises in staircase steps while training loss falls. It then shows that initializing networks to start with high epsilon-rank can accelerate training and improve accuracy.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The necessity claim is not established: Theorem 3.2 is a static lower bound that becomes vacuous for targets well approximated by a fixed low-rank feature set, a regime the paper neither excludes nor tests.","rationale":"The reader's weakest-assumption analysis identifies the same load-bearing gap, and I agree with it. Theorem 3.2 is not incorrect as a static inequality, but the inference to a dynamic necessity claim is invalid without a positive lower bound on dist(u*,F_p), and the paper's u*∉F_n assumption does not supply one. A target can lie outside the network function class and still be extremely well approximated by a fixed low-dimensional feature subspace, making the lower bound vacuous for small p. The proposed test targets exactly that gap: if a low-rank target can be fit to low loss while ε-rank remains O(2), the central claim collapses to the weaker statement that loss lower bounds decrease with rank, not that rank growth is essential for loss reduction. The empirical staircase observations and the UDI speed-ups may still be valid findings; the theoretical necessity overreach is what needs to be removed or qualified. No change to the reader's CONDITIONAL verdict is needed.","tokens_in":17497,"tokens_out":17335,"duration_ms":187072,"concrete_test":"Run the 1D setting of Example 2.1 (L=2, n=50, tanh, ε=10^-6) with target u*(x)=tanh(3x)+0.5tanh(7x)+δ cos(20x), δ=10^-3, so u*∉F_50 but dist(u*,F_2)=O(δ). Initialize neurons 1–2 to tanh(3x) and tanh(7x), initialize neurons 3–50 to the zero function with output weights 0, and run standard SGD on all parameters, recording ε-rank and loss. If loss reaches O(δ^2)≈10^-6 while ε-rank stays at 2, the necessity claim is falsified; if rank must rise before loss falls, the claim survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central necessity claim ('a sufficiently high ε-rank is essential for significant loss reduction') rests on Theorem 3.2 and the Section 3.1 assumption that u*∉F_n. That assumption is too weak. Theorem 3.2 states √L(un) ≥ (dist(u*,F_p) − O(√ε))/C_s for any network of ε-rank p. This is a lower bound at a fixed p; it forces p to increase only if the loss actually drops below the bound computed at the initial p. If dist(u*,F_p) is already tiny for p≪n, the bound is near zero from the outset and no rank growth is required. The paper never verifies dist(u*,F_p)>0 for small p, and for the MNIST/cross-entropy example the theorem does not even apply because the loss is not an L2 residual of a well-posed operator G(u)=f. A concrete low-rank target (e.g. a sum of two tanh features plus a tiny perturbation) satisfies u*∉F_n yet can be fit to O(δ^2) loss with ε-rank O(2), directly contradicting the universal necessity statement.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the ε-rank of the Gram matrix of last-hidden-layer neuron functions, reports a 'staircase phenomenon' in which loss plateaus are accompanied by stepwise increases in this rank across function fitting, PINN, and MNIST experiments, and proves in Theorem 3.2 a lower bound on L2 residual loss in terms of dist(u*, F_p). It interprets this bound as showing that a sufficiently high ε-rank is a necessary condition for significant loss reduction. It then proposes a uniform-distribution initialization (UDI) that gives O(n) ε-rank at initialization and reports faster convergence and improved accuracy in 1D/2D function fitting and PINN tests.","tokens_in":17704,"tokens_out":6162,"duration_ms":66043,"significance":"The ε-rank is a clean, computable diagnostic, and the empirical staircase pattern across several tasks is a useful observation that connects naturally to spectral bias and feature-learning dynamics. The UDI initialization is simple, cheap, and appears to help in the tested settings, with numerical settings documented in Appendix A. The theoretical result, however, is a static approximation bound rather than a training-dynamics necessity theorem; the paper's central claim overstates what Eq. (9) actually establishes. With a weaker, properly qualified statement and additional numerical checks of dist(u*, F_p), the contribution would be valuable to the deep-learning theory community.","major_comments":[{"comment":"The central claim that a sufficiently high ε-rank is essential for significant loss reduction is not a consequence of Theorem 3.2. For a fixed p, Eq. (9) is a static lower bound; the loss can decrease toward dist(u*,F_p)/C_s − O(√ε) while the ε-rank remains p. Necessity of rank growth follows only if the final loss is below the bound computed at the initial p and dist(u*,F_p)>0 for all smaller p. The manuscript never estimates dist(u*,F_p) for its problems, and the assumption u*∉F_n is too weak to imply it: a target fδ = tanh(a1·x)+tanh(a2·x)+δg with g∉F_n and δ small is not in F_n yet has dist(fδ,F_2)=O(δ), so Eq. (9) does not rule out reaching O(δ²) loss with ε-rank O(2). The abstract, Section 1, and Remark 3.2 should be revised to state this conditional form of the claim.","section":"§3.1, Eq. (9), Remark 3.2"},{"comment":"Theorem 3.2 is stated for losses L(u)=||G(u)−f||² with a stable solution operator G^{-1}, but Figure 5 (MNIST) uses cross-entropy loss, for which the theorem does not apply. Since Fig. 5 is presented as evidence of universality and Section 3 as its theoretical explanation, this is a gap. Either extend the theory to classification losses or explicitly present the MNIST observation as an empirical phenomenon outside the scope of the theorem.","section":"§2.2, Example 2.3 and §3"},{"comment":"The pre-training strategy is motivated by the layer-wise observation in Fig. 8, but that observation is reported for a single network (L=4, n=50) on one task. Since Section 4.2 states a general 'depth-dependent growth in ε-rank', supporting evidence across widths, depths, and tasks, or an explicit disclaimer that this is a single-run observation, is needed before the motivation for UDI is fully established.","section":"§4.2, Eq. (11), Fig. 8"}],"minor_comments":[{"comment":"The example title contains a typo: 'Possion's equation' should be 'Poisson's equation'.","section":"Example 4.3"},{"comment":"Several axis labels are malformed (for example, 'L oss' in Figures 2 and 3, and broken superscripts in Figure 10); please regenerate the figures with correct typography.","section":"Figures 2, 3, 10"},{"comment":"The term 'pre-training' is used for a one-time initialization scheme rather than a separate training phase; consider renaming it 'specialized initialization' to avoid confusion with standard pretraining.","section":"§4"},{"comment":"The ε-rank is computed in practice from a quadrature or Monte Carlo approximation of the Gram matrix, but the paper does not state an error bound connecting the discrete and continuous ranks; a short remark on how m and the quadrature weights affect the reported ranks would help.","section":"Definition 2.4 and Eq. (12)"},{"comment":"The assumption u*∉F_n is not verified for any of the numerical examples; the paper should at least acknowledge that this is an assumption rather than a checked condition.","section":"§3.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope, and the empirical findings are potentially useful. My main concern is that the theoretical language overpromises relative to what Theorem 3.2 proves; a careful revision that weakens the necessity claim, adds low-rank-target checks, and separates the classification experiments from the L2-residual theory is feasible within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper has a genuinely useful empirical observation and a simple initialization trick, but the theory is oversold. The staircase phenomenon — loss plateaus punctuated by jumps in the effective rank of the last-layer Gram matrix — is a real and clearly demonstrated pattern across tanh-MLP and PINN tasks. The UDI initialization (random directions and biases on the first layer) is simple, adds no parameters, and the experiments show it breaks the early plateau and speeds convergence. That part is worth taking seriously.\n\nThe soft spot is the necessity claim. Theorem 3.2 is a static lower bound: if the network has epsilon-rank p and the target is well approximated by some p-dimensional feature subspace, the bound is near zero and no rank growth is required to reach low loss. The paper's Remark 3.2 slides from 'loss lower bound depends on dist(u*,F_p)' to 'therefore rank must increase,' which only follows if you already know the loss is below the bound for the initial p. The assumption u*∉F_n does not fix this, because u* can be outside F_n while still being within O(ε) of F_p for small p. The MNIST cross-entropy experiment doesn't even fit the theorem's L2-residual framework. So the 'universal necessity' statement is not established. The authors should either drop it or state it as a conditional observation.\n\nOther issues are minor: epsilon is fixed at 1e-6 with no sensitivity check; there are no error bars or released code; the 'universal' claim rests on a handful of single-run curves. None of this suggests fabrication — just a paper that needs another pass.\n\nRecommendation: send it out. A good referee can push the authors to soften the theory, add repeated runs, and release code. If the staircase observation and UDI replicate, this is a useful paper for the scientific-computing/PINN community.","headline":"Useful empirical observation and a simple initialization trick, but the 'rank growth is necessary for loss reduction' claim is oversold; worth a serious referee.","tokens_in":18254,"tokens_out":1866,"would_cite":true,"duration_ms":18865,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper establishes that as a neural network trains with standard SGD, the effective rank of its last hidden layer's features rises in steps, and it proves a lower bound making high effective rank necessary for large loss reductions.","keywords":["epsilon-rank","staircase phenomenon","neural network training dynamics","Gram matrix","feature diversity","pre-training initialization","physics-informed neural networks","loss plateau"],"falsifier":"Train a wide network on a target that is exactly a linear combination of two fixed basis functions (for example, $u^*(x)=\\cos x+2\\sin x$). If the training loss can be driven to machine precision while the last hidden layer's Gram matrix has $\\epsilon$-rank near 2 throughout the run, the necessity claim is refuted. The same test can be run on a classification dataset whose labels depend linearly on a low-dimensional projection of the input.","tokens_in":17262,"feed_emoji":"📈","tokens_out":8935,"duration_ms":80722,"temperature":0.7,"pith_summary":"The paper introduces $\\epsilon$-rank, a computable measure of how many genuinely independent features a neural network's last hidden layer represents: it counts the eigenvalues of the Gram matrix of the neuron functions that exceed a small tolerance. Across function fitting, PDE solving with physics-informed networks, and MNIST classification, the authors observe a staircase phenomenon under standard stochastic gradient descent training: the loss falls in plateaus, and each rapid drop coincides with a stepwise increase in $\\epsilon$-rank. The paper proves a lower bound relating loss to $\\epsilon$-rank: if the last hidden layer has effective rank $p$, the loss cannot go below a constant multiple of the distance from the true solution to the class of networks with only $p$ effective features. A sufficiently high $\\epsilon$-rank is therefore necessary for a significant loss reduction, which explains why training stalls when feature diversity is low. Based on this, the paper proposes a first-layer pre-training strategy that starts the network with high $\\epsilon$-rank and shows it removes the initial plateau and speeds up training.","feed_headline":"Network feature rank climbs in stair-steps as loss drops","feed_subtitle":"A new metric ties training plateaus to missing feature diversity, and an initialization fix removes them.","key_machinery":"The load-bearing object is the $\\epsilon$-rank $r_\\epsilon(M_u)$, defined as the number of eigenvalues of the Gram matrix $M_u$ (with entries $\\int_\\Omega \\phi_i(x;\\theta)\\phi_j(x;\\theta)\\,dx$ for the last-hidden-layer neuron functions $\\phi_j$) that exceed a tolerance $\\epsilon$. It converts the question of how many useful features a layer has into a linear-algebra count. The argument then runs through two steps: a matrix perturbation lemma (Lemma 3.1, based on rank-revealing QR factorizations) showing that any $n$-function set with $\\epsilon$-rank $p$ has a $p$-function subset that approximates the original span to within a controlled error, and Theorem 3.2, which combines that approximation with a stability condition on the solution operator to lower-bound the loss of any low-$p$ approximator. The same machinery motivates the pre-training strategy: by constructing the first hidden layer with localized, well-separated neurons (via explicit centers in 1D and random directions on the unit sphere in higher dimensions), the $\\epsilon$-rank is forced to $O(n)$ from the start, bypassing the staircase.","core_discovery":"On the paper's own terms, the central discovery is that the evolution of a network's last-hidden-layer feature diversity is not smooth but stepwise, and that this stepwise behavior is forced by a quantitative relation between loss and effective rank. The relevant bound is $\\sqrt{L(u_n)} \\ge \\frac{1}{C_S}\\left(\\operatorname{dist}(u^*, F_p) - \\sqrt{C(p+1)(n-p)^2\\epsilon}\\right)$ for any approximator $u_n$ whose $\\epsilon$-rank is $p$; here $u^*$ is the exact solution, $F_p$ is the class of networks with $p$ neurons, $C_S$ is a stability constant, and $\\epsilon$ is the tolerance defining the rank. Since $\\epsilon$ is small, the bound says the best possible loss is controlled by how far the true solution is from the $p$-neuron network class. Reducing the loss therefore requires reducing that distance, which in turn requires increasing $p$. The paper interprets this as the theoretical explanation of the staircase: when the $\\epsilon$-rank is stuck at a low level, the loss is trapped at a plateau; when the rank jumps up, the loss drops sharply. It then shows experimentally that raising the initial $\\epsilon$-rank of the first hidden layer through a tailored initialization produces the same final accuracy in fewer iterations.","pith_inferences":["The paper stops short of prescribing rank monitoring as an adaptive training control; an immediate extension would be to trigger learning-rate changes or early stopping when $\\epsilon$-rank plateaus, which the staircase observation suggests.","The necessity result may fail to transfer to classification losses: for a dataset whose classes are separated by a low-dimensional manifold, cross-entropy loss can plausibly reach near zero while the last-layer $\\epsilon$-rank stays small. That is testable by measuring ranks on well-trained classifiers.","The UDI initialization samples directions uniformly on the sphere; a natural extension is to replace random directions with data-dependent or learned directions, or to apply the same rank-boosting idea to convolutional filters, which the paper does not explore.","The conjecture that the sharp constant in Lemma 3.1 is $1/\\sqrt{n}$ for all $p$ is a standalone mathematical question; resolving it would tighten the loss lower bound and sharpen the claimed necessity."],"forward_implications":["If the necessity claim holds, a flat loss curve with a stagnant $\\epsilon$-rank is a capacity signal: the network is not failing because of learning rate or optimizer choice, but because its last hidden layer lacks functionally independent features.","A practical corollary is that initialization and architecture choices that raise the initial $\\epsilon$-rank should reduce or remove the early-training plateau, which the paper demonstrates with its first-layer pre-training strategy.","The lower bound ties optimization to approximation theory: the achievable loss after training is limited by $\\operatorname{dist}(u^*, F_p)$, so progress requires effective rank to grow until the target is well approximated by the feature span.","Layer-wise experiments showing higher $\\epsilon$-rank in deeper layers suggest that feature diversity is built up through the network, so shallow bottlenecks can be identified by comparing per-layer ranks.","The staircase is observed across function fitting, PDE solving, and image classification, so if universal it applies to the common deep-learning pipeline for scientific computing as much as to vision tasks."],"supporting_citations":[{"why":"Supplies Lemma 3.1, the rank-revealing QR submatrix bound used to control the approximation error in Theorem 3.1.","marker":"[13]"},{"why":"Guarantees that the activation-based function class satisfies the universal approximation property, which frames the assumption that the exact solution lies outside the network class.","marker":"[14]"},{"why":"Defines the standard initialization whose low functional diversity produces the initial rank deficiency and plateau the paper analyzes.","marker":"[9]"},{"why":"Provides the random-direction sampling idea that the UDI pre-training strategy uses to avoid explicit grids in high dimensions.","marker":"[38]"},{"why":"The partition-of-unity random feature method is the high-rank baseline compared against extreme learning machines in Example 3.2.","marker":"[2]"},{"why":"Extreme learning machine is the no-partition-of-unity baseline in the comparative rank study.","marker":"[16]"},{"why":"Random features provide the underlying approximation model for the ELM/RFM comparison and motivate the random-direction construction.","marker":"[26]"},{"why":"Physics-informed neural networks are the framework used for the PDE experiments that show the staircase phenomenon.","marker":"[27]"}],"fun_headline_variants":["ϵ-rank predicts loss plateaus: neural net staircase decoded","Staircase loss drops explained by new ϵ-rank metric","Feature rank climbs stepwise as neural net loss falls","Fix training plateaus by pre-raising network feature rank","Deep net training steps: rank up, loss down"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's necessity conclusion rests on the assumption, stated in Section 3.1, that the exact solution $u^*$ does not belong to the network function class $F_n$; if the target can be represented exactly or very accurately by a small number of features, loss can go to zero while the $\\epsilon$-rank stays low, and the claimed necessity would fail.","fun_headline_variants_meta":{"raw":{"variants":["ϵ-rank predicts loss plateaus: neural net staircase decoded","Staircase loss drops explained by new ϵ-rank metric","Feature rank climbs stepwise as neural net loss falls","Fix training plateaus by pre-raising network feature rank","Deep net training steps: rank up, loss down"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00062,"raw_usage":{"total_tokens":2945,"prompt_tokens":1086,"completion_tokens":1859,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":702,"completion_tokens_details":{"reasoning_tokens":1790}},"tokens_in":702,"tokens_out":1859,"duration_ms":15161,"temperature":1.0,"reasoning_tokens":1790,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:50:29.369621+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a wide network on a target that is exactly a linear combination of two fixed basis functions (for example, $u^*(x)=\\cos x+2\\sin x$). If the training loss can be driven to machine precision while the last hidden layer's Gram matrix has $\\epsilon$-rank near 2 throughout the run, the necessity claim is refuted. The same test can be run on a classification dataset whose labels depend linearly on a low-dimensional projection of the input.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies Lemma 3.1, the rank-revealing QR submatrix bound used to control the approximation error in Theorem 3.1."},{"cited_title":"Understanding the difficulty of training deep feedforward neural networks","cited_arxiv_id":null,"evidence_quote":"Defines the standard initialization whose low functional diversity produces the initial rank deficiency and plateau the paper analyzes."},{"cited_title":"Transferable Neural Networks for Partial Differential Equations","cited_arxiv_id":null,"evidence_quote":"Provides the random-direction sampling idea that the UDI pre-training strategy uses to avoid explicit grids in high dimensions."},{"cited_title":"The Random Feature Method for Time-Dependent Problems","cited_arxiv_id":null,"evidence_quote":"The partition-of-unity random feature method is the high-rank baseline compared against extreme learning machines in Example 3.2."},{"cited_title":"Extreme learning machine: Theory and applications","cited_arxiv_id":null,"evidence_quote":"Extreme learning machine is the no-partition-of-unity baseline in the comparative rank study."},{"cited_title":"Random features for large-scale kernel machines","cited_arxiv_id":null,"evidence_quote":"Random features provide the underlying approximation model for the ELM/RFM comparison and motivate the random-direction construction."}],"review_version":1}