{"id":"daf0a122-fe3c-41c8-acbf-dd9a5d44a4db","arxiv_id":"1908.02400","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A condition-number-guided heuristic for pruning and rescaling hidden layers produces small feed-forward networks with competitive accuracy on MNIST and Adult Income data.","lead":"This paper proposes using the condition numbers of a network's weight matrices, measured after a few training epochs, to automatically decide how many neurons each layer needs, and to shrink already-trained models. If the heuristic holds up, it could replace trial-and-error architecture sizing with a quick, low-cost computation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The conditioning heuristic is not invariant under neuron rescaling: for ReLU networks, κ(Ŵ^(i)) can be inflated or hidden by scaling a neuron's incoming and outgoing weights without changing the function, so Algorithm 4.1's pruning decisions (and the Section 3 insight) may reflect parameterization…","rationale":"The Reader's weakest assumption is close to mine: the conditioning heuristic lacks a demonstrated connection to true layer redundancy. I sharpen this into a concrete formal objection: condition numbers of stacked weight matrices are not invariant under an exact reparameterization of ReLU networks, so high κ can be manufactured by scaling a single neuron's weights without changing the network function. This directly threatens Algorithm 4.1, whose pivoted-QR column selection depends on column norms, and it also undermines the general 'insight' in Section 3. The paper's empirical results on MNIST and Adult are internally consistent, and Tables 8 and 9 show a correlation between added neurons and increased condition numbers in that particular training setup, so I do not think the paper should be rejected outright. But the central claim, as stated, is not robust until this invariance question is settled. A conditional acceptance is therefore appropriate, with the condition being a demonstrated invariance test or an explicit restriction of the method to a canonical parameterization. The unfinished editorial note in Section 5.5 is a separate draft-quality issue and is not the main technical concern.","tokens_in":17316,"tokens_out":8388,"duration_ms":104324,"concrete_test":"Train a small ReLU MLP on MNIST (e.g., the β=2 model from Section 5.1) and record its test accuracy and per-layer κ(Ŵ^(i)). Pick a layer i with all κ < τ (say τ=25). Choose one neuron j in that layer, multiply column j of W^(i) and b^(i)_j by c=100, and divide row j of W^(i+1) by 100; verify the network output is unchanged up to numerical precision. Run Algorithm 4.1 on both the original and the rescaled network with the same τ. If the original is not squeezed but the rescaled version prunes neurons (or vice versa), then Algorithm 4.1's pruning criterion is not a function-intrinsic measure of excess neurons. Repeating with c=10 and c=1000 shows how quickly the artifact appears.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central premise, stated as the first insight of Section 3 and used by Algorithms 3.1 and 4.1, is that the relative sizes of κ(Ŵ^(i)) indicate which layers have too many neurons. This premise is not invariant under an exact symmetry of the model. For a ReLU (or leaky ReLU) network, choose any neuron j in layer i and any c>0; multiply column j of W^(i) and the j-th entry of b^(i) by c, and divide row j of W^(i+1) by c. The function computed by the network is unchanged, but the singular values of Ŵ^(i) are multiplied by c in the direction of that column, and the singular values of Ŵ^(i+1) are modified by the row scaling. Consequently κ(Ŵ^(i)) can be made arbitrarily large (and κ of other layers changed) without adding or removing a single neuron. Pivoted QR, used in Algorithm 4.1, is also column-norm dependent, so the 'redundant' columns selected are exactly the ones that can be changed by such a reparameterization. Thus two numerically identical trained models—same architecture, same function—can receive opposite pruning recommendations depending on the arbitrary scale of one neuron. The paper never addresses this; Section 5.5 only discusses epoch count and τ selection. This is a correctness risk for the load-bearing assumption, independent of dataset breadth or error bars.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes three algorithms for automatically determining the number of neurons in each layer of a feed-forward neural network. Algorithm 3.1 removes neurons from layers whose stacked weight-bias matrix has a condition number exceeding a threshold τ after short partial training, with the aim of equalizing condition numbers across layers. Algorithm 3.2 scales the resulting layer proportions up or down by factors β, partially trains each candidate q times, and selects the model minimizing 2ε_v − ε_tr. Algorithm 4.1 squeezes an already trained network by using pivoted QR on the stacked matrices to remove columns and associated neurons below a threshold. Experiments on MNIST (using 200 Haar-wavelet features) and the Adult Income dataset report compact networks with accuracies of 98.68% and 86.05%, and the selected β is confirmed as the best in the candidate pool after full training on both datasets.","tokens_in":17665,"tokens_out":7832,"duration_ms":81331,"significance":"If the conditioning heuristic were reliable, the paper would offer a very cheap alternative to neural architecture search: the matrix decompositions are inexpensive relative to training, Algorithms 3.1 and 3.2 avoid training many full models, and Algorithm 4.1 gives a simple post-hoc squeezing procedure. The internal verification is a real strength: the β selected by the validation-based score is indeed the best of the eight candidates on the test set for both datasets, and the algorithm descriptions are precise enough to be reproduced. However, the central premise is an empirically motivated heuristic whose main risk—dependence on arbitrary neuron scaling—is not examined, and the evaluation is too narrow to establish the general claim. The paper would be substantially strengthened by a scale-invariant conditioning measure and a broader evaluation with baselines.","major_comments":[{"comment":"The central premise that a high condition number of the stacked matrix Ŵ^(i) indicates an over-proportioned layer is not invariant under a symmetry of the network. For a ReLU (or leaky ReLU) network, choose any hidden neuron j in layer i and any c > 0; multiplying column j and bias entry j of Ŵ^(i) by c and dividing row j of W^(i+1) by c leaves the network function unchanged, but changes the singular values of Ŵ^(i) and can make κ(Ŵ^(i)) arbitrarily large. Because Algorithm 4.1 uses pivoted QR, which is column-norm dependent, the same trained function can receive opposite squeezing recommendations depending on the arbitrary scale of a single neuron. The paper does not address this; Section 5.5 only discusses η and τ. Please either define a scale-invariant conditioning measure (e.g., column equilibration of Ŵ^(i) before computing κ) or explicitly restrict the claims to a canonical parameterization and provide evidence that the effect does not occur under the used initialization and optimizer.","section":"Section 3 (first insight; Algorithms 3.1 and 4.1)"},{"comment":"The method's free parameters τ and η are calibrated on the same two datasets on which the method is evaluated. Section 5.5 states that τ should be chosen from the observed variance of condition numbers and that η is chosen after an initial experiment on the user's dataset; the reported experiments use τ = 25 and 40 for MNIST and Adult, respectively, and η = 1 and 3. The 'Verifying the results' subsections (Tables 3 and 5) only confirm that the β selected by Algorithm 3.2 is the best in the pool after full training on the same datasets; this is an internal consistency check, not an out-of-sample evaluation of the design procedure. Since the motivation is to handle unfamiliar datasets, an evaluation that holds out the τ/η selection rule (or tests on additional datasets without recalibration) is needed to support the generalization claim.","section":"Section 5.5 (hyperparameter selection) and Tables 3, 5"},{"comment":"The empirical support for the general claims is limited to two datasets, and the MNIST setup is nonstandard: inputs are 200 Haar wavelet coefficients, the network is a plain feed-forward net with no regularization, and accuracy is 98.68%. There are no comparisons with standard architecture-search or pruning baselines under the same budget, and reported accuracies are single runs without error bars (except the q = 5 averaging in Algorithm 3.2). The conclusion in Section 6 that these are 'a complete set of inexpensive and approachable tools' for arbitrary datasets goes beyond the evidence. Additional datasets, baselines, and variance estimates would considerably strengthen the paper.","section":"Sections 5.1–5.4 (evaluation scope)"}],"minor_comments":[{"comment":"The text 'What does \"adaptive\" mean in the next sentence? I meant τ is the mean of condition numbers obtained at each iteration. I changed the sentence.' appears to be an editing note accidentally left in the manuscript; please remove or rewrite the passage.","section":"Section 5.5"},{"comment":"The affiliation footnote reads 'Department of of Computer Science', with a duplicated 'of'; please correct the typo.","section":"Footnote 1 (author affiliation)"},{"comment":"The phrase 'Our three algoritms' contains a typo; it should be 'algorithms'.","section":"Section 1.4"},{"comment":"In the sentence about starting with a modest network, 'contacting it with Algorithm 3.2' should likely be 'contracting it with Algorithm 3.2' (or 'connecting').","section":"Section 5.5"}],"recommendation":"major_revision","confidential_remarks":"The invariance issue in Section 3 is the central technical risk and should be the primary focus of the revision. The paper's own editing artifact in Section 5.5 also suggests it was submitted before final proofreading. I would not recommend acceptance until the conditioning measure is made scale-invariant (or the claims are appropriately restricted) and the evaluation breadth is addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis paper proposes using condition numbers of stacked weight-bias matrices to decide how many neurons each layer of a feed-forward network should have. The authors partially train a network, count small singular values of each layer's matrix, remove neurons from layers whose κ exceeds a threshold, then scale the whole network up or down and pick the size by validation. A separate algorithm prunes already-trained networks using pivoted QR.\n\nWhat's genuinely new is the specific pipeline: condition-number-guided neuron allocation during partial training, followed by proportion-preserving scaling. Earlier SVD work prunes or compresses trained networks; this tries to design from scratch. The internal verification is honest: in both MNIST and Adult, they fully train every candidate model in the β-pool and confirm the validation rule picked the best one. The method is cheap and easy to implement.\n\nThe soft spots are real. First and most important: the central insight is not invariant under an exact symmetry of ReLU networks. Scale any neuron's incoming column and bias by c>0 and divide its outgoing row by c; the function is identical, but κ of that layer's stacked matrix can be inflated arbitrarily and the κ of adjacent layers change. Pivoted QR is column-norm dependent, so the neurons flagged as redundant are exactly the ones whose representation you can change without changing the model. Two parameterizations of the same function can get opposite pruning recommendations. The paper never addresses this; Section 5.5 only discusses choosing η and τ.\n\nSecond, the evaluation is narrow: two datasets, no error bars, hyperparameters tuned on those same datasets, no released code. The MNIST setup is nonstandard (200 Haar wavelet coefficients, feed-forward only). Third, Section 5.5 contains an unfinished editorial note—\"What does 'adaptive' mean in the next sentence? I meant τ is the mean...\"—which suggests the manuscript wasn't fully cleaned before submission.\n\nNone of this makes the empirical results worthless. The method might work for a practical range of networks, but the stated rationale is shaky, and the paper doesn't yet justify that the heuristic tracks function rather than parameterization. As is, I'd send it to a serious referee, but the authors need to engage with the rescaling symmetry, add more datasets and code, and clean up the note. This is a conditional accept, not a reject.\n\nRecommendation: engage with review, but insist on the invariance issue.","headline":"A cheap, clearly specified architecture-search heuristic, but the load-bearing condition-number criterion is not invariant under neuron rescaling, and the manuscript carries an unfinished editorial note.","tokens_in":18141,"tokens_out":2279,"would_cite":false,"duration_ms":24072,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T05","62M45","65F30"],"pacs":[],"model":"deepseek-v4-flash","headline":"Matrix conditioning of partially trained weights identifies which layers carry redundant neurons, so a feed-forward network can be sized without brute-force search.","keywords":["neural architecture design","matrix conditioning","condition number","singular value decomposition","pivoted QR decomposition","neuron pruning","feed-forward neural networks","model compression"],"falsifier":"Run Algorithm 3.1 on a network in which one hidden layer is deliberately made far too wide and a later hidden layer is deliberately made too narrow, then fully train the pruned network. If the intentionally narrow layer ends up with a higher condition number after one epoch and the algorithm therefore removes neurons from it, while the over-wide layer's condition number stays below $\\tau$, the resulting network will be less accurate than the same starting network pruned by removing the same number of neurons from the over-wide layer, refuting the claim that condition number tracks over-proportioning.","tokens_in":17088,"feed_emoji":"🧠","tokens_out":8374,"duration_ms":86307,"temperature":0.7,"pith_summary":"This paper claims that the condition number of each layer's stacked weight-bias matrix, inspected after only one to three training epochs, reveals which layers have more neurons than they need. If one layer's matrix is much worse conditioned than the others, or nearly rank-deficient, the layer is over-proportioned; removing neurons until all layers have condition numbers below a common threshold yields the right relative layer sizes. Scaling that skeleton up or down by uniform factors and choosing the scale with the lowest validation-plus-generalization error then produces a compact network. The authors demonstrate the pipeline on MNIST and the Adult Income dataset, reaching accuracies comparable to the best reported models, and add a QR-based procedure that removes redundant neurons from already-trained networks, sometimes improving accuracy after a short retrain.","feed_headline":"Condition numbers expose where neural networks have too many neurons","feed_subtitle":"Partial training plus a condition-number threshold designs compact models with accuracy near hand-tuned networks.","key_machinery":"The central object is the stacked weight-bias matrix $\\widehat{W}^{(i)}$ and its 2-norm condition number $\\kappa(\\widehat{W}^{(i)}) = \\sigma_{\\max}/\\sigma_{\\min}$, computed with SVD, rank-revealing QR, or pivoted QR decomposition. Algorithm 3.1 repeatedly partially trains the network for a small number of epochs $\\eta$ and deletes neurons from every layer whose condition number exceeds the threshold $\\tau$, using the number of small singular values (or small diagonal entries of $R$ in a QR variant) to decide how many neurons to remove. Algorithm 3.2 then scales all hidden layers by factors $\\beta_j$, partially trains each candidate a few times, and picks the size minimizing $2\\hat{\\epsilon}_{v} - \\hat{\\epsilon}_{tr}$, a combination of validation error and estimated generalization gap. Algorithm 4.1 applies pivoted QR to a fully trained network so the permutation matrix identifies exactly which columns and neurons are redundant and can be removed while keeping the rest of the trained network intact.","core_discovery":"The paper's central claim is that matrix conditioning of partially trained parameters can drive architecture design. Let $\\widehat{W}^{(i)}$ be the weight matrix of layer $i$ with the bias vector appended as a row, and let $\\kappa(\\widehat{W}^{(i)})$ be its 2-norm condition number. The claim is that if $\\kappa(\\widehat{W}^{(i)})$ is large relative to other layers, or if the matrix is close to rank deficient, layer $i$ has an over-proportioned number of neurons; trimming those layers until all condition numbers fall below a threshold $\\tau$, then scaling the whole network while preserving layer proportions, finds a small network with high accuracy. The paper also claims that pivoted QR decomposition of a fully trained network identifies precisely which neurons to remove, and that squeezing an oversized trained network in this way can reduce overfitting and even improve test accuracy after retraining. These claims are supported on MNIST and Adult Income, where the automatically designed networks reach about 98.7% and 86.1% test accuracy.","pith_inferences":["An unstated consequence is that the same two-step recipe could serve as a cheap first pass in neural architecture search: use the condition-number signal to shrink the candidate space of depths and widths before any expensive search begins, since the paper's evidence suggests the signal is visible within a single epoch.","The threshold $\\tau$ is effectively a free parameter tuned by looking at the spread of condition numbers; a natural extension the authors do not pursue is to set $\\tau$ automatically as a function of that spread, for example as a quantile, so the pipeline becomes fully parameter-free.","Because the paper only tests two fully connected datasets, the most direct test of the mechanism on other architectures would be to apply the same partial-training condition-number check inside convolutional or residual networks and see whether the flagged layers align with layers that are actually over-wide."],"forward_implications":["A practitioner with a new feed-forward task and a chosen depth can obtain a near-appropriate per-layer neuron distribution from a few epochs of partial training on a rough starting network, without training many full models.","Because Algorithm 3.2 only changes overall scale and not proportions, the expensive search over network size reduces to evaluating a handful of scaled models, and the selected model should be close to the best achievable among all scales.","For an already-trained oversized network, squeezing with pivoted QR removes redundant neurons cheaply, and retraining the squeezed model can match or improve the original accuracy while cutting parameter count.","The methods do not aim to set benchmark records; their value is reaching modest target accuracy with a compact network and avoiding overfitting, so modest applications with limited compute can afford a tailored architecture."],"supporting_citations":[{"why":"Supplies the early SVD-based idea that small singular values flag redundant neurons in a single-layer network, which the paper extends to multilayer nets.","marker":"[26]"},{"why":"Relates the rank and singular values of a trained weight matrix to the network's complexity and hidden-layer size, grounding the rank-deficiency heuristic.","marker":"[30]"},{"why":"Uses SVD to restructure deep acoustic-model layers by discarding small singular components, a precedent for the paper's layer-by-layer matrix approach.","marker":"[33]"},{"why":"Provides rank-revealing QR factorization, the decomposition used to estimate numerical rank and to detect redundant neurons cheaply.","marker":"[8]"},{"why":"Is the standard source for QR and SVD algorithms and condition-number machinery that the paper's algorithms rely on.","marker":"[14]"},{"why":"Is the earlier method for learning neuron counts without full training that the paper positions its own design algorithm against.","marker":"[1]"},{"why":"Shows SVD-based regularization that encourages rank deficiency to identify compressible layers, supporting the role of rank deficiency as a redundancy signal.","marker":"[2]"},{"why":"Is the dropout regularization baseline the paper compares against when squeezing an oversized Adult model.","marker":"[28]"}],"fun_headline_variants":["Matrix conditioning auto-sizes neural networks","Use condition numbers to trim neural nets","Neural network shapes by matrix conditioning","Conditioning scores guide network pruning","Auto-design neural net layers with matrix conditioning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole method rests on trusting that after one to three epochs of training, the condition number of a layer's weight matrix tells you whether that layer has too many neurons; if early-training conditioning does not track each layer's eventual value to the network, the recipe for pruning and scaling collapses.","fun_headline_variants_meta":{"raw":{"variants":["Matrix conditioning auto-sizes neural networks","Use condition numbers to trim neural nets","Neural network shapes by matrix conditioning","Conditioning scores guide network pruning","Auto-design neural net layers with matrix conditioning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000125,"raw_usage":{"total_tokens":1091,"prompt_tokens":911,"completion_tokens":180,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":527,"completion_tokens_details":{"reasoning_tokens":119}},"tokens_in":527,"tokens_out":180,"duration_ms":2440,"temperature":1.0,"reasoning_tokens":119,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:45:26.524839+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Algorithm 3.1 on a network in which one hidden layer is deliberately made far too wide and a later hidden layer is deliberately made too narrow, then fully train the pruned network. If the intentionally narrow layer ends up with a higher condition number after one epoch and the algorithm therefore removes neurons from it, while the over-wide layer's condition number stays below $\\tau$, the resulting network will be less accurate than the same starting network pruned by removing the same number of neurons from the over-wide layer, refuting the claim that condition number tracks over-proportioning.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the early SVD-based idea that small singular values flag redundant neurons in a single-layer network, which the paper extends to multilayer nets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Relates the rank and singular values of a trained weight matrix to the network's complexity and hidden-layer size, grounding the rank-deficiency heuristic."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Uses SVD to restructure deep acoustic-model layers by discarding small singular components, a precedent for the paper's layer-by-layer matrix approach."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides rank-revealing QR factorization, the decomposition used to estimate numerical rank and to detect redundant neurons cheaply."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Is the standard source for QR and SVD algorithms and condition-number machinery that the paper's algorithms rely on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Is the earlier method for learning neuron counts without full training that the paper positions its own design algorithm against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows SVD-based regularization that encourages rank deficiency to identify compressible layers, supporting the role of rank deficiency as a redundancy signal."},{"cited_title":"Srivastava, G","cited_arxiv_id":null,"evidence_quote":"Is the dropout regularization baseline the paper compares against when squeezing an oversized Adult model."}],"review_version":1}