{"id":"eb18bc86-4167-48ff-b5de-523dd8be8087","arxiv_id":"2608.13335","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Permutation symmetry alone forces the leading behavior of a neural layer into a quadratic form, and the resulting dynamics unify sudden learning and training-time scaling laws.","lead":"The paper derives a universal quadratic approximation for neural network layers from permutation symmetry, in which the architecture is encoded in a single structure matrix. It shows that this approximation produces both sudden step-like learning and smooth power-law scaling from one Lotka-Volterra dynamics.","discovery_kind":"unification","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Truncation error is uncontrolled exactly at the mode-ignition events the theory predicts, so the NQF dynamics are only proven for the quadratic surrogate, not for the original networks.","rationale":"The reader's weakest assumption and my independent stress-test converge on the same load-bearing concern: Theorem 1 is purely local, while the derived sudden-learning and scaling-law predictions concern the nonlocal training trajectory. The mathematical core of the paper is sound — Theorem 1 follows from a Taylor expansion under S_d symmetry and ZGZ, the closure on (M, μ) is exact for the quadratic form, and the Lotka-Volterra solutions are internally consistent. The issue is the bridge from the quadratic surrogate to the original architecture. The paper itself provides evidence that this bridge is fragile: Figure 6 shows divergence after about 100 steps in teacher-student settings, and Limitation 1 explicitly concedes that the approximation fails at larger initialization. Crucially, the very events the theory explains — modes switching on and saturating — take the weights from O(sqrt(ϵ)) to O(1), so the expansion parameter stops being small. No trajectory-length error bound is supplied. This does not invalidate the paper's local normal-form result, nor its exact analysis of NQFs; it does mean the central claim about real perceptrons, attention layers, and other architectures should remain conditional, which is exactly what the reader's CONDITIONAL verdict expresses. I therefore recommend no change to the verdict. The proposed concrete test would settle the matter directly by measuring the truncation error at the predicted ignition time and checking whether it vanishes as ϵ → 0; if it does, the concern is resolved, and if it does not, the transfer claim would need a new argument.","tokens_in":54774,"tokens_out":5584,"duration_ms":57188,"concrete_test":"Run the Figure 6 teacher-student experiment for a two-layer tanh MLP with ϵ ∈ {1e-3, 1e-4, 1e-5}, compute the NQF structure matrix A(x) from Proposition 1, and integrate the NQF gradient flow (or discrete GD) to find the first ignition time t*_1 where z_1(t) reaches half saturation. At t*_1, compare the true MLP loss gap Δ_true = |L_MLP(t*_1) - L_NQF(t*_1)| with the total drop L_NQF(0) - L_NQF(∞), and also estimate the cubic remainder R = ||f_MLP(W) - f_NQF(W)|| at that same W via finite differences with respect to a scale factor. If Δ_true > 10% of the total drop, or if R / Tr[W W^T A] > 0.1 at t*_1, the truncation error is not negligible at the mode-activation event and the transfer claim fails. Repeat across ϵ; if the normalized gap does not shrink as ϵ → 0, the small-initialization limit does not rescue the trajectory approximation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The weakest link is not the local expansion but the transfer of the NQF dynamics to the original network across the whole training trajectory. Theorem 1 is a statement at W=0 with remainder O(||W||^3); Sections 5–6 then solve the dynamics of the truncated quadratic model and use those solutions to predict plateau times t*_k ~ (1/ζ_k) log(1/ϵ) and scaling exponents. But the predicted feature activation is itself a process that drives W away from the expansion point: at ignition z_k grows from O(ϵ) to O(1), so ||W|| grows like ϵ^{-1/2} times a data-dependent constant, and the cubic remainder becomes O(1) in the very regime the theory is about. No trajectory-length bound is given, and Appendix Figure 6 is direct evidence: in a teacher-student two-layer MLP and in a query-key-only attention model, the NQF and the true network agree for roughly 100 SGD steps and then diverge. Limitation 1 (small initialization) concedes failure at larger initialization, but the feature-saturation events in the theory occur at O(log(1/ϵ)) time, at which the weights are no longer small. Thus the Lotka-Volterra plateaus and the exponent predictions are rigorously proven only for the quadratic surrogate, not for the original architectures; the claim that sudden learning and scaling laws of actual networks are explained by the NQF is conditional on an unverified uniformity of the truncation error.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces the 'neural quadratic form' (NQF), a local normal form for neural network layers with exchangeable units. Theorem 1 states that any C^3 module that is S_d-symmetric and has zero gradient at the origin satisfies f_x(W)=f_x(0)+\\sum_i Tr[w_i w_i^T A(x)]+O(||W||^3), with all architecture-specific information in A(x). The authors compute A(x) for MLPs, CNNs, multi-head attention, query-key-only attention, mixtures of experts, and several matrix factorization models. They show that SGD dynamics of the truncated quadratic model close on M=WW^T and \\mu=\\sum_i w_i (Theorem 2), that the model is compressible (Theorem 3), and that under commutativity or isotropy assumptions the dynamics reduce to Lotka-Volterra equations solvable in four regimes (Theorems 4-7). From these solutions they derive saddle-to-saddle plateaus with t_k^* ~ (1/\\zeta_k) ln(1/\\epsilon) and power-law decay of excess loss with exponents (\\alpha_1-1)/\\alpha_2 and (2\\gamma_1-1)/\\gamma_2. Experiments on synthetic data confirm the surrogate dynamics; a Fourier MLP experiment shows a power-law training loss. The paper candidly lists limitations: small initialization, smoothness, and the assumption of power-law spectra.","tokens_in":55096,"tokens_out":11963,"duration_ms":120546,"significance":"If the central transfer step were established, the paper would provide a genuinely unifying explanation of sudden learning and scaling laws across architectures, with quantitative, testable exponent predictions and explicit architecture-specific matrices A(x). The local expansion theorem and the exact solution theorems are derived in appendices and appear algebraically correct; the code for reproducing the figures is provided. The main weakness is that the solved dynamics are those of the truncated quadratic surrogate, and the O(||W||^3) remainder is not controlled along the trajectory up to feature saturation; the paper's own Figure 6 shows the approximation failing at finite times. The scaling-law exponents are also conditional on assumed power-law spectra. For these reasons the work is best regarded as a significant and useful surrogate-model theory whose transfer to real networks is not yet established.","major_comments":[{"comment":"Theorem 1 is a local statement at W=0: the remainder O(||W||^3) in Eq. (2) is controlled only while the weights remain near the expansion point. In the predicted ignition events, z_k(t) evolves from \\Theta(\\epsilon) to \\Theta(\\zeta_k/C_k)=\\Theta(1) (Eqs. (39) and (42)), so ||W(t)|| grows from \\Theta(\\sqrt{\\epsilon}) to \\Theta(1), and the cubic remainder becomes O(1) precisely in the regime the theory is about. No trajectory-length bound is provided, and Appendix C Figure 6 shows that in a teacher-student MLP and in a query-key-only attention model the NQF and the original network agree for roughly 100 steps and then diverge. Limitation 1 (small initialization) does not rescue the claim because the divergence occurs at the timescale on which the theory's predicted signatures appear. Consequently, Theorems 8-11 establish the plateau times and exponent relations only for the truncated quadratic surrogate, not for the original architectures; the transfer to actual networks is an unverified uniformity assumption. I ask the authors to provide a trajectory error bound or to reframe the main claims as statements about the surrogate.","section":"§2, Eq. (2); §5, Eq. (39); Appendix C, Fig. 6"},{"comment":"Section 5's exact solutions and Section 6's quantitative predictions require Assumption 1 (mutually commuting data matrices) or Assumption 2 (isotropic second moment), and the feature-wise results additionally require orthogonal features. These are strong, non-generic conditions. For example, for the two-layer MLP of Proposition 1, A(x) is proportional to [[0,x],[x^T,0]], and A(x)A(x') is not equal to A(x')A(x) for generic x and x', so Assumption 1 already fails on typical data; the empirical demonstrations in Figures 3 and 4 are engineered to satisfy the assumptions exactly. The paper gives no bound on the error when Assumptions 1 or 2 fail and does not show that plateaus or power laws persist for generic structure matrices. This scope restriction should be stated prominently in the introduction or abstract.","section":"§5, Assumptions 1 and 2; §6, Eq. (46)"},{"comment":"The empirical support for the abstract's claim that sudden learning and scaling laws are confirmed 'across training methods and architectures' is limited. Figure 2 trains the original modules and their NQF approximations on an NQF teacher, which is a favorable setting; Figures 3 and 4 validate the Lotka-Volterra dynamics of the surrogate, not of original networks; and Figure 6, in the teacher-student setting, shows the NQF approximation diverging from the original after about 100 steps. The only original-network power-law experiment is the Fourier MLP in Figure 5. I recommend either adding experiments that verify the plateau times and the exponent predictions on original architectures under small initialization, or explicitly limiting the paper's claims to the NQF surrogate.","section":"§7, Figures 2-6"}],"minor_comments":[{"comment":"The initialization conventions are inconsistent: Theorem 8 assumes z_k(0)=\\Theta(\\epsilon), Figure 3 sets W(0)=\\sqrt{\\epsilon} I so that z_k(0)=\\epsilon, while Figure 5 initializes W from N(0,\\epsilon^2), giving z_k(0)=\\Theta(\\epsilon^2). This changes the prefactor of t_k^* by a factor of two; please standardize the definition of \\epsilon or state the relation between \\epsilon and z_k(0).","section":"§6, Eq. (43); Appendix C"},{"comment":"The text uses \\sigma and \\epsilon interchangeably for the initialization scale; please use one symbol throughout and define it precisely.","section":"§7, Figure 2"},{"comment":"The text says the NQF matches the original for the first 100 steps, but the agreement window is not marked in the figure; please annotate the divergence onset.","section":"Appendix C, Figure 6"},{"comment":"The limitation 'Other plateaus' states that the experiments show plateaus not accounted for by the NQF, but no specific experiment or figure is cited; please add a reference to the relevant panel.","section":"§9, Limitation 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is well-written and the limitations section is honest, which is a strength. The main risk is overclaiming transfer from the local normal form to global training dynamics; the divergence in Figure 6 is direct evidence of that gap. If the authors can add a trajectory error bound, or substantially qualify the claims as statements about the surrogate, the paper could become publishable. No concerns about citation pattern or novelty disclosure."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: the local normal form is the real contribution, and it holds up. Theorem 1 — permutation symmetry plus zero-gradient-at-zero plus C^3 forces f_x(W)=f_x(0)+Σ Tr[w_i w_i^T A(x)]+O(||W||^3) — is a short, self-contained Taylor argument, and the computed structure matrices for attention, MoE, and CNN are genuinely new and useful. Attention reducing to a value/readout average at leading order is a striking, checkable claim. The dynamics theorems (closure on (M, μ), the Lotka–Volterra reduction under commuting data, and the exact solutions) look algebraically correct; I spot-checked the GLV derivation and the Riccati solution in Theorem 7. That is real formal content.\n\nThe soft spot is exactly where the stress test lands: the paper solves the dynamics of the truncated quadratic model, then asserts those dynamics describe the original network over the whole trajectory. Theorem 1 is local at W=0 with an O(||W||^3) remainder, but at the mode-ignition events the theory predicts, z_k grows from O(ε) to O(1), so ||W|| grows like ε^{-1/2} times a data constant and the remainder is O(1). No trajectory-length bound is given, and Figure 6 shows NQF and original diverging after ~100 steps in a teacher-student MLP and an attention model. Limitation 1 concedes small-init only. So the plateau times and scaling exponents are rigorously proven for the surrogate, not for the architectures; that transfer is plausible but unproven. The scaling exponent claims also rest on assumed power-law spectra — Limitation 4 says so plainly — so the title slightly overstates the empirical scope, though the honesty is commendable.\n\nMinor: the appendix has visible proof corruption in a few displays (A.4, B.3, B.5 have scrambled tokens). It doesn't affect the main derivations but needs cleanup. Citation pattern is fine; the overlap with Ref. [80] is acknowledged, and the new structure-matrix results stand on their own.\n\nVerdict: this deserves a serious referee. The normal form and the structure-matrix computations are solid, citable, and likely to be reused. The trajectory-transfer gap should be the main request: either a bound on the truncation error along the flow, or a careful statement that the results are for the NQF surrogate, with the architectural experiments as evidence of approximation only in the small-init regime. I'd take the local derivation now and treat the universal scaling-law explanation as a conditional research program.","headline":"NQF is a clean local normal form with real new content; the transfer to full-trajectory sudden learning and scaling laws is conditional on an unverified truncation bound, so read the derivation and the scaling claims separately.","tokens_in":55614,"tokens_out":1757,"would_cite":true,"duration_ms":17246,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"One symmetry reduces MLPs, attention, and convolutions to a single quadratic model.","keywords":["neural quadratic form","permutation symmetry","sudden learning","neural scaling laws","Lotka-Volterra dynamics","feature learning","structure matrix","small initialization"],"falsifier":"Train a two-layer MLP from a large initialization (say $\\sigma=0.2$ or larger) on a low-rank target and check whether the loss trajectory diverges from the NQF prediction over hundreds of steps; the paper's own Figure 6 indicates divergence after about 100 steps in a teacher-student setting.","tokens_in":54527,"feed_emoji":"📉","tokens_out":2285,"duration_ms":19099,"temperature":0.7,"pith_summary":"This paper tries to establish that the many architectures used in deep learning—perceptrons, attention heads, mixtures of experts, and convolutions—share one leading-order mathematical description when training starts from small weights. The description is a neural quadratic form, in which the only architecture-specific object is a structure matrix $A(x)$. The paper argues that this single model produces both sudden, plateau-like learning and smooth power-law scaling, with exponents predicted from the spectrum of $A(x)$.","feed_headline":"One symmetry unifies MLPs, attention, and convolutions","feed_subtitle":"A single quadratic model predicts plateaus and power-law scaling from the architecture's structure matrix.","key_machinery":"The central object is the neural quadratic form plus the structure matrix $A(x)$, which encodes the architecture. The argument then uses the order parameter $M=WW^\\top$ and, under a commutative-eigenbasis assumption, reduces training to a generalized Lotka–Volterra system for the eigenvalues $z_k$ of $M$.","core_discovery":"Theorem 1 states that any three-times differentiable module with $S_d$-symmetric interchangeable components and zero gradient at zero obeys $f_x(W)=f_x(0)+\\sum_i\\mathrm{Tr}[w_iw_i^\\top A(x)]+O(\\|W\\|^3)$. This forces a universal quadratic leading term where all architecture-specific detail is confined to $A(x)$. The paper computes $A(x)$ for specific architectures and shows how training dynamics close on the order parameter $M=WW^\\top$, reduce to a Lotka–Volterra equation under commuting data, and generate sharp feature-wise or sample-wise transitions whose aggregate produces power-law scaling with predicted exponents.","pith_inferences":["The theory implies that spectral properties of $A(x)$, rather than detailed parameter counts, govern which model sizes show a given learning curve.","One could test whether the predicted exponent holds for real datasets by estimating the spectrum of $A(x)$ from architecture and data alone.","Extending NQF to cubic and quartic terms (as the paper suggests) might explain additional plateaus in training curves."],"forward_implications":["The same NQF predicts plateau-and-drop learning and smooth power-law loss decay within one framework.","Architectures with the same structure matrix have identical training dynamics at leading order.","Sudden learning is a singular limit in initialization scale; smaller initialization separates feature activation times as $\\ln(1/\\epsilon)$.","The scaling-law exponent is determined by the spectral decay of $A(x)$ and the target alignment.","A network can be compressed to a width proportional to the data subspace dimension without changing its learning dynamics."],"supporting_citations":[{"why":"Defines permutation symmetry as the operational definition of a neuron, motivating the main symmetry assumption.","marker":"[77]"},{"why":"The universal compression/lottery-ticket theory that the NQF compression result extends.","marker":"[69]"},{"why":"Exact solutions of deep linear networks showing mode-wise plateaus; the NQF dynamics reduce to that solution in a special case.","marker":"[62]"},{"why":"Quadratic factorization as a solvable proxy for nonconvex learning; its implicit solution is a special case of NQF dynamics.","marker":"[33]"},{"why":"The neural tangent kernel viewpoint that NQF contrasts with and generalizes through feature learning.","marker":"[38]"},{"why":"Spectral theory of neural scaling laws for trained loss; the paper distinguishes its time-based law from this size-based law.","marker":"[7]"},{"why":"Saddle-to-saddle dynamics used to explain sudden learning, which the NQF connects to the same dynamics.","marker":"[1]"},{"why":"Original multi-head attention architecture whose structure matrix is computed in the NQF framework.","marker":"[67]"}],"fun_headline_variants":["A single symmetry unifies all neural architectures","Why neural networks learn in steps: a universal quadratic form","One structure matrix explains plateaus and power laws","Neural learning's hidden symmetry: a universal quadratic","From MLPs to attention: one model for sudden learning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The quadratic truncation remains accurate throughout training, which requires the weights to stay small compared with the feature length scale set by the data; the paper shows empirically that this fails for large initialization.","fun_headline_variants_meta":{"raw":{"variants":["A single symmetry unifies all neural architectures","Why neural networks learn in steps: a universal quadratic form","One structure matrix explains plateaus and power laws","Neural learning's hidden symmetry: a universal quadratic","From MLPs to attention: one model for sudden learning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000541,"raw_usage":{"total_tokens":2594,"prompt_tokens":948,"completion_tokens":1646,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":564,"completion_tokens_details":{"reasoning_tokens":1571}},"tokens_in":564,"tokens_out":1646,"duration_ms":10686,"temperature":1.0,"reasoning_tokens":1571,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:25:49.861236+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a two-layer MLP from a large initialization (say $\\sigma=0.2$ or larger) on a low-rank target and check whether the loss trajectory diverges from the NQF prediction over hundreds of steps; the paper's own Figure 6 indicates divergence after about 100 steps in a teacher-student setting.","supporting_citations":[{"cited_title":"Symmetry induces structure and constraint of learning","cited_arxiv_id":null,"evidence_quote":"Defines permutation symmetry as the operational definition of a neuron, motivating the main symmetry assumption."},{"cited_title":"A universal compression theory for lottery ticket hypothesis and neural scaling laws","cited_arxiv_id":null,"evidence_quote":"The universal compression/lottery-ticket theory that the NQF compression result extends."},{"cited_title":"Implicit regularization in matrix factorization","cited_arxiv_id":null,"evidence_quote":"Quadratic factorization as a solvable proxy for nonconvex learning; its implicit solution is a special case of NQF dynamics."},{"cited_title":"Neural tangent kernel: Convergence and generalization in neural networks","cited_arxiv_id":null,"evidence_quote":"The neural tangent kernel viewpoint that NQF contrasts with and generalizes through feature learning."}],"review_version":1}