{"id":"1f5b0627-146c-470b-a1b2-a898fd852607","arxiv_id":"2411.13512","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Weight matrix dynamics during training is modeled as Dyson Brownian motion, with the learning-rate-to-batch-size ratio controlling stochasticity; verified analytically in a Gaussian RBM and empirically in a nano-GPT.","lead":"During training, neural network weight matrices update in a noisy, random way, and this paper argues that the randomness follows a classic physics model, Dyson Brownian motion, in which the eigenvalues of the weight matrix repel each other. The payoff is a simple ratio, learning rate divided by batch size, which controls how much noise training has, plus a way to watch a small transformer's eigenvalue spectrum evolve while it learns.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Generic DBM claim assumes rotationally invariant SGD noise and a closed, separable eigenvalue process; neither is proven for non-quadratic losses, so the α/|B| Coulomb-gas scaling is established only for the Gaussian RBM.","rationale":"The reader identifies the separable-potential assumption as the weakest point. My stress-test concurs but locates the fragility one step earlier: Eq. (2.3) requires not only K_i = -dV_i/dx_i, but also that the eigenvalue process is closed in x_i at all, which in turn requires the SGD noise to be effectively rotationally invariant or its eigenvector dependence to cancel. The paper's Eq. (2.1) writes the noise as elementwise independent, ignoring correlations that are generically present in minibatch gradients; these correlations couple the eigenvalue dynamics to the eigenvectors. Neither the closure nor the separability is proven for non-quadratic losses. The transformer section does not test the α/|B| scaling and thus does not supply generic evidence. This does not invalidate the Gaussian RBM result, which is exact, so the verdict stays CONDITIONAL as the reader recommended. My concern overlaps with the reader's weakest_assumption but is broader: the separable potential is a special case of the closure problem, and the noise covariance issue is an independent technical gap. Hence 'partial' agreement. The proposed concrete test—an exact perturbative expansion with the full covariance on a solvable linear network—would settle whether Eq. (2.3) follows from Eq. (2.1) at all, and if not, the universal scaling claim must be explicitly restricted to models with isotropic noise and separable drift, of which the Gaussian RBM is the prototype.","tokens_in":6623,"tokens_out":6527,"duration_ms":80134,"concrete_test":"Derive the exact eigenvalue update to first order in α for a single training step of the actual update (2.1) on a one-layer linear network with loss ||Wx - y||^2, using the full empirical minibatch covariance C_{ij,kl} = Cov(∂L/∂W_ij, ∂L/∂W_kl) rather than the diagonal approximation. Expand Δx_i to order α and (α/√|B|) including all eigenvector-dependent terms; if the resulting drift or noise depends on the eigenvectors u_i beyond the apparent Coulomb term, Eq. (2.3) is not the exact consequence of Eq. (2.1). Alternatively, run the same expansion for a small tanh MLP and numerically verify whether the conditional distribution of Δx_i given x varies with the eigenvector basis; a nonzero dependence refutes the closed, autonomous eigenvalue process on which the paper's generic claim rests.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Equation (2.1) replaces the minibatch gradient fluctuation by elementwise independent Gaussians with variance Var(∂L/∂W_ij). In reality, the same minibatch perturbs every weight, so the noise is correlated across matrix elements: Cov(δW_ij, δW_kl) is generically nonzero for (i,j) ≠ (k,l). The Dyson Brownian motion framework requires the underlying matrix Brownian motion to have entry-wise independent, rotationally invariant noise. With a non-isotropic covariance, the eigenvalue process does not close in the variables x_i alone. First-order perturbation of X = W^T W gives d x_i = u_i^T (W^T A + A^T W) u_i dt + u_i^T C dη u_i, where A is the deterministic gradient term, C is the full noise covariance, and u_i are the instantaneous eigenvectors; both terms depend on u_i, not only on x_i. Equation (2.3) omits this eigenvector coupling. Even if one grants closure, the stationary Coulomb gas (2.5) is derived only under the additional assumption that the projected drift is a gradient of a separable potential, K_i(x_i) = -dV_i/dx_i. This is exact for the quadratic Gaussian RBM of Sec. 3.1, where the model is explicitly solvable, but for generic loss landscapes the paper gives no derivation. Section 3.2 concedes that the potential is not known for the transformer ('the exact spectrum or Coulomb gas potential are not known'). Hence the abstract's claim that the dynamics 'can generically be described' by Dyson Brownian motion, and the universal α/|B| scaling of the stationary distribution in Eq. (2.6), rest on an assumption that is tested only in the Gaussian RBM and is not a consequence of Eq. (2.1) for generic architectures.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper claims that the stochastic gradient descent update of a weight matrix W can generically be described by Dyson Brownian motion on the eigenvalues of X = W^T W. The authors write a stochastic eigenvalue equation, Eq. (2.4), and the corresponding stationary Coulomb gas distribution, Eq. (2.5), from which they derive the universal dependence of the stationary spectrum on the ratio alpha/|B| of learning rate to batch size, thereby explaining the empirically observed linear scaling rule. The Gaussian restricted Boltzmann machine is treated analytically, and a nano-GPT transformer is studied empirically, where the authors report Wigner-surmise level statistics and a spectral evolution from a Marchenko-Pastur distribution toward a heavier-tailed distribution.","tokens_in":6956,"tokens_out":3457,"duration_ms":35819,"significance":"If the central claim were established, connecting stochastic gradient descent to Dyson Brownian motion and predicting alpha/|B| scaling of the stationary spectral distribution would be a valuable contribution with consequences for hyperparameter tuning and for the random-matrix description of learning. The paper is honest about the limits of the transformer analysis, and it ships code and data for the RBM part, which is a concrete strength. However, the generic claim is only proven for the Gaussian RBM; for generic architectures it rests on unproven assumptions about noise isotropy, eigenvector decoupling, and separability of the effective potential. The paper's value at present is therefore as a controlled RBM result plus an intriguing but incomplete empirical study, not as a demonstration of the generic universality claimed in the abstract.","major_comments":[{"comment":"The reduction of the matrix update to the closed eigenvalue equation (2.3) is not derived for general losses. For X = W^T W, the first-order perturbation of an eigenvalue involves the instantaneous eigenvectors u_i: d x_i = u_i^T (W^T A + A^T W) u_i dt plus a noise term whose covariance is u_i^T W^T C W u_i dt, where A is the deterministic gradient term and C is the covariance of the minibatch gradient noise. Because the same minibatch perturbs all entries of W jointly, Cov(delta W_ij, delta W_kl) is generically nonzero for (i,j) != (k,l), so the noise is not entry-wise independent. Eq. (2.3) assumes that the eigenvalue process closes in the variables {x_i} alone with a scalar noise strength g_i; without an argument that the eigenvector coupling vanishes or decouples, the statement in the abstract that the dynamics 'can generically be described' by Dyson Brownian motion is not supported.","section":"§2, Eq. (2.3)-(2.4)"},{"comment":"The stationary Coulomb gas distribution (2.5) and the alpha/|B| scaling conclusion (2.6) rest on the assumption, stated just before Eq. (2.5), that the drift K_i can be written as K_i = -dV_i/dx_i with a separable potential V_i(x_i). For a generic loss landscape, the projection of the gradient flow of W^T W onto its eigenvalues need not be a gradient flow on a separable potential; if the potential is non-separable, Eq. (2.5) is not the stationary distribution and the universal alpha/|B| scaling does not follow. The paper proves this assumption only in the Gaussian RBM (via Ref. [8]), and Section 3.2 explicitly states that the Coulomb gas potential is unknown for the transformer. This is a load-bearing gap, not a presentation issue.","section":"§2, Eq. (2.5)-(2.6)"},{"comment":"The transformer experiment does not test the central scaling claim. Agreement with the Wigner surmise after spectral unfolding is a universal fluctuation property that is compatible with many stochastic dynamics and does not identify the specific Dyson Brownian motion of Eq. (2.4). The Marchenko-Pastur fits in Fig. 4 use two free parameters (sigma^2 and area A) and leave a 15-25% tail unexplained, and the paper states that the exact spectrum or Coulomb gas potential is not known. Most importantly, no variation of alpha or |B| is reported for the transformer, so Eq. (2.6) is neither confirmed nor refuted there. The empirical support for the generic claim is therefore substantially weaker than the abstract suggests.","section":"§3.2, Figs. 3-4"}],"minor_comments":[{"comment":"The noise term in Eq. (2.2) is written as sqrt(Var(delta X_ij)) eta_ij, but delta X = W'^T W' - W^T W is nonlinear in delta W and includes the cross term W^T delta W + delta W^T W; the variance and the covariance structure of the noise for X should be computed from this full expression rather than inherited from Eq. (2.1).","section":"§2, Eq. (2.2)"},{"comment":"The continuous-time Dyson equation (3.2) and the discrete update (2.4) are not explicitly matched; the paper should state how the discrete SGD step with learning rate alpha and batch size |B| maps to the continuous-time parameter tau, so that the alpha/|B| dependence in Fig. 1 is unambiguous.","section":"§3.1, Eq. (3.2)"},{"comment":"The text says that the statistical uncertainty is determined by repeating training at least 50 times with bootstrap analysis, but Fig. 4 shows no error bars; please add them or state explicitly that they are smaller than the symbol size.","section":"§3.2, Fig. 4"},{"comment":"There is a typo in 'an Marchenko-Pastur distribution'; it should be 'a Marchenko-Pastur distribution'.","section":"§3.2"},{"comment":"In Eq. (2.5), Z is defined as the integral of P_s, but P_s already contains 1/Z; the definition is circular as written. It would be clearer to define Z as the integral of the unnormalized density product over |x_i - x_j| exp(-sum_i V_i(x_i)/g_i^2).","section":"§2, Eq. (2.5)"}],"recommendation":"major_revision","confidential_remarks":"The paper is essentially a proceedings-style exposition of Ref. [8] for the Gaussian RBM, extended by a transformer case study at an exploratory level. The generic Dyson Brownian motion claim is not yet backed by a derivation that addresses correlated noise and eigenvector coupling. I think the manuscript can be made publishable by explicitly restricting the analytical claim to the Gaussian RBM, presenting the transformer part as an empirical observation, and adding a concrete test of the alpha/|B| scaling where feasible. The current abstract overstates the result."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The genuinely new content in this paper is thin: the DBM equation, the α/|B| scaling, and the Gaussian RBM verification are all carried over from the authors' earlier paper (Ref [8]). What this paper adds is a concise restatement plus an exploratory nano-GPT study. Viewed as a workshop synthesis, that is fine; but the abstract's claim that the dynamics 'can generically be described' by DBM is stronger than what is shown.\n\nWhat the paper does well: the Gaussian RBM part is analytically controlled. The drift K_i and potential V_i are known explicitly, the noise strength is computed, and the numerical demonstration of α/|B| scaling—varying α and |B| independently—is convincing. The picture of eigenvalue evolution from Marchenko-Pastur to Coulomb gas with eigenvalue repulsion is clean and pedagogically useful. The transformer study shows Wigner-surmise level spacing and a non-MP tail, and the authors are honest that it is empirical and incomplete.\n\nNow the soft spots, in proportion. The main issue is the assumption before Eq. (2.5): the drift must be a gradient of a separable potential, K_i = -dV_i/dx_i. For a generic loss and generic architecture, the projected gradient dynamics on eigenvalues does not automatically take that form, and the paper provides no argument beyond the solvable RBM. Relatedly, Eq. (2.1) replaces minibatch gradient fluctuations with entrywise independent Gaussians. In reality the same minibatch perturbs all weights, so the noise is correlated across matrix elements; the DBM framework requires independent, rotationally invariant matrix noise for the eigenvalue process to close. Neither of these is a fatal flaw in the RBM, where everything is solvable, but they undercut the 'generic' claim and the universal α/|B| conclusion for arbitrary architectures.\n\nThe transformer section is the most novel but also the least complete. No code or data are provided for it; the spacing histograms lack error bars; the MP fits use two free parameters with no uncertainty or cross-checks. The authors explicitly say work is in progress, so I read it as a case study, not as evidence for the universal claim.\n\nCitation-wise, leaning on Ref [8] is legitimate—it is the source of the framework—and the paper cites the relevant RMT and ML literature. No citation red flags.\n\nBottom line: as a full journal paper, this deserves genuine refereeing rather than desk rejection. The RBM derivation is worth checking, and the scaling claim is important if true. I'd ask the authors to soften the generic claim, justify or drop the separable-potential assumption, discuss the noise-correlation issue, and either add code/data for the transformer or present it as an explicit preliminary study. For a workshop paper, it is acceptable as is. I would not cite this paper in my own work; I'd cite Ref [8] instead.","headline":"A clean restatement of the authors' earlier DBM framework with a thin, unfinished transformer add-on; the α/|B| scaling is proven for the Gaussian RBM, but the abstract's 'generic' claim outruns the assumptions.","tokens_in":7575,"tokens_out":3757,"would_cite":false,"duration_ms":41506,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["15B52","60J65","82C31"],"pacs":[],"model":"deepseek-v4-flash","headline":"Learning weight matrices are governed by Dyson Brownian motion.","keywords":["Dyson Brownian motion","random matrix theory","eigenvalue repulsion","Coulomb gas","linear scaling rule","restricted Boltzmann machine","transformer","Marchenko-Pastur distribution"],"falsifier":"Train the same transformer on the same data at several pairs $(\\alpha, |B|)$ with equal ratio $\\alpha/|B|$ and compare the stationary eigenvalue spacing distribution after unfolding; any systematic dependence on the individual values of $\\alpha$ or $|B|$ would falsify the claimed universal scaling.","tokens_in":6388,"feed_emoji":"🎲","tokens_out":9516,"duration_ms":94336,"temperature":0.7,"pith_summary":"This paper claims that mini-batch stochastic gradient descent on weight matrices can be reduced to Dyson Brownian motion for the eigenvalues of $X = W^T W$. The resulting eigenvalue update carries a drift, a repulsive Coulomb interaction, and Gaussian noise, and the stationary distribution is a Coulomb gas whose dependence on learning rate $\\alpha$ and batch size $|B|$ enters only through the ratio $\\alpha/|B|$. This derives the empirically observed linear scaling rule from stochastic matrix dynamics rather than treating it as a heuristic. The authors confirm the full Coulomb-gas picture analytically in a Gaussian restricted Boltzmann machine and give empirical support in a small transformer, where the spectrum flows from a Marchenko-Pastur law at initialization to a distribution with a heavy tail.","feed_headline":"Weight-matrix learning is Dyson Brownian motion","feed_subtitle":"New derivation ties the learning-rate/batch-size ratio to a Coulomb gas, with RBM proof and transformer evidence.","key_machinery":"The engine of the argument is the Dyson Brownian motion eigenvalue equation for $X = W^T W$: each eigenvalue moves under a deterministic drift, a pairwise repulsive Coulomb term $\\sum_{j \\neq i} g_i^2/(x_i-x_j)$, and a Gaussian noise term, with explicit factors of $\\alpha$ and $|B|$ shown in Eq. 2.4. Passing to the Fokker-Planck equation converts this into the stationary Coulomb gas measure (Eq. 2.5), in which the potential is a sum of single-eigenvalue terms $V_i(x_i)$ and the noise variance controls the repulsion strength. The combination $\\alpha/|B|$ in Eq. 2.6 is the universal scaling variable; the model-dependent part is the ratio $\\tilde V_i/\\tilde g_i^2$. The Marchenko-Pastur distribution plays the role of the initial condition for this flow.","core_discovery":"The central claim is that the learning dynamics of weight matrices can be projected onto the eigenvalues $x_i$ of $X = W^T W$, yielding the update $x_i \\to x_i + \\alpha \\tilde K_i + \\frac{\\alpha^2}{|B|} \\sum_{j \\neq i} \\tilde g_i^2/(x_i - x_j) + \\frac{\\alpha}{\\sqrt{|B|}} \\sqrt{2} \\tilde g_i \\eta_i$ (Eq. 2.4). The induced Coulomb repulsion among eigenvalues makes the level-spacing statistics Wigner-Dyson, and the stationary solution of the associated Fokker-Planck equation is the Coulomb gas $P_s \\propto \\prod_{i