{"id":"3594247c-19db-4d42-8f9e-a4c9359d9c48","arxiv_id":"2412.18756","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"The paper reviews fixed-kernel neural network theory and proposes an over-parameterized Gaussian sequence model as a prototype for feature learning.","lead":"An arXiv paper surveys neural tangent kernel and kernel regression theories, then proposes an over-parameterized Gaussian sequence model as a toy model for how neural networks learn features. The proposal is a starting point: the authors state that the needed equivalence proofs and dynamic analysis remain open problems.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central prototype rests on the §4.2 equivalence; the required Gram-matrix concentration cannot hold in operator norm when active features outnumber samples, so the model's link to neural feature learning is unestablished in the high-dimensional regime.","rationale":"Both the reader and I identify the same load-bearing point: the §4.2 equivalence between kernel gradient flow and the Gaussian sequence model. The review material is accurate and well-cited, and the proposed model is a reasonable research direction, but the bridge is a heuristic concentration assumption that the paper itself lists as open in §5. I sharpen the concern: when the active feature dimension exceeds the sample size, the empirical Gram matrix is finite-rank and cannot approximate the identity in operator norm, so the equivalence is structurally doubtful in the high-dimensional regime that motivates feature learning. This does not change the verdict: the paper is honestly framed as a proposal, so a conditional acceptance remains appropriate. Secondary issues—unspecified orthogonal projection in Definition 4.3.1 and a notation inconsistency between D in §4.3.1 and D_t in Definition 4.3.1—are fixable and do not alter the core conditional.","tokens_in":25866,"tokens_out":14508,"duration_ms":136557,"concrete_test":"Simulate the exact finite-N kernel gradient flow (19) and the GSM gradient flow (21) for an inner-product kernel on S^{d−1} with n≍d^γ, N=2n, a structured f*, and the same noise realization; measure sup_t∥β_t^{KGF}−β_t^{GSM}∥ and the generalization-error gap. If the gap does not vanish as d,n grow, the §4.2 equivalence fails in the high-dimensional regime, and the over-parameterized GSM loses its stated connection to adaptive features. An analytic companion would be to prove or disprove the weighted operator-norm bound ∥Λ^{1/2}(G−I)Λ^{1/2}∥_op = o(1) under the eigenvalue decay of §2.4.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2.1 derives Eq. (22) by replacing G=(1/n)Ψ(X)Ψ(X)⊤ with I and (1/n)Ψ(X)Y with Λ^{1/2}β* plus Gaussian noise. Entrywise LLN supports this for any fixed coordinates, but the gradient-flow equivalence requires a uniform approximation over the evolving β_t. When the active feature dimension N exceeds n, G has rank at most n, so ∥G−I∥_op ≥ 1; the identity is not a uniform limit. This finite-rank effect is not a negligible technicality: the high-dimensional phenomena reviewed in §2.4—polynomial approximation barrier, multiple descent, phase diagrams—are driven by the non-identity structure of empirical kernel matrices. The paper states in §5 that a rigorous equivalence, especially in high dimensions, remains open. Because the over-parameterized Gaussian sequence model is introduced as a prototype through exactly this equivalence, the central claim is conditional on a hypothesis that is unsupported in the regime where feature learning is supposed to matter.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper is a review-and-proposal article. It first reviews the neural tangent kernel theory and recent kernel regression results in both fixed and high dimensions, then discusses limitations of fixed-feature theories and recent work on feature learning. Its original contribution is the over-parameterized Gaussian sequence model (Definition 4.3.1), in which an orthogonal matrix A_t and a diagonal matrix D_t evolve together with a coefficient vector alpha_t under gradient flow; the paper argues that this model is a prototype for the adaptive feature model of neural networks. The link between kernel gradient flow and the Gaussian sequence model is explicitly presented as a hypothesis (Section 4.2), with supporting heuristics and a review of matching convergence rates, while the rigorous equivalence in high dimensions is listed as an open problem in the discussion (Section 5).","tokens_in":26116,"tokens_out":7468,"duration_ms":69638,"significance":"The review component is competent and genuinely useful: it synthesizes a large body of recent work on kernel regression learning curves, interpolation, and high-dimensional phenomena, and it frames feature learning through kernel alignment in a way that connects several threads of the literature. The paper's main value is conceptual: the over-parameterized Gaussian sequence model is a clean, tractable object that isolates the possible effect of learnable features and eigenvectors on gradient-flow dynamics, and the authors are transparent about the unproven nature of the kernel-to-sequence equivalence. The proposal is falsifiable in the sense that its usefulness depends on a specific conjectured equivalence, and the simulation in Section 4.3.3 demonstrates a qualitative alignment effect. However, the manuscript does not establish the model's connection to actual neural networks in the high-dimensional regime where feature learning matters, and it lacks any quantitative demonstration that the full (A_t, D_t) model improves upon simpler variants already in the literature.","major_comments":[{"comment":"The approximation (1/n)Psi(X)Psi(X)^T = I used to pass from kernel gradient flow (19) to the sequence-model dynamics (22) is asserted to hold 'as the sample size n increases', but the claimed equality of the two gradient flows requires uniform control over the evolving beta_t. When the number of active features exceeds n, the empirical feature covariance has rank at most n, so its operator norm distance to the identity is at least 1; moreover, the non-identity structure of the empirical kernel is precisely what drives the high-dimensional phenomena reviewed in Section 2.4 (polynomial approximation barrier, multiple descent, phase diagrams). Section 5 lists the rigorous equivalence, especially in high dimensions, as an open problem. Since Definition 4.3.1 is motivated through this equivalence, the paper's central prototype is at present a model of the Gaussian sequence model rather than of neural-network feature learning in the high-dimensional regime. Please either restrict the claimed scope, provide quantitative evidence (for example, simulations comparing kernel gradient flow and the sequence model in high dimensions) that the finite-rank effect does not qualitatively change the dynamics, or state the conjectured equivalence with the precise conditions under which it is expected to hold.","section":"Section 4.2.1, Eq. (22)"},{"comment":"The gradient flow in (28) is written as unconstrained Euclidean gradient descent, yet the model requires A_t to remain orthogonal and D_t to remain diagonal. The text states that these properties are 'kept' during gradient flow, but it does not specify the mechanism (for example, projection onto the Stiefel manifold, reparametrization, or a natural-gradient update). The simulation in Section 4.3.3 also does not describe the update rule used for A_t and D_t. Without this specification the model is not fully well-defined and the simulation is not reproducible; please define the constrained dynamics explicitly, or clarify that the gradient-flow statement is a formal abbreviation for a constrained gradient flow.","section":"Section 4.3.2, Definition 4.3.1 and Eq. (28)"},{"comment":"The simulation in Section 4.3.3 shows that the top-projection percentage increases with iterations, but it does not quantify generalization error or compare the over-parameterized model's risk against the vanilla gradient flow (21) or the D-only model of Li and Lin (2024). The expectation stated in Section 4.4 that the additional A_t dynamics yield 'greater improvements' is therefore not supported by the paper's own experiments. To substantiate the claimed benefit of the orthogonal component, the paper should provide at least one direct comparison of the three models' risks (for instance, as a function of n or t) or, alternatively, explicitly mark this as an open conjecture rather than a demonstrated advantage.","section":"Section 4.3.3 and Section 4.4"}],"minor_comments":[{"comment":"The notation Omega-tilde is defined in the text but is not included in the global notation list at the end of Section 1; adding it would help readers. Also, the sentence before Eq. (9) states 'with beta > 1, s > 0' but the embedding index assumption is invoked without being displayed again; a brief reminder would improve readability.","section":"Section 2.3.2, Eq. (9)"},{"comment":"The caption mentions 'dashed lines' and 'Figure 1 (a)', but the displayed figure has no panel label (a); please label the panel or update the caption so the reference is meaningful.","section":"Figure 1"},{"comment":"The relation between the feature dimension m = 500 used in the projection percentage and the network widths described in the experimental details is not stated explicitly. Please clarify how m is obtained from the fully connected network (for MNIST) and the convolutional network (for CIFAR-10).","section":"Section 4.1, Figure 5"},{"comment":"In the sentence beginning 'As in Section 3.2, we hope that projections...', the notation u_j is introduced as the j-th column vector of A_t, but if N is infinite this is informal; please clarify whether N is assumed finite in Definition 4.3.1 or use an index-set formulation for the infinite case.","section":"Section 4.3.2"},{"comment":"The inner product notation <f*, psi_j>_{L2} is used before the underlying space L^2(X, mu) is explicitly identified in the context of Mercer's decomposition; a one-line definition at first use would avoid ambiguity.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is best classified as a perspective or proposal paper rather than a standard technical contribution. The editorial question is whether the journal welcomes such papers; if it does, this paper could be publishable after revision. The central hypothesis should be more carefully scoped, and at least one direct quantitative comparison of the proposed model with the vanilla and D-only baselines is needed. The authors rely heavily on their own related work (especially Li and Lin 2024) for the main evidence that the over-parameterized model improves generalization; an independent or more detailed experimental test would strengthen the proposal. The review sections are solid and will be useful to readers even if the proposed model's status remains conjectural."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about arXiv:2412.18756. First, the survey parts (Sections 2–3) are well done and genuinely useful: they organize the NTK and kernel regression literature around learning curves, the polynomial approximation barrier, multiple descent, interpolation phase diagrams, and recent feature-learning results. Second, the new contribution is Definition 4.3.1, the over-parameterized Gaussian sequence model, in which both an orthogonal A_t and a diagonal D_t evolve under gradient flow. That is a real, if small, extension of Li and Lin (2024), and the intuition—that feature learning should be about aligning A_t and D_t with θ*—is sensible and cleanly stated.\n\nThe honest reading is that the paper proposes a direction rather than proving a theorem. The bridge from kernel gradient flow to the Gaussian sequence model in Section 4.2 rests on the approximation (1/n)Ψ(X)Ψ(X)^T ≈ I. Entrywise, for fixed coordinates, a law of large numbers gives something like this, but the gradient flow requires a uniform approximation over the evolving β_t, and when the active feature dimension exceeds n, the empirical Gram matrix has rank at most n, so the operator-norm distance to I is at least 1. That is not a technicality: the high-dimensional phenomena reviewed in §2.4 are driven by the non-identity structure of the empirical kernel matrix. The authors know this—Section 5 explicitly says a rigorous equivalence, especially in high dimensions, is open. So the central prototype is conditional on a hypothesis that is unsupported in exactly the regime where feature learning is supposed to matter.\n\nThe simulation in §4.3.3 is illustrative: it shows the expected concentration of projections, but there is no code, no error bars, and the orthogonality-preserving update for A_t is not specified. Those are addressable, but they make the empirical support thin.\n\nWho should read this? People who want a compact, accurate survey of kernel regression and feature learning, and people working on sequence-model prototypes. The paper deserves a serious referee: the survey alone is worth publishing, and the prototype is well-defined enough to invite follow-up work. A referee should insist on a precise statement of the equivalence conditions and on code or at least a full description of the simulation.\n\nMy recommendation: send it out for peer review, but expect heavy revision. I would cite it as the reference for the over-parameterized Gaussian sequence model.","headline":"Solid survey, genuine new prototype, but the bridge to neural networks is a hypothesis that fails in the high-dimensional regime where it matters most.","tokens_in":26624,"tokens_out":3776,"would_cite":true,"duration_ms":35442,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes that feature learning in neural networks can be studied through an over-parameterized Gaussian sequence model, where training is gradient flow on an orthogonal rotation and a diagonal scaling that align with the true…","keywords":["neural networks","generalization ability","feature learning","kernel regression","neural tangent kernel","over-parameterized Gaussian sequence model","adaptive feature model","gradient flow"],"falsifier":"Compute the empirical feature covariance $n^{-1}\\Psi(X)\\Psi(X)^\\top$ for an inner-product kernel on the $d$-sphere with $n \\asymp d^\\gamma$ and a source-condition truth, then compare the actual kernel-gradient-flow generalization error with the Gaussian-sequence-model prediction $n^{-s\\alpha} + n^{-(1-\\alpha/\\beta)}$ for the same regularization schedule; if the two diverge as the covariance deviates from $I$, the equivalence is falsified at that scaling.","tokens_in":25674,"feed_emoji":"🧠","tokens_out":8194,"duration_ms":97353,"temperature":0.7,"pith_summary":"The paper argues that the statistical essence of feature learning in neural networks can be captured by a much simpler object: an over-parameterized Gaussian sequence model, in which data are noisy coefficients of the truth in a fixed basis and training evolves an orthogonal rotation and a diagonal scaling alongside a linear readout. The supporting claim is that kernel gradient flow—the infinite-width limit of neural-network training—is strongly equivalent to gradient flow in a classical Gaussian sequence model once the empirical feature matrix is close to orthonormal. If that equivalence holds, changing the network's features corresponds exactly to multiplying the sequence model by a learnable orthogonal-diagonal pair, so feature learning reduces to tracking how that pair aligns with the true parameter vector. The paper reviews why fixed-kernel theory cannot explain this alignment, presents supporting rate-matching calculations and simulations, and leaves the rigorous high-dimensional equivalence as an open problem. A sympathetic reader would take the proposal as a new prototype for turning the question 'how do networks learn features?' into a tractable statistical question about rotation-and-scale alignment.","feed_headline":"Neural feature learning reduces to a Gaussian sequence model","feed_subtitle":"A learnable rotation and scaling explain how networks align features with the signal, beyond fixed-kernel theory.","key_machinery":"The central object is the over-parameterized Gaussian sequence model: noisy coefficients $z_j = \\theta^*_j + \\xi_j$ are explained by a product $A_t D_t \\alpha_t$, with $A_t$ orthogonal (rotation of the basis), $D_t$ diagonal (rescaling of eigenvalues), and $\\alpha_t$ the readout. The work it does is to convert the intractable coupled dynamics of an adaptive feature model into a structured matrix-factorization flow: rotating and scaling the frame is exactly what a neural network does when it learns features, and the model reduces to vanilla Gaussian sequence flow when $A_t$ and $D_t$ are frozen. Carrying the argument is the hypothesized equivalence between kernel gradient flow and Gaussian sequence gradient flow, made plausible by the formal identity of the coordinate-wise dynamics and by the matching learning-curve rates in the regularized regime.","core_discovery":"The paper's central proposal is the over-parameterized Gaussian sequence model (Definition 4.3.1): observations $z_j = \\theta^*_j + \\xi_j$ with $\\xi_j \\sim N(0,n^{-1})$ are fitted by gradient flow on the loss $L(A,D,\\alpha) = \\|Z - AD\\alpha\\|_2^2$, where $A_t$ stays orthogonal, $D_t$ stays diagonal, and $\\alpha_0 = 0$. The authors hypothesize that this model is the adaptive-feature counterpart of the classical Gaussian sequence model, in the same way that the adaptive feature model $\\Phi_{\\eta_t}(x)^\\top \\beta_t$ is the adaptive counterpart of kernel regression. Concretely, they argue that replacing a kernel's eigenfunctions and eigenvalues by a new feature is equivalent to multiplying the sequence model by an orthogonal-diagonal matrix pair, so learning features means learning $A_t$ and $D_t$. They support the equivalence by showing that kernel gradient flow and the Gaussian sequence gradient flow obey the same coordinate-wise dynamics under the approximation $\\frac{1}{n}\\Psi(X)\\Psi(X)^\\top \\approx I$ and $\\frac{1}{n}\\Psi(X)Y \\approx \\Lambda^{1/2}\\beta^* + \\text{noise}$, and by showing the two models produce identical learning-curve rates $n^{-s\\alpha} + n^{-(1-\\alpha/\\beta)}$ in the regularized regime. The intended payoff is that feature learning and its generalization benefits can be studied through the alignment of $A_t$ and $D_t$ with $\\theta^*$, a question the paper poses as the next step and partly illustrates by simulation.","pith_inferences":["The model can be read as a nonconvex matrix factorization $Z \\approx A_t D_t \\alpha_t$; existing results on gradient-flow implicit regularization in factorization problems could give the first rigorous statements about which structures of $\\theta^*$ the flow aligns with, a direction the paper does not pursue.","A direct way to test the equivalence is to simulate kernel gradient flow on an inner-product kernel in high dimensions where the empirical covariance is known not to concentrate, then compare the trajectory against the Gaussian sequence flow; any rate gap would delimit where the prototype stops being faithful.","If the equivalence holds, the paper's prototype reframes the benign-overfitting debate: interpolation in the over-parameterized model corresponds to late-time growth of $D_t$, so the model could explain when interpolation is benign versus catastrophic as a function of $\\theta^*$ alignment, which the paper only touches implicitly.","The alignment measure used in the simulations (percentage of first-$p$ projections) is a testable statistic; applying the same measure to real networks with varying width and depth would provide a scaling-law check of whether the prototype's rotation dynamics match practice."],"forward_implications":["Feature learning becomes a quantitative alignment problem: the key observable is how the projections $u_j^\\top \\theta^*$ (columns of $A_t$) concentrate on the large entries of $D_t$, and generalization should improve as this concentration increases.","The over-parameterized model reproduces kernel-gradient-flow rates when $A_t$ and $D_t$ are frozen, so it generalizes NTK and kernel theory as a special case rather than contradicting it.","The model predicts that allowing $A_t$ and $D_t$ to move yields faster convergence than vanilla sequence flow, consistent with the one-step feature-learning gains and the over-parameterized regression results reviewed in Section 4.4.","If the equivalence is made rigorous, the polynomial approximation barrier for fixed kernels becomes a statement about frozen $A$ and $D$; adaptive features should beat it whenever the truth is aligned with a low-dimensional learnable subspace.","The framework suggests concrete comparisons between architectures: different network structures induce different families of orthogonal and diagonal updates, so their feature-learning efficiency can be compared through the resulting alignment dynamics."],"supporting_citations":[{"why":"Introduces the neural tangent kernel and establishes the kernel-gradient-flow limit that the paper treats as the fixed-feature baseline.","marker":"Jacot et al. (2018)"},{"why":"Shows that infinitely wide networks match kernel prediction, providing the NTK-regime result whose limitations motivate adaptive features.","marker":"Arora et al. (2019b)"},{"why":"Proves that wide two-layer ReLU networks uniformly approximate kernel gradient flow, the link that justifies the kernel-regression framing.","marker":"Lai et al. (2023)"},{"why":"Defines the classical Gaussian sequence model that the proposed over-parameterized model adapts.","marker":"Johnstone (2017)"},{"why":"Supplies the learning-curve rates for kernel gradient flow that the Gaussian sequence model reproduces in Section 4.2.2.","marker":"Li et al. (2024a)"},{"why":"Establishes the polynomial approximation barrier for high-dimensional inner-product kernels, the benchmark that feature learning must beat.","marker":"Ghorbani et al. (2021)"},{"why":"Shows that one gradient step can improve features over random features, a concrete feature-learning phenomenon the prototype aims to generalize.","marker":"Ba et al. (2022)"},{"why":"Analyzes the special case of the proposed model with only the diagonal learnable and demonstrates faster rates than vanilla flow.","marker":"Li and Lin (2024)"},{"why":"Provides evidence that over-parameterized gradient descent can enjoy faster convergence in sparse linear regression, supporting the over-parameterization intuition.","marker":"Zhao et al. (2022)"}],"fun_headline_variants":["A Gaussian sequence model for neural feature learning","Feature learning as adaptive rotation and scaling","Beyond kernels: features learn via Gaussian model","Prototype model reveals how networks learn features","From NTK to adaptive features: a Gaussian lens"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole argument rests on treating the data's feature vectors as nearly uncorrelated and the labels as nearly equal to the true coefficients plus independent noise; the paper itself notes that a rigorous justification of this concentration, especially in high dimensions, remains open.","fun_headline_variants_meta":{"raw":{"variants":["A Gaussian sequence model for neural feature learning","Feature learning as adaptive rotation and scaling","Beyond kernels: features learn via Gaussian model","Prototype model reveals how networks learn features","From NTK to adaptive features: a Gaussian lens"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000176,"raw_usage":{"total_tokens":1322,"prompt_tokens":1011,"completion_tokens":311,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":627,"completion_tokens_details":{"reasoning_tokens":244}},"tokens_in":627,"tokens_out":311,"duration_ms":3395,"temperature":1.0,"reasoning_tokens":244,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:29:42.703231+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the empirical feature covariance $n^{-1}\\Psi(X)\\Psi(X)^\\top$ for an inner-product kernel on the $d$-sphere with $n \\asymp d^\\gamma$ and a source-condition truth, then compare the actual kernel-gradient-flow generalization error with the Gaussian-sequence-model prediction $n^{-s\\alpha} + n^{-(1-\\alpha/\\beta)}$ for the same regularization schedule; if the two diverge as the covariance deviates from $I$, the equivalence is falsified at that scaling.","supporting_citations":[],"review_version":1}