{"id":"064e19e5-b15b-4280-a286-e29803f4ccce","arxiv_id":"2505.11347","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Training a neural network's tangent kernel with the KARE risk estimate produces a kernel predictor that matches or beats the network itself and its after-training kernel on several benchmarks.","lead":"Instead of training a network to fit the data it has seen, this paper trains a related object, the network's similarity kernel, to minimize an estimate of future test error. Across simulations and several real datasets, this kernel-based predictor matched or beat the original network and its after-training kernel.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"KARE's validity as a training objective for a data-dependent finite-width NTK is unproven; because the same labels y select the kernel and enter KARE, minimizing KARE may not minimize generalization error, leaving the central mechanism unsupported.","rationale":"The reader's weakest assumption is exactly that KARE remains a valid approximation of expected risk after many gradient steps on the kernel parameters. I agree: this is the load-bearing condition for the paper's central claim. If KARE is biased during optimization, then minimizing it does not imply minimizing generalization error, and the empirical wins have no supporting mechanism. The paper explicitly acknowledges in Section 5 that rigorous guarantees are absent, which is an in-scope limitation. A further, distinct concern is the UCI protocol of reporting the highest out-of-sample accuracy over the hyperparameter grid, which can inflate the reported numbers; however, I treat this as a secondary evaluation issue rather than the core conceptual weakness. The proposed concrete test would settle the main concern by checking whether KARE and true test error move together on a representative reported configuration. If they do, the concern does not land for that configuration; if they diverge, the central mechanism is not supported. Since the reader's verdict was already CONDITIONAL, my read does not change it: the paper should be accepted only conditional on a demonstration that KARE tracks generalization error during its own optimization, or on a proper theoretical justification.","tokens_in":18110,"tokens_out":8636,"duration_ms":96224,"concrete_test":"Using the released code, instrument the KARE training loop on the Higgs dataset (depth 2, width 64) and log both the KARE objective and the held-out MSE every 10 epochs over the full 300 epochs. If KARE decreases while held-out MSE increases or plateaus at any extended segment of training, then KARE is not tracking generalization error during optimization and the central mechanism fails. As a complementary check, train NTK-KARE on 800 of the 1,000 training points, evaluate KARE on the remaining 200 points for early stopping or model selection, and compare final test MSE with the reported value; if the validation-selected model performs substantially worse, the original KARE-selected result was likely overfit to the training labels.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on KARE being a faithful proxy for the expected risk of the kernel predictor while the kernel parameters are being optimized. KARE (Section 3.3) is derived and justified for a fixed kernel matrix K(X,X) that does not depend on the labels y used to evaluate the estimator. In NTK-KARE, the kernel parameters theta are updated by gradient descent on KARE (Equation 7), so K(X,X;theta) becomes a function of the same training labels y. The leave-one-out / GCV-style unbiasedness argument for KARE no longer applies: the expectation over y of y^T (K(y)+zI)^{-2} y is not the same as the trace identity that holds for a label-independent K. Section 5 explicitly leaves 'rigorous theoretical guarantees for KARE-based training methods' to future work. With P comparable to or larger than n (e.g., depth 4 width 128 on n=1000), the optimized finite-width NTK may drive KARE down artificially without improving true test error. If this occurs, the reported outperformance of NTK-KARE over the DNN and after-NTK reflects overfitting to the KARE objective rather than a genuine 'training the NTK to generalize' mechanism. This is the least secure condition for the paper's central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes NTK-KARE, a method that explicitly trains the parameters of a neural tangent kernel by minimizing the Kernel Alignment Risk Estimator (KARE) rather than by minimizing empirical risk. The authors interpret the after-training NTK as a gradient booster (Proposition 1), then present experiments on simulated low-rank regression, MNIST, Higgs, and the UCI classification benchmark, claiming that NTK-KARE consistently matches or outperforms both the trained DNN and the after-NTK, and is competitive with Recursive Feature Machines on UCI.","tokens_in":18357,"tokens_out":5160,"duration_ms":56199,"significance":"If the central claim holds, the paper would provide evidence that explicit kernel learning can outperform end-to-end DNN training, and that the after-training NTK is the main event in network training. The paper is clearly written, includes a public code repository, and reports experiments across several settings. However, the validity of KARE as a training objective for a label-dependent, finite-width NTK is not established, and the UCI comparison is weakened by test-set hyperparameter selection and by mismatched loss functions. These issues affect the two most important empirical claims.","major_comments":[{"comment":"KARE's risk-estimation property is established for a fixed kernel matrix that does not depend on the labels y used to evaluate the estimator. In NTK-KARE, the kernel parameters are updated by minimizing KARE on the same y, so K(X,X;theta) becomes label-dependent; the leave-one-out / GCV-style argument no longer applies. Minimizing KARE may then reduce the KARE objective artificially without reducing true test error, especially when P is comparable to n. The paper acknowledges this gap in Section 5. I recommend adding a diagnostic experiment that tracks KARE and test risk over KARE-training iterations, and a variant in which KARE is computed on a label-split or on a hold-out subset, to show that KARE-based optimization tracks generalization rather than merely overfitting the objective.","section":"Section 3.3, Eq. (7); Section 5"},{"comment":"For the UCI experiments, the text states that for NTK and NTK-KARE the authors \"report the highest out-of-sample accuracy obtained over the grid of hyperparameters.\" Selecting hyperparameters by test-set accuracy makes the reported averages, P90/P95, PMA, and Friedman ranks systematically optimistic, and it is unfair relative to baseline models whose hyperparameters were presumably selected by a validation procedure. The UCI comparison should be repeated using a validation-based selection rule, or the full grid results should be reported so that the selection protocol is transparent.","section":"Appendix A.2; Table 1"},{"comment":"The kernel methods and the DNN baseline are trained with different loss functions on classification tasks: NTK-KARE and after-NTK use MSE on {-1,1} (or one-hot) targets, while the DNN baseline uses cross-entropy. This confounds the comparison, since the models are optimizing different objectives and the reported accuracy gains could reflect the choice of training loss rather than the kernel-learning mechanism. Please train the DNN baseline on the same {-1,1} MSE objective, or report results for both losses.","section":"Section 4.3; Appendix A.2"},{"comment":"The proof of Proposition 1 bounds the difference between the true dynamics and the frozen-kernel dynamics by epsilon * eta/n * (t-T) * sup ||ell_hat||. This is not O(epsilon) uniformly in t: for t arbitrarily large after stabilization, the error grows linearly in t-T. The proposition should be restated with a bounded horizon (t-T fixed) or with a more careful dependence on t; as written, the claimed O(epsilon) approximation is not justified for late-time gradient boosting.","section":"Section 3.2, Proposition 1"},{"comment":"The experiments are repeated k=10 times, but the figures report only averages, with no error bars, standard deviations, or confidence intervals. Since the central claim is that NTK-KARE \"consistently\" outperforms DNNs and after-NTK, the reported averages alone do not support that claim; the authors should provide variance information or a paired-test analysis.","section":"Figures 1 and 2; Section 4"}],"minor_comments":[{"comment":"The text attributes KARE to \"Jacot et al. (2018)\"; the correct reference is Jacot et al. (2020).","section":"Section 5"},{"comment":"In the displayed equation for the time derivative of the empirical loss, the kernel should be K(X,X;theta_t), not K(x,X;theta_t); as written, the expression mixes a test-point kernel with the training loss.","section":"Section 3.2, proof of Proposition 1"},{"comment":"The statement that NTK-KARE \"is unaware of the parametric model f(x;theta)\" is misleading: the kernel is constructed from the gradients of f, so the architecture and parameterization of f determine the search space. Please clarify.","section":"Section 3.3"},{"comment":"The ridge parameter in the final predictor is denoted lambda, while the KARE objective uses z_KARE; the relationship between these two regularization parameters should be stated explicitly, since the main text only defines lambda as a scaled trace.","section":"Section 3.3, Eq. (8)"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know about arXiv:2505.11347. The authors propose to train a neural tangent kernel explicitly by minimizing KARE, rather than training the DNN by ERM and then using its after-kernel. That is a genuinely new combination, and the empirical pattern is consistent: across their simulations, MNIST/Higgs, and the UCI benchmark, NTK-KARE beats both the DNN and the after-NTK. If it holds up, it is a practical kernel-learning method that matches RFM on tabular data and sharpens the case that feature learning in DNNs is kernel learning.\n\nProposition 1 is a clean formal statement: once the NTK stabilizes, further gradient steps act like kernel ridge regression on the residuals. That part is standard gradient boosting, and the paper says so. They also provide a GitHub repo; I did not execute the code, so treat the reported numbers as unverified.\n\nThe soft spots are real but not disqualifying. The main one is theoretical: KARE is derived and analyzed for a fixed kernel, but here the kernel parameters are being optimized using the same labels that appear in the estimator. The leave-one-out unbiasedness argument does not automatically survive that coupling. The authors explicitly say in Section 5 that rigorous guarantees for KARE-based training are future work. That is honest, but it means the central mechanism is unproven. The empirical wins could in principle be an artifact of minimizing a biased surrogate. I would not call this fatal—the experiments are plausible and the method is a natural thing to try—but a referee should ask for a sensitivity analysis or a simple condition under which KARE remains a valid training objective.\n\nTwo experimental issues are worth flagging. On UCI, hyperparameters are selected on test accuracy, which inflates the comparison; the same protocol may have been used by prior work, but it is still a weakness. And classification is done with MSE for the kernel methods while the DNN baseline uses cross-entropy, which is an apples-to-oranges comparison. There are also no error bars on the UCI table. None of this looks like fabrication, just standard empirical sloppiness that could be fixed in revision.\n\nThe citation pattern is fine; the authors lean on the after-kernel literature and on Jacot et al. for KARE, and to the best of my reading they do not misrepresent prior work. The RFM benchmark comparison is appropriately cautious.\n\nBottom line: this paper deserves a serious referee. It has a new method, a plausible mechanism, and results interesting enough to spend time on. I would bring it to a reading group to debate the KARE-as-loss question. For my own work, I would cite it as an example of explicit NTK training, though I would not rely on it as proof of the generalization mechanism.","headline":"Explicit NTK training with KARE is a new and plausible method with consistent empirical wins, but the missing theory for KARE as an optimized objective and some protocol issues in the benchmarks mean the central claim is not yet proven.","tokens_in":18905,"tokens_out":2428,"would_cite":true,"duration_ms":23830,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Directly training the kernel of a deep network, using an estimator of test error, can produce a predictor that matches or beats the network's own output.","keywords":["neural tangent kernel","kernel alignment risk estimator","kernel ridge regression","feature learning","generalization","gradient boosting","deep learning","after-training kernel"],"falsifier":"Track the KARE objective and fresh-sample test error jointly during kernel training on a fixed dataset; if there is a step where KARE decreases but held-out error increases substantially, then minimizing KARE is not minimizing generalization and the claimed mechanism fails.","tokens_in":17882,"feed_emoji":"🧠","tokens_out":8699,"duration_ms":77429,"temperature":0.7,"pith_summary":"Deep networks trained by gradient descent implicitly reshape their neural tangent kernel, and the resulting after-training kernel often predicts as well as the full network. This paper claims the kernel itself can be the trained object: instead of minimizing empirical risk with the network output, one can optimize the NTK parameters directly against KARE, an estimator of kernel ridge regression's expected test error. In simulations, MNIST, Higgs, and 121 UCI datasets, the resulting NTK-KARE predictor consistently matches or outperforms both the trained DNN and its after-training NTK. If the claim holds, explicit kernel training is a viable alternative to end-to-end deep learning, and the after-training NTK is not a byproduct but the main event.","feed_headline":"Directly training a network's kernel often beats the network itself","feed_subtitle":"An objective that estimates test error makes the after-training kernel outperform the original DNN in experiments.","key_machinery":"The machinery has three pieces. The data-dependent NTK $K(x,\\tilde x;\\theta_T)=\\nabla_\\theta f(x;\\theta_T)^\\top\\nabla_\\theta f(\\tilde x;\\theta_T)$ is the kernel induced by a trained finite-width network. KARE, the Kernel Alignment Risk Estimator, is a data-only approximation of expected squared error for kernel ridge regression, written in terms of the kernel matrix and ridge parameter: $\\mathrm{KARE}(y,K(X,X),z)=\\frac{1}{n} y^\\top(\\frac{1}{n}K(X,X)+zI)^{-2}y\\,/\\,\\bigl(\\frac{1}{n}\\mathrm{tr}\\bigl((\\frac{1}{n}K(X,X)+zI)^{-1}\\bigr)\\bigr)^2$. The paper uses KARE as the objective for gradient steps on kernel parameters, producing the NTK-KARE predictor. The gradient-boosting decomposition of Proposition 1, $f(x;\\theta_t)\\approx f(x;\\theta_T)+K(x,X;\\theta_T)U_t$, is what ties the after-kernel's predictive power to the trained network, and it motivates dropping the DNN output entirely.","core_discovery":"The paper's central claim is that the after-training neural tangent kernel $K(x,\\tilde x;\\theta_T)=\\nabla_\\theta f(x;\\theta_T)^\\top\\nabla_\\theta f(\\tilde x;\\theta_T)$ is the object that carries generalization in deep learning, and that it can be trained directly. Proposition 1 shows that once the NTK stabilizes, late-time gradient descent on the DNN is equivalent to kernel ridge regression on the residuals, so the network output is redundant once the after-kernel is known. The paper therefore proposes NTK-KARE, which trains the kernel parameters by minimizing KARE, a consistent estimator of the expected risk of kernel ridge regression, rather than by minimizing empirical risk of the network. Empirically, NTK-KARE matches or outperforms the trained DNN and the after-training NTK across synthetic data, MNIST, Higgs, and the UCI benchmark, and the authors argue this is a form of over-parameterized feature learning.","pith_inferences":["If the result extends to larger architectures and harder benchmarks, part of what is credited to end-to-end representation learning may be reproducible by direct kernel optimization, which would make generalization easier to audit.","A useful stress test is to record true test error alongside KARE during kernel training: if the two ever move apart sharply, the objective is not doing what the paper assumes.","KARE could be applied to kernel selection over random feature or deep feature maps, where parameter count is high but gradient flow is cheaper.","One open scaling question the paper leaves implicit: whether NTK-KARE's advantage grows or shrinks with width and depth beyond the tested ranges (up to depth 5 and width 512 on UCI)."],"forward_implications":["The trained DNN's output function can be discarded: prediction can be carried by the after-training NTK or the KARE-trained version of it.","The after-training NTK is improvable by direct optimization, so implicit kernel learning during empirical risk minimization is not the only route to good generalization.","KARE-based kernel training applies beyond NTK to any heavily parameterized kernel, including deep kernel learning models.","Kernel methods with many learnable parameters are feature learners, not just fixed-feature regressors.","In settings where the after-NTK already matches the DNN, explicit kernel training can convert that tie into a consistent advantage."],"supporting_citations":[{"why":"Defines the neural tangent kernel $K(x,\\tilde x;\\theta)=\\nabla_\\theta f(x)^\\top\\nabla_\\theta f(\\tilde x)$ and its role in gradient descent dynamics.","marker":"Jacot et al. (2018)"},{"why":"Introduces KARE, the risk estimator the paper uses as training objective.","marker":"Jacot et al. (2020)"},{"why":"Documents that after-training NTK predicts as well as or better than the DNN, motivating the claim that the kernel is the main event.","marker":"Fort et al. (2020)"},{"why":"Defines feature complexity, supplies the RFM baseline, and sets the UCI evaluation protocol that NTK-KARE is compared against.","marker":"Radhakrishnan et al. (2024)"},{"why":"Provides the 121-dataset UCI classification benchmark used for the final comparison.","marker":"Fernàndez-Delgado et al. (2014)"},{"why":"Supports consistency of KARE as an approximation of expected risk for kernel ridge regression.","marker":"Misiakiewicz and Saeed (2024)"},{"why":"Adds evidence on after-training NTK performance and the feature versus lazy training distinction.","marker":"Geiger et al. (2020)"},{"why":"Shows silent alignment, i.e., that networks train their kernels toward useful features, supporting implicit kernel learning.","marker":"Atanasov et al. (2021)"},{"why":"Documents limitations of the fixed NTK and the evolution of the data-dependent NTK, motivating explicit kernel training.","marker":"Vyas et al. (2022)"},{"why":"Provides adaptive kernel predictors from feature-learning limits as another after-NTK comparison point.","marker":"Lauditi et al. (2025)"}],"fun_headline_variants":["Train the kernel, not the network: NTK beats DNN","Kernel training with KARE outperforms the original DNN","Explicit NTK training beats end-to-end DNNs","Optimize the after-kernel directly: win with KARE","Forget the network: train its NTK with KARE"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method requires that KARE remains a faithful estimate of test error while the kernel parameters are being optimized, not only for a fixed kernel; the paper does not prove this for finite-width NTKs.","fun_headline_variants_meta":{"raw":{"variants":["Train the kernel, not the network: NTK beats DNN","Kernel training with KARE outperforms the original DNN","Explicit NTK training beats end-to-end DNNs","Optimize the after-kernel directly: win with KARE","Forget the network: train its NTK with KARE"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000551,"raw_usage":{"total_tokens":2609,"prompt_tokens":907,"completion_tokens":1702,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":523,"completion_tokens_details":{"reasoning_tokens":1615}},"tokens_in":523,"tokens_out":1702,"duration_ms":9797,"temperature":1.0,"reasoning_tokens":1615,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:54:21.379693+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Track the KARE objective and fresh-sample test error jointly during kernel training on a fixed dataset; if there is a step where KARE decreases but held-out error increases substantially, then minimizing KARE is not minimizing generalization and the claimed mechanism fails.","supporting_citations":[{"cited_title":"Ker- nel alignment risk estimator: Risk prediction from training data,","cited_arxiv_id":null,"evidence_quote":"Introduces KARE, the risk estimator the paper uses as training objective."}],"review_version":1}