{"id":"d952cbbe-e180-4c76-ad1b-38b1b894f35a","arxiv_id":"2412.11521","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A neural kernel theory shows that deep networks trained in the NTK regime generalize on partially sampled symmetry orbits only when class separation and orbit density are high enough, so they cannot genuinely learn new symmetries from data.","lead":"This paper derives a formula for how well kernel-viewed deep networks generalize on datasets with partially observed symmetries. It shows conventional networks only generalize when the data's local structure dominates, implying they lack an inbuilt mechanism to learn symmetries from examples.","discovery_kind":"first_principles","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The spectral error formula (Eq. 1) is exact only for circulant Gram matrices on interleaved orbits, but Section 4.4 manufactures that structure on real MNIST pairs by diagonal-wise averaging that the authors admit is unjustified, leaving the rotated-MNIST validation on an untested approximation.","rationale":"The reader's weakest assumption identifies exactly the same point: the rotated-MNIST application depends on a post-hoc symmetrization of the Gram matrix that is not justified a priori. I agree that this is the most load-bearing concern. The theoretical contributions of the paper—the spectral error for exactly circulant kernels and the finite-group generalization—are transparent and independently verifiable, and the RBF circular dataset case is fully rigorous. The problem is the bridge from that exact setting to real MNIST orbits. Section 4.4 explicitly states that the NTK matrix is not circulant for two non-interleaved MNIST orbits, and the paper's only response is to average diagonals and assert empirical faithfulness. Because the central negative claim about conventional deep networks is a universal claim, and because the rotated-MNIST experiment is the primary empirical support for that claim, the circularization step is where the argument is least secure. The proposed test isolates exactly whether Ktilde is a faithful surrogate for K in the leave-one-out regression problem; this is a single, cheap computational check. If Ktilde and K give materially different predictions, the spectral error formula is not describing the actual kernel regression being performed, and the conclusions about rotated-MNIST would need to be downgraded from validated results to a heuristic observation. If the test passes, the reader's CONDITIONAL verdict is supported and no further adjustment is needed. I therefore leave the verdict unchanged: the concern is real and addressable, and the proposed test decides it.","tokens_in":65495,"tokens_out":3888,"duration_ms":43556,"concrete_test":"On the same MNIST orbit-pair datasets used in Fig. 4 (Nrot = 8), compute leave-one-out NTK regression predictions for the missing point using the true non-circulant Gram matrix K and using the diagonal-wise circularized matrix Ktilde, and record the per-pair predicted label and error. If the circularized predictions diverge materially from the true ones (e.g., sign flips or mean absolute error shift of more than 10%), then Eq. 1 is computing the error of a different regression problem and the agreement in Fig. 4B is not evidence for the paper's conclusion about rotated-MNIST.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is in Section 4.4. Real MNIST orbits are not geometrically interleaved: the paper notes xA_i · xB_i != xA_i · xB_{i+1} and unequal intra-orbit spacings, so the NTK Gram matrix K is not circulant. The authors then define Ktilde by diagonal-wise averaging and apply Eq. 1, while saying the circularization 'is not justified a priori'. Eq. 1 is derived under the assumption of a circulant Gram matrix on interleaved classes (Prop. 4, App. B); for non-circulant K it is not a theorem. The empirical agreement in Figs. 4B and 5A-C is a correlation across many random pairs, not a demonstration that Ktilde preserves the regression problem. The authors themselves report that circularization 'may reshape its structure too drastically' for GAP-CNN rotation orbits (Fig. 5E and Section 4.6.3). The multi-seed extension in Section 4.5 adds an unproven pairwise-linear averaging of spectral errors, and the multi-class comparison uses a one-vs-many heuristic; so even if the circularized formula were exact for pairs, the headline conclusion about conventional deep networks on rotated-MNIST would still rest on an additional approximation. This is the single point where the central claim could fail: if Ktilde predicts differently from the true K, the paper's only nontrivial validation of the negative result is removed.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies supervised classification on datasets with cyclic-group symmetry where some classes are fully observed over a group orbit while others are only partially observed. In the infinite-width NTK limit, the authors treat training as kernel regression and derive the spectral error formula ε_s = λ_N^{-1}/⟨λ^{-1}⟩ (Eq. 1) for a two-class interleaved dataset with a circulant Gram matrix, interpreting the numerator and denominator as class separation and orbit density in kernel space. They extend the formula to general finite groups (Theorem 15), apply it to rotated-MNIST via a diagonal-wise circularization approximation, extend it to multi-seed and multi-class settings through pairwise averaging and a one-vs-many heuristic, and analyze equivariant CNN architectures. The empirical comparisons across MLP, CNN, and ViT lead to the conclusion that conventional deep networks lack a mechanism to learn symmetries that are not embedded in their architecture a priori.","tokens_in":65846,"tokens_out":7541,"duration_ms":71748,"significance":"The exact part of the paper is elegant and reproducible: Eq. 1 is derived rather than fitted, the RBF circular-dataset experiments in Sec. 4.3 verify it exactly, and the non-abelian extension in App. C broadens the reach of the theory. The broad qualitative match between the spectral theory and finite-width network behavior on rotated-MNIST, together with the released code, makes this a useful contribution to the neural-kernel literature. However, the rotated-MNIST and multi-class validations, which carry the headline negative conclusion, rest on approximations that are acknowledged but not quantified. If those approximations are shown to be faithful, the paper provides an interpretable criterion for symmetry generalization; without such a check, the central claim for real data remains a plausible conjecture rather than a demonstrated consequence of the exact theory.","major_comments":[{"comment":"The application of Eq. 1 to rotated-MNIST replaces the true non-circulant Gram matrix K_NTK with the diagonal-wise averaged matrix \\tilde{K}_NTK, a procedure the authors state 'is not justified a priori'. Because Prop. 4 and App. B prove Eq. 1 only for circulant Gram matrices on exactly interleaved orbits, the agreement in Figs. 4B and 5A-C is a correlation between the spectral error of a modified regression problem and the exact error of the original problem, not a test of Eq. 1 on the actual Gram matrix. This is load-bearing, since the same approximation enters the multi-seed and multi-class validation (Fig. 6) and therefore the paper's headline conclusion about rotated-MNIST. Please quantify the approximation error (e.g., the relative Frobenius norm ||K - \\tilde{K}|| / ||K|| and the L2 difference between the leave-one-out predictors under K and \\tilde{K}) and either prove a bound for dot-product kernels on non-interleaved orbits or provide a controlled dataset where the circulant assumption is satisfied exactly. The GAP-CNN discussion in Sec. 4.6.3, where the authors say the circularization 'may reshape its structure too drastically', makes clear that the approximation's fidelity is architecture-dependent and cannot be assumed from the MLP agreement alone.","section":"Sec. 4.4, 'Approximation: Circularity'"},{"comment":"The multi-seed extension replaces the exact two-orbit problem with an arithmetic average of spectral errors over all cross-class orbit pairs, introduced as a 'simplifying assumption'; the text immediately notes that the resulting error 'does not capture well the magnitude of the exact error anymore'. No derivation or error bound is given for this linear averaging, and the subsequent multi-class extension relies on a one-vs-many heuristic. These steps are needed to connect the exact theorem to the realistic rotated-MNIST experiments in Fig. 6, so the conclusion that finite-width networks display no symmetry-learning mechanism on this dataset is not a direct corollary of Eq. 1. Please either derive the averaging approximation, or demonstrate its validity by comparing the average of pairwise exact NTK errors to the exact multi-orbit NTK error on the same datasets, ideally for more than two classes.","section":"Sec. 4.5"},{"comment":"Propositions 11-14, which state that FC-CNN and GAP-CNN kernels are circulant or constant over translation and rotation orbits, are proved for the NNGP kernel, while the main-text experiments and spectral-error computations use the NTK; the authors write that 'essentially the same proofs should hold for the NTK' but do not provide them. Since Eq. 1 applies only to the exact Gram matrix, the empirical CNTK comparisons in Sec. 4.6.3 and App. I are only justified if the NTK Gram matrix inherits the circulant structure. Please supply the NTK versions of these proofs, or state explicitly which of the CNTK figures would be affected if the NTK matrix deviates from circulancy.","section":"Sec. 4.6 and App. E"}],"minor_comments":[{"comment":"The phrase 'anex-post circularization procedure' should read 'an ex-post circularization procedure'.","section":"Sec. 4.4, p. 15"},{"comment":"The references to 'Prop. 3', 'Prop. 4', 'Prop. 1', and 'Prop. 2' should point to Props. 11-14.","section":"Sec. 4.6.3"},{"comment":"The caption says 'the assumptions of the theory are too crude to capture empirical phenomenology', while the main text describes the agreement as 'worse, although still reasonable'; these two characterizations should be reconciled.","section":"Fig. 5E and Sec. 4.6.3"},{"comment":"The label formula µ_i = µ ω^{iN} + (µ0 − µ)δ_i0 implicitly assumes the missing point is at index 0; this is stated nearby but could be made more prominent for readers applying the formula to other missing points.","section":"App. B, Eq. (6)"},{"comment":"The symmetrized NTK error averages over removing a point from either orbit, but the individual class-specific errors are not reported; reporting them would help assess whether the symmetrization hides a systematic asymmetry between the two orbits.","section":"Sec. 4.4, 'Adjustment: Symmetrization'"}],"recommendation":"major_revision","confidential_remarks":"The exact two-orbit result is solid and the broad empirical failure of finite-width networks on rotated-MNIST (Fig. 1) is an important observation. My main concern is that the spectral theory's application to real data is validated only through approximations whose error is not quantified; I would ask the editor to request the numerical checks described in the major comments before acceptance. I do not see a novelty disclosure issue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper deserves a serious referee. The genuinely new result is Eq. 1: for kernel regression on two interleaved cyclic orbits with a circulant Gram matrix, the error on a missing point is λ_N^{-1}/⟨λ^{-1}⟩. The derivation in App. B is clean and does not rely on fitting anything. The finite-group generalization (Theorem 15) is also new and looks correct, and the geometric reading in terms of class separation and orbit density is useful. The paper does well at showing that the spectral quantity tracks exact NTK error across MLP and CNN architectures, and that finite-width trained networks behave the same way. Credit where due: the theory is derived, not reverse-engineered, and the code is public.\n\nThe soft spots are real but not fatal. The rotated-MNIST validation depends on a diagonal-wise circularization of the Gram matrix, and the authors themselves say it is not justified a priori. The stress-test note is right that Eq. 1 is a theorem for circulant matrices; for non-circulant K it is an approximation. The empirical correlation is suggestive, not a proof that the symmetrized kernel preserves the regression problem. The multi-seed extension is a pairwise averaging heuristic, and the multi-class comparison uses a one-vs-many rule. None of this kills the theoretical contribution, but it does mean the headline claim—conventional deep networks lack a mechanism to learn symmetries—is broader than the NTK evidence supports. The authors mostly qualify this in the discussion, but the abstract draws the strong conclusion.\n\nI would send this to peer review. The right referee will push for a perturbation bound or direct test that the circularized Gram matrix is a faithful proxy, and for a sharper statement about which architectures the conclusion covers. The paper is also likely to be cited: the spectral error formula is a compact, usable result for the symmetry-learning literature.","headline":"A genuinely useful spectral error formula for kernel regression on cyclic datasets, with a real but addressable gap between the exact circulant theory and the rotated-MNIST validation.","tokens_in":66300,"tokens_out":1906,"would_cite":true,"duration_ms":19113,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Deep networks trained from data alone cannot learn symmetries their architecture lacks, a neural kernel theory shows.","keywords":["deep learning","symmetry learning","neural tangent kernel","kernel regression","spectral error","circulant matrices","equivariant networks","rotated-MNIST"],"falsifier":"Run exact NTK regression on rotated-MNIST orbit pairs without circularizing the Gram matrix and compare the prediction error to Eq. 1 per orbit pair; if the discrepancy grows with the degree to which the two orbits fail to be interleaved, for example when seeds lie near the rotation stabilizer, then the circularity approximation is load-bearing.","tokens_in":65297,"feed_emoji":"🔄","tokens_out":4256,"duration_ms":41035,"temperature":0.7,"pith_summary":"The paper tries to establish a precise condition under which conventional deep networks generalize a partially observed symmetry to unseen class poses: they do so only when a simple spectral error, the ratio of the inverse highest-frequency eigenvalue to the average inverse spectrum of the neural kernel, is small. That ratio is small exactly when classes are well separated in kernel space and orbits are densely sampled, so generalization is driven by local data geometry rather than by any learned non-local group structure. Because standard architectures have no mechanism for the kernel to adapt to the dataset's symmetry, the paper concludes that conventional networks cannot learn symmetries that were not embedded a priori. This matters because it turns an observed empirical failure, networks trained on rotated-MNIST with one pose missing per class, into a quantitative prediction about when the failure will and will not occur.","feed_headline":"Standard deep nets cannot learn unseen symmetries, kernel theory shows","feed_subtitle":"A spectral error formula ties symmetry generalization to class separation and orbit density, not to symmetry discovery.","key_machinery":"The central object is the spectral error $\\epsilon_s = \\lambda_N^{-1}/\\langle\\lambda^{-1}\\rangle$ of Proposition 4, derived for a circulant Gram matrix over an interleaved two-class cyclic-orbit dataset. Because a circulant matrix is diagonalized by the discrete Fourier transform, the difficulty of kernel regression on a missing orbit point reduces to a ratio of inverse frequency powers: the contribution of the class-alternating highest frequency against all frequencies. The paper uses this identity as a lens across RBF kernels, dot-product MLP kernels, and convolutional kernels, and extends it in Theorem 15 to arbitrary finite groups via non-commutative Fourier analysis.","core_discovery":"On a dataset of two interleaved cyclic-group orbits with alternating labels, kernel regression on a held-out orbit point has error $\\epsilon_s = \\lambda_N^{-1}/\\langle\\lambda^{-1}\\rangle$, where $\\lambda_N$ is the eigenvalue of the highest Fourier frequency of the circulant Gram matrix and $\\langle\\lambda^{-1}\\rangle$ is the average inverse eigenvalue over all frequencies. The paper shows this formula accounts for the generalization of MLPs, CNNs, and a ViT, both in the infinite-width NTK limit and for finite-width trained networks on rotated-MNIST, and that it recovers exact invariance only when the architecture is fully equivariant to the data symmetry, such as a translation-invariant global-average-pooling CNN on translated images. The central conclusion is that conventional deep networks lack a mechanism to learn symmetries that are not already built into their kernel or architecture, and that apparent symmetry learning is actually successful local interpolation in kernel space.","pith_inferences":["Editorial inference: a testable extension is to measure the spectral error directly on non-circularized data, because if real image orbits violate the interleaving assumption, the theory's quantitative predictions should degrade in a measurable way.","Editorial inference: augmentation and self-supervised invariance losses effectively raise orbit density or class separation in kernel space, so comparing the spectral ratios they induce could predict which pretraining strategy will transfer a given symmetry.","Editorial inference: since the frozen-kernel premise is essential to the argument, genuine symmetry learning should be sought in feature-learning regimes where the kernel changes during training; the paper's conclusion does not rule out such regimes, but leaves them unexplored."],"forward_implications":["Increasing the number of sampled angles in a class orbit monotonically lowers the predicted error, so denser symmetry sampling produces gradual improvement without a phase transition or grokking.","Increasing class separation in kernel space increases the highest-frequency eigenvalue and lowers the error; separable classes need less orbit density to generalize.","Equivariant architectures only guarantee perfect generalization when their built-in invariance matches the data symmetry; a global-average-pooling CNN gives zero spectral error on translations but not on rotations.","The same spectral error formula applies to any finite group, with representation-theoretic Fourier coefficients replacing scalar DFT coefficients, so the conclusion is not an artifact of cyclic rotations."],"supporting_citations":[{"why":"Establishes the neural tangent kernel limit in which training wide networks is kernel regression, the foundation of the spectral error derivation.","marker":"(Jacot et al., 2018)"},{"why":"Provides the Gaussian-process view of infinite-width networks and the dot-product kernel property used to show MLP kernels are circulant over orbits.","marker":"(Neal, 1996)"},{"why":"Relates generalization to kernel spectra; the paper extends this spectral lens to the symmetry-specific circulant case.","marker":"(Bordelon et al., 2020; Canatar et al., 2021)"},{"why":"Gives the exact convolutional kernel computation used to prove convolutional Gram matrices are circulant over translation and rotation orbits.","marker":"(Arora et al., 2019)"},{"why":"Defines group-equivariant networks, the baseline that achieves guaranteed invariance when the architecture matches the data symmetry.","marker":"(Cohen and Welling, 2016)"},{"why":"Supplies MNIST, the base dataset for the rotated-MNIST and translated-MNIST experiments.","marker":"(LeCun et al., 1998)"}],"fun_headline_variants":["Deep nets can't learn unseen symmetries—only interpolate locally","Symmetry generalization hinges on orbit density, not discovery","Kernel theory: No new symmetries without built-in equivariance","Standard architectures lack symmetry-learning mechanism, theory confirms","Cyclic orbits reveal why deep nets fail at symmetry generalization"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The exact spectral error formula requires the Gram matrix to be circulant over perfectly interleaved orbits; for real MNIST orbit pairs the paper obtains this by diagonal-wise averaging, an approximation the authors state is not justified a priori, so the rotated-MNIST agreement depends on that post-hoc symmetrization being faithful.","fun_headline_variants_meta":{"raw":{"variants":["Deep nets can't learn unseen symmetries—only interpolate locally","Symmetry generalization hinges on orbit density, not discovery","Kernel theory: No new symmetries without built-in equivariance","Standard architectures lack symmetry-learning mechanism, theory confirms","Cyclic orbits reveal why deep nets fail at symmetry generalization"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000296,"raw_usage":{"total_tokens":1772,"prompt_tokens":1050,"completion_tokens":722,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":666,"completion_tokens_details":{"reasoning_tokens":640}},"tokens_in":666,"tokens_out":722,"duration_ms":7339,"temperature":1.0,"reasoning_tokens":640,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:50:20.942228+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run exact NTK regression on rotated-MNIST orbit pairs without circularizing the Gram matrix and compare the prediction error to Eq. 1 per orbit pair; if the discrepancy grows with the degree to which the two orbits fail to be interleaved, for example when seeds lie near the rotation stabilizer, then the circularity approximation is load-bearing.","supporting_citations":[{"cited_title":"Neural tangent kernel: Convergence and generalization in neural networks","cited_arxiv_id":null,"evidence_quote":"Establishes the neural tangent kernel limit in which training wide networks is kernel regression, the foundation of the spectral error derivation."}],"review_version":1}