{"id":"278f101b-6388-478d-8c8c-89411f6c6ecb","arxiv_id":"2606.00542","paper_version":2,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Different Bregman divergences distribute unavoidable Kronecker approximation error differently across covariance spectrum, with top eigenspace better aligned to Hessian, motivating a subspace-aware optimizer using eigenvalue preconditioning on top subspace and adaptive isotropic acceleration on bott","lead":"The paper analyzes how Frobenius, von Neumann, and LogDet Bregman divergences distribute Kronecker approximation error differently across the spectrum of gradient covariance matrices. A smart generalist might read it to understand potential ways to improve second-order optimizers used in training large machine learning models.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Empirical alignment claim between top covariance eigenspace and Hessian is invoked to motivate subspace split but lacks quantified support in the given abstract","rationale":"The reader's weakest_assumption directly identifies the empirical alignment observation as the load-bearing step for the optimizer proposal; the theoretical spectral-distribution claims cannot be evaluated from the abstract alone, so the same limitation applies. No stronger internal inconsistency is visible in the stated claims.","tokens_in":1714,"tokens_out":350,"duration_ms":8078,"concrete_test":"On the same models and data used for the paper's experiments, compute the average principal angles (or cosine of leading eigenvectors) between the top-10% covariance eigenspace and the corresponding Hessian eigenspace, versus the bottom-10% subspace; repeat over 3 independent runs with different random seeds. If the top-subspace alignment advantage is <0.15 on average or fails a paired t-test at p<0.05, the subspace-split motivation is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central theoretical claim concerns how Frobenius/von Neumann/LogDet divergences weight the Kronecker approximation residual differently across the covariance spectrum. The proposed optimizer, however, rests on the additional claim that the top eigenspace is substantially better aligned with the Hessian while the tail is noisier; this observation is stated without reference to specific models, datasets, quantitative metrics (e.g., principal-angle or trace overlap), or controls for finite-sample effects. If the alignment difference is small, statistically insignificant, or architecture-dependent, the justification for eigenvalue-based preconditioning in the top block and isotropic acceleration in the bottom block does not follow from the spectral analysis.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper analyzes the effect of Bregman matrix divergences (Frobenius, von Neumann, LogDet) on how Kronecker approximation error is distributed across the spectrum of a covariance matrix that is not exactly Kronecker-factored. It shows that the resulting Kronecker factors are governed by divergence-weighted residuals rather than raw error. Empirically, the top covariance eigenspace is observed to align better with the Hessian than the noisy tail; this motivates a subspace-aware optimizer that applies eigenvalue-based preconditioning to the top block and adaptive isotropic acceleration to the bottom block.","tokens_in":1864,"tokens_out":542,"duration_ms":16812,"significance":"If the spectral weighting analysis is correct, the work supplies a principled explanation for why different divergences produce distinct preconditioners and could inform divergence selection in practice. The proposed subspace split is a direct consequence of the claimed alignment difference; its practical value hinges on whether that difference is robustly quantified and whether the resulting optimizer yields measurable gains over standard Shampoo variants.","major_comments":[{"comment":"Abstract (final paragraph) and the empirical motivation section: the claim that 'the top covariance eigenspace is substantially better aligned with the Hessian matrix, while the tail spectrum is much noisier and unreliable' is load-bearing for the subspace split in the proposed optimizer, yet the manuscript provides no quantitative metrics (principal angles, trace overlap, cosine similarity), no specific models or datasets, and no controls for finite-sample bias or architecture dependence. Without these, the justification for eigenvalue preconditioning in the top block versus isotropic acceleration in the tail does not follow from the spectral analysis.","section":"Abstract (final paragraph)"},{"comment":"§3 (theoretical development of weighted residuals): the statement that Kronecker factors are 'governed by divergence-weighted residuals rather than the raw approximation error' is central, but the derivation must explicitly show how the weighting arises from the Bregman projection for each divergence and confirm that the weighting is not an artifact of the particular matrix factorization chosen.","section":"§3"}],"minor_comments":[{"comment":"Notation for the three divergences and their associated residuals should be introduced once with a single table or equation block rather than redefined inline in multiple sections.","section":"§2"},{"comment":"The experimental section should report the precise hyper-parameter settings used for the baseline Shampoo variants and the new subspace-aware method so that the claimed improvements can be reproduced.","section":"§5"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive comments. We address each major point below and indicate where revisions will be made to strengthen the manuscript.","responses":[{"response":"We agree that the alignment claim requires quantitative support to justify the subspace split. In the revised manuscript we add a dedicated empirical subsection reporting principal angles, cosine similarities, and trace overlaps between the top covariance eigenvectors and the Hessian. Results are shown for ResNet-18 trained on CIFAR-10 and a 6-layer transformer on WikiText-2, with controls varying batch size to assess finite-sample effects and architecture dependence. These additions directly substantiate the motivation for eigenvalue preconditioning on the top block and isotropic acceleration on the tail.","revision_made":"yes","referee_comment":"[Abstract (final paragraph)] Abstract (final paragraph) and the empirical motivation section: the claim that 'the top covariance eigenspace is substantially better aligned with the Hessian matrix, while the tail spectrum is much noisier and unreliable' is load-bearing for the subspace split in the proposed optimizer, yet the manuscript provides no quantitative metrics (principal angles, trace overlap, cosine similarity), no specific models or datasets, and no controls for finite-sample bias or architecture dependence. Without these, the justification for eigenvalue preconditioning in the top block versus isotropic acceleration in the tail does not follow from the spectral analysis."},{"response":"We accept that the weighting derivation needs to be shown explicitly. The revised Section 3 now contains the full stationarity conditions for the Bregman projections under Frobenius, von Neumann, and LogDet divergences, deriving the residual weighting factors directly from each optimality equation. We further add a short argument establishing that the weighting depends only on the chosen divergence and the Kronecker constraint, independent of any particular factorization algorithm, by repeating the projection under an alternative parameterization of the factors.","revision_made":"yes","referee_comment":"[§3] §3 (theoretical development of weighted residuals): the statement that Kronecker factors are 'governed by divergence-weighted residuals rather than the raw approximation error' is central, but the derivation must explicitly show how the weighting arises from the Bregman projection for each divergence and confirm that the weighting is not an artifact of the particular matrix factorization chosen."}],"tokens_in":1422,"tokens_out":488,"duration_ms":83105,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main thing to know is that the authors extend the 2026 analysis by showing how Frobenius, von Neumann, and LogDet divergences spread the unavoidable Kronecker residual differently across the covariance spectrum, with the factors themselves driven by divergence-weighted residuals rather than raw error. They then use an empirical observation about top-eigenspace alignment with the Hessian to motivate a subspace-aware optimizer that preconditions the top block with eigenvalues and uses isotropic acceleration below.\n\nWhat is actually new is the explicit spectral distribution properties for each divergence and the concrete subspace split in the optimizer design. The weighted-residual explanation is a useful clarification that follows from the prior projection view.\n\nThe theoretical distinction between the divergences looks like the stronger part if the derivations hold. It gives a cleaner account of why the resulting preconditioners behave differently.\n\nThe softer spot is the justification for the split itself. The abstract states that the top covariance eigenspace aligns substantially better with the Hessian while the tail is noisier, but supplies no metrics, principal angles, trace overlaps, model details, or controls for sample size. If that difference turns out small or architecture-dependent, the eigenvalue-versus-isotropic choice does not follow directly from the spectral analysis. The lack of any derivation sketch or experimental protocol in the abstract also leaves soundness hard to assess from the summary alone.\n\nThis is for people already working on Kronecker-factored second-order methods in deep learning. A reader who cares about preconditioner design would get value from the spectral perspective and the proposed variant. It is worth sending to peer review so the derivations and the alignment evidence can be checked properly.","headline":"The paper gives a spectral breakdown of how three Bregman divergences weight Kronecker approximation error and proposes a subspace split, but the alignment claim that motivates the split lacks visible numbers or controls.","tokens_in":2346,"tokens_out":409,"would_cite":false,"duration_ms":17304,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Different Bregman divergences distribute Kronecker approximation errors differently across the covariance spectrum.","keywords":["Bregman divergence","Kronecker factorization","covariance approximation","second-order optimization","Shampoo optimizer","preconditioner","spectral analysis","subspace optimization"],"falsifier":"Compute the principal angles or overlap between the top covariance eigenvectors and the corresponding Hessian eigenvectors on a trained model; if alignment is no stronger in the top block than in the tail, the justification for the proposed subspace split disappears.","tokens_in":2607,"feed_emoji":"📊","tokens_out":657,"duration_ms":19823,"temperature":0.7,"pith_summary":"Shampoo-style optimizers approximate gradient covariances with Kronecker factors. When the covariance is not exactly Kronecker-structured, approximation error must go somewhere. The paper shows that the Frobenius, von Neumann, and LogDet divergences each spread this error across the eigenvalue spectrum in distinct patterns. The Kronecker factors themselves are produced by residuals that are weighted according to the chosen divergence. These spectral preferences, together with the observation that the leading eigenspace aligns better with the Hessian than the noisy tail, motivate a subspace-aware optimizer that applies eigenvalue preconditioning only on top and isotropic acceleration below.","feed_headline":"Divergence choice controls error spread in Kronecker covariance approximations","feed_subtitle":"Frobenius, von Neumann and LogDet divergences weight residuals differently, producing distinct spectral preferences in the factors.","key_machinery":"Kronecker projection under Bregman matrix divergences, with factors realized through divergence-weighted residuals.","core_discovery":"When a covariance matrix is projected onto Kronecker structure under a Bregman divergence, the unavoidable approximation error is allocated differently across the spectrum by the Frobenius, von Neumann, or LogDet divergence. The resulting Kronecker factors are governed by divergence-weighted residuals rather than the raw error. This produces preconditioners with different emphases on reliable versus unreliable parts of the spectrum. The top covariance eigenspace aligns substantially better with the Hessian while the tail is noisier, which directly motivates splitting the space into an eigenvalue-preconditioned top subspace and an isotropically accelerated bottom subspace.","pith_inferences":["The weighted-residual perspective may extend to other structured low-rank or factored approximations beyond Kronecker products.","A per-subspace divergence choice could be tested to match the reliability profile of each block.","The alignment observation could be checked on additional architectures to see whether the subspace split generalizes."],"forward_implications":["The approximation error is allocated according to divergence-specific weighting rather than uniformly.","Each divergence produces Kronecker factors whose spectral behavior follows from its own weighted residuals.","Eigenvalue-based preconditioning is reliable only in the top subspace where alignment with the Hessian holds.","An adaptive isotropic acceleration term is appropriate for the tail where eigenvalue estimates are unreliable."],"fun_headline_variants":["Divergence dictates spectral error split in Kronecker factors","Residual weights shape distinct Kronecker preconditioners","Hessian alignment peaks in top covariance eigenspace","Split subspaces enable eigenvalue top and isotropic bottom steps"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The top eigenvectors of the covariance matrix align substantially better with the Hessian than those in the tail of the spectrum.","fun_headline_variants_meta":{"raw":{"variants":["Divergence dictates spectral error split in Kronecker factors","Residual weights shape distinct Kronecker preconditioners","Hessian alignment peaks in top covariance eigenspace","Split subspaces enable eigenvalue top and isotropic bottom steps"]},"model":"grok-4.3","cost_usd":0.003265,"raw_usage":{"total_tokens":1755,"prompt_tokens":683,"num_sources_used":0,"completion_tokens":60,"cost_in_usd_ticks":32649500,"prompt_tokens_details":{"text_tokens":683,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1012,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":683,"tokens_out":60,"duration_ms":7313,"temperature":1.0,"reasoning_tokens":1012,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-28T18:48:53.252555+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Compute the principal angles or overlap between the top covariance eigenvectors and the corresponding Hessian eigenvectors on a trained model; if alignment is no stronger in the top block than in the tail, the justification for the proposed subspace split disappears.","supporting_citations":[],"review_version":1}