Pith. sign in

REVIEW 4 cited by

Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessians

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 1901.08244 v1 pith:4TJ4X5XL submitted 2019-01-24 cs.LG stat.ML

Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessians

Vardan Papyan This is my paper
classification cs.LG stat.ML
keywords outliersstructurehessianspectrumcovariancegradientsmeansused
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
read the original abstract

We consider deep classifying neural networks. We expose a structure in the derivative of the logits with respect to the parameters of the model, which is used to explain the existence of outliers in the spectrum of the Hessian. Previous works decomposed the Hessian into two components, attributing the outliers to one of them, the so-called Covariance of gradients. We show this term is not a Covariance but a second moment matrix, i.e., it is influenced by means of gradients. These means possess an additive two-way structure that is the source of the outliers in the spectrum. This structure can be used to approximate the principal subspace of the Hessian using certain "averaging" operations, avoiding the need for high-dimensional eigenanalysis. We corroborate this claim across different datasets, architectures and sample sizes.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. A Defense of the Quadratic Model

    cs.LG 2026-07 conditional novelty 7.0

    Local Taylor-expanded quadratic models reproduce a 150M-parameter LLM's validation loss for up to 10% of training late in the run, and LLM pretraining operates within a factor of 2 of a stochastic or deterministic edg...

  2. The Implicit Bias of Depth: From Neural Collapse to Softmax Codes

    cs.LG 2026-05 unverdicted novelty 7.0

    Depth induces an implicit low-rank bias in deep unconstrained feature models trained with unregularized multiclass cross-entropy, promoting softmax codes over neural collapse via more efficient norm propagation.

  3. First-Order Predictable but Pairwise Fragile: Local Task Adaptation in Trained Transformers

    cs.LG 2026-07 conditional novelty 5.0

    Single task-vector perturbations around a multitask LoRA point are first-order linear to 1e-2, but pairwise update-order sensitivity is set by a per-pair Lie bracket and can appear inside that window.

  4. Hessian based analysis of SGD for Deep Nets: Dynamics and Generalization

    cs.LG 2019-07 unverdicted novelty 4.0

    Provides Hessian-based theoretical characterizations of SGD dynamics and a scale-invariant generalization bound for deep nets, backed by experiments on synthetic data, MNIST, and CIFAR-10.