Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Eigenspectrum Analysis of Neural Networks without Aspect Ratio Bias

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that the aspect ratio of a neural network weight matrix systematically skews the heavy-tail spectral estimates used to judge layer training quality, and that replacing the full-matrix spectrum with the averaged spectrum…

desk verdict Aspect-ratio bias in HT-SR is real, and FARMS mostly fixes it; the paper overclaims uniformity and leaves the preservation of localized heavy-tail spikes unproven. read the letter →

arxiv 2506.06280 v2 pith:6TEDC3ST submitted 2025-06-06 cs.LG cs.AI

Yuanzhe Hu , Kinshuk Goel , Vlad Killiakov , Yaoqing Yang This is my paper
classification cs.LGcs.AI MSC 60B2068T07
keywords heavy-tailedself-regularizationempiricalspectraldensityaspectratiobiasmatrixsubsamplinglayer-wisehyperparameterassignmentLLMpruningrandomtheoryPLAlphaHill
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that a standard diagnostic for how well a neural network layer is trained, the heavy-tail exponent of the empirical spectral density (ESD) of its weight matrix, is systematically distorted by the matrix's aspect ratio, the ratio of its row count to its column count. Even for a well-trained layer, a large aspect ratio makes the spectrum look more concentrated and less heavy-tailed, so existing methods can mislabel the layer as undertrained and misassign layer-wise learning rates or pruning ratios. The authors propose FARMS (Fixed-Aspect-Ratio Matrix Subsampling), which replaces the single full-matrix spectrum with the average spectrum of many overlapping submatrices that all share one fixed aspect ratio. Across image classification, scientific machine learning fine-tuning, and LLM pruning, the paper reports that this correction improves model diagnosis and leads to better layer-wise hyperparameter assignment, including a 17.3% perplexity reduction on pruned LLaMA-7B in one setting.

What carries the argument

The central mechanism is the Marchenko-Pastur law, which fixes the limiting spectrum of a random matrix as a function of aspect ratio gamma = m/n; the bias enters because heavy-tail metrics like PL Alpha Hill are computed on spectra whose bulk shape already depends on gamma. FARMS counters this by a fixed-aspect-ratio matrix subsampling procedure: each weight matrix is partitioned by a sliding window into overlapping submatrices of constant shape Q = m'/n', the eigenvalue series of their correlation matrices are merged (equivalent to averaging the ESDs), and the heavy-tail exponent of the averaged ESD is reported. The rank-one, delocalized structure of heavy-tail 'feature spikes' is invoked to justify that subsampling preserves the signal being measured.

What would settle it

A synthetic experiment with a planted rank-one heavy-tail spike: fix the spike and vary only the matrix aspect ratio; if FARMS reports materially different PL Alpha values for matrices with identical planted signals, the bias is not removed. Conversely, construct a matrix whose heavy tail comes from a single localized block; if FARMS misses the spike that full-matrix analysis detects, the delocalization assumption fails.

Watch

Extended reading notes

Core claim

The central claim is that aspect ratio bias, not just training quality, alters the shape of empirical spectral densities of weight matrices, and that this bias is non-negligible in real networks. Concretely, the authors show that for random matrices the Marchenko-Pastur law makes the spectrum more concentrated as m/n moves away from 1, and they present evidence that real well-trained layers with tall-and-skinny weight matrices (e.g., the 512x100 final layer of ResNet-18) are consequently measured as less heavy-tailed than they are. FARMS corrects this by subsampling overlapping submatrices of a fixed aspect ratio Q from each weight matrix, averaging their ESDs, and computing the PL Alpha Hill metric on the averaged spectrum. The paper reports that this yields PL Alpha estimates that are stable across model widths and layers with different shapes, align with toy-model training quality in a controlled experiment, and improve downstream methods: TempBalance for layer-wise learning rate assignment, AlphaPruning for LLM layer-wise pruning, and TB Sigmoid for SciML fine-tuning.

Load-bearing premise

The method assumes that the heavy-tailed signal lives in approximately rank-one structures that extend across the whole matrix, so that random submatrix sampling captures it; if that fails, FARMS could discard the very signal it aims to measure.

Editorial extensions

If this is right

  • Replacing full-matrix spectral analysis with FARMS in TempBalance improves test accuracy on CIFAR-100 ResNet/VGG models and makes the method robust to tall-and-skinny layers even without layer-selection heuristics.
  • In AlphaPruning, FARMS lowers WikiText perplexity of pruned LLaMA models across sparsity levels and pruning methods, for example from 96.02 to 79.42 for LLaMA-7B with SparseGPT at 0.8 sparsity, while improving or matching zero-shot accuracy.
  • In SciML fine-tuning, FARMS reduces L2 relative error of DPOT models compared with TB Sigmoid at all data subsampling ratios tested.
  • PL Alpha Hill distributions across layers become more concentrated and lower on average when measured with FARMS, which prior work associates with more balanced training.
  • Ablations show that square submatrices (Q=1) and moderate window sizes give the best downstream performance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension would be to use FARMS as a preprocessing normalization for any spectral diagnostic that assumes comparable matrix shapes, potentially including activation or gradient spectra rather than weights alone.
  • The window size and number of sampling steps become new hyperparameters; the optimal choice likely reflects a bias-variance tradeoff between capturing delocalized spikes and averaging out noise.
  • The rank-one delocalization justification suggests a testable prediction: FARMS should fail to detect heavy-tail signals that are localized in a small block of the matrix, which would indicate when subsampling is unsafe.
  • If the aspect-ratio bias is as pervasive as claimed, previously reported cross-layer comparisons of heavy-tail exponents that were made without shape normalization may need to be revisited.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper identifies an aspect-ratio bias in heavy-tailed self-regularization (HT-SR) analyses of neural-network weight matrices: the empirical spectral density (ESD) of a weight matrix W depends on the aspect ratio m/n even in the uncorrelated Marchenko–Pastur limit, so comparing heavy-tailedness metrics such as PL Alpha Hill across layers with different shapes can misreport training quality. To fix this, the authors propose FARMS (Fixed-Aspect-Ratio Matrix Subsampling), which partitions each weight matrix into overlapping or sliding submatrices of a fixed aspect ratio Q, computes the average ESD of these submatrices, and then fits the power-law tail. The method is validated on randomly initialized ResNet/VGG models, a toy teacher-student alignment experiment, image classification with TempBalance, SciML fine-tuning with TB Sigmoid, and LLM pruning with AlphaPruning. The paper reports improved downstream performance in most evaluated settings, including a large perplexity reduction on LLaMA-7B/13B pruning, and provides an ablation study of the submatrix aspect ratio and sampling hyperparameters.

Significance. If the central claim holds, FARMS is a practically useful and simple preprocessing step for HT-SR-based layer diagnostics, because modern networks contain layers with very different aspect ratios (e.g., 512×100 final layers in ResNets) and the existing HT-SR pipeline ignores this shape dependence. The paper has several strengths: it demonstrates the bias with clean experiments on randomly initialized networks, provides an external toy experiment where FARMS correlates better with a ground-truth alignment metric, evaluates across multiple application domains (CV, SciML, LLM pruning), reports compute costs, and makes code available. The main weakness is that the theoretical justification for why subsampling preserves heavy-tail structure is heuristic and rests on an untested delocalization assumption about the rank-one feature-spike component. In addition, the paper's abstract claims uniform improvement, but the appendix contains counterexamples where FARMS is worse than the baseline. These issues affect the strength of the central claim but are addressable with additional analysis and more careful wording.

major comments (3)
  1. [Appendix E.3 and D.2] The load-bearing step for FARMS is the assertion in Appendix E.3 that feature spikes are approximately rank-one updates that 'cover the whole matrix with probability one,' so that random subsampling preserves the heavy-tail information. This is not established for localized rank-one components. For a rank-one component A = u v^T, a submatrix indexed by row set I and column set J has entries u_i v_j for i in I, j in J; if u or v is concentrated on a small coordinate subset, a fixed-size random or sliding submatrix will mostly see entries near zero and the corresponding spike will be attenuated or absent. The only mechanistic validation, the toy experiment in Appendix D.2, uses a single-index teacher with a dense planted signal, which is exactly the favorable regime and does not test localization. Please provide a synthetic experiment with a localized/sparse rank-one signal (or a real layer known to have concentrated structure) and show whether FARMS still recovers the tail, or explicitly state the delocalization condition as an assumption and discuss which practical layers satisfy it.
  2. [Abstract and Section 4.2, with Appendix C.3] The abstract and Section 4.2 claim that FARMS 'uniformly improves' eigenspectrum analysis/model performance. The appendix data contradict this. In Table 8, for OPT-1.3B with Wanda at 0.6 sparsity, FARMS reports perplexity 27.15 versus the AlphaPruning baseline of 27.09. In Table 9, for LLaMA-V3.1-8B with Wanda, FARMS reports 107.00 versus 105.64 at 0.7 sparsity and 3766.13 versus 3498.94 at 0.85 sparsity. These are not isolated noise-level differences; the last case is a large regression. Please revise the 'uniformly improves' claim to 'improves in most settings' and either explain these failures or analyze what property of these models causes FARMS to hurt.
  3. [Section 4.6, Table 6] The performance of FARMS is highly sensitive to its hyperparameters. Table 6 shows that for LLaMA-7B at 0.8 sparsity with SparseGPT, the perplexity ranges from 79.42±3.86 (window 2000, 10 sampling steps) to 99.23±3.53 (window 500, 15 steps). The worst FARMS configuration is worse than the AlphaPruning baseline of 96.02 reported in the text. The paper provides no principled selection rule for m', n', and the number of submatrices beyond heuristics and ablations on two models. Since these parameters are free, the central claim that FARMS improves layer-wise pruning should be conditioned on hyperparameter choice, and the authors should provide guidance or a robustness analysis across a wider range of settings.
minor comments (4)
  1. [Throughout] There are several typos and spacing inconsistencies: 'moodels' in Section 4.1, 'Leanring Rate' in the Figure 12 caption, 'F ARMS' instead of 'FARMS' in multiple appendix headings, and 'W anda' in Table 8. These should be corrected.
  2. [Section 3.3] The CNN subsampling description says the result is [C1/m'] × [C2/n'] × l' submatrices, but it is not stated whether floor/ceiling or padding is used when m' or n' do not divide C1 or C2. Please clarify the exact indexing and how leftover rows/columns are handled.
  3. [Appendix D.1] The sentence 'This is because the specific form of the rank-one update makes it cover the whole matrix with probability one' is presented as a justification, but the cited works (Ba et al. 2022; Wang et al. 2023) analyze the infinite-width limit with specific assumptions. Please quote the precise conditions (e.g., delocalization of the planted signal) under which this coverage property holds.
  4. [Section 4.3 and Table 3] Table 3 reports mean and standard deviation over five scaling ratios, but the text does not state how many random seeds are used for each scaling ratio. Appendix F says three seeds are used in the image-classification experiments, which should be stated in the main text for reproducibility.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: FARMS's claims are evaluated on external benchmarks, and the aspect-ratio normalization is a design property rather than a fitted input; a minor same-group citation supports but does not force the preservation argument.

full rationale

The derivation chain is not circular. FARMS fixes the submatrix aspect ratio Q by construction, so the resulting ESD and PL Alpha are invariant to the original matrix's aspect ratio; this is a design property, not a fitted parameter renamed as a prediction. The paper's positive claims are validated externally: WikiText perplexity, seven zero-shot tasks, CIFAR-100 accuracy, and PDEBench L2RE are not defined in terms of FARMS's HT metric, and the toy experiment in Appendix D.2 uses an independent alignment score. The baselines (TempBalance, AlphaPruning, TB Sigmoid) come from the same research group, creating a conflict of interest, but the comparison is a fair replacement of the HT-measurement subroutine and does not itself constitute the evidence for the theory. The only notable self-citation is Kothapalli et al. (2025), an overlapping-author paper cited in Appendix E.3 for the rank-one feature-spike argument that subsampling preserves heavy-tail information; however, that argument is also supported by the non-overlapping Wang et al. (2023) and by the toy experiment, so it is not the sole load-bearing step. Separately, the manuscript contains correctness concerns that are not circularity: the abstract's 'uniformly improves' is contradicted by Table 8 (OPT-1.3B Wanda 0.6: 27.15 vs 27.09) and Table 9 (LLaMA-V3.1-8B Wanda 0.7 and 0.85: 107.00 vs 105.64 and 3766.13 vs 3498.94), and Appendix E.3's assumption that heavy-tail spikes are approximately rank-one and delocalized is untested for localized feature directions. These are risks to the strength of the empirical claim, not evidence that the derivation reduces to its inputs.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on tunable parameters (Q, window size, sampling steps) and on the domain assumption that subsampling preserves heavy-tail features. No new theoretical entities are introduced.

free parameters (4)
  • Submatrix aspect ratio Q = 1.0 (optimal in ablation)
    Tunable parameter of FARMS; ablation in Table 5 shows Q=1.0 gives best test accuracy on ResNet18/VGG16.
  • Window size m' = varies by model; e.g., 2000 for LLaMA-7B
    Determines submatrix dimensions; selected based on validation perplexity (Table 6).
  • Sampling steps (number of submatrices) = e.g., 15 for LLaMA-7B
    Controls sliding window stride; selected based on validation perplexity (Table 6).
  • Hill estimator threshold k = not reported explicitly
    Inherited from prior HT-SR work; affects PL Alpha Hill value.
assumptions (4)
  • standard math Marchenko-Pastur law describes ESD of random matrices
    Used in Section 3.2 and Appendix B to show aspect ratio affects ESD shape.
  • domain assumption Heavy-tailedness of ESD correlates with layer training quality
    Core premise of HT-SR theory (Martin and Mahoney 2021); adopted without re-derivation.
  • domain assumption Heavy-tail features are approximately rank-one and delocalized, so subsampling preserves them
    Stated in Appendix E.3, citing Ba et al. 2022 and Wang et al. 2023.
  • ad hoc to paper Averaging ESDs of fixed-aspect-ratio submatrices removes aspect ratio bias without losing heavy-tail signal
    Central methodological claim of FARMS; not formally proven, only empirically supported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Eigenspectrum Analysis of Neural Networks without Aspect Ratio Bias." pith.science (2026). https://pith.science/paper/6TEDC3ST

@misc{pith2026250606280,
  author       = {Pith},
  title        = {Pith review of: Eigenspectrum Analysis of Neural Networks without Aspect Ratio Bias},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6TEDC3ST}},
  note         = {Machine review of arXiv:2506.06280}
}
read the original abstract

Diagnosing deep neural networks (DNNs) by analyzing the eigenspectrum of their weights has been an active area of research in recent years. One of the main approaches involves measuring the heavytailness of the empirical spectral densities (ESDs) of weight matrices. This analysis has been shown to provide insights to help diagnose whether a model is well-trained or undertrained, and has been used to guide training methods involving layer-wise hyperparameter assignment. In this paper, we address an often-overlooked challenge in estimating the heavytailness of these ESDs: the impact of the aspect ratio of weight matrices. We demonstrate that matrices of varying sizes (and aspect ratios) introduce a non-negligible bias in estimating the heavytailness of ESDs, leading to inaccurate model diagnosis and layer-wise hyperparameter assignment. To overcome this challenge, we propose FARMS (Fixed-Aspect-Ratio Matrix Subsampling), a method that normalizes the weight matrices by subsampling submatrices with a fixed aspect ratio. Instead of measuring the heavytailness of the original ESD, we measure the average ESD of these subsampled submatrices. We show that this method effectively mitigates the aspect ratio bias. We validate our approach across various optimization techniques and application domains that involve eigenspectrum analysis of weights, including image classification in computer vision (CV) models, scientific machine learning (SciML) model training, and large language model (LLM) pruning. Our results show that despite its simplicity, FARMS uniformly improves the accuracy of eigenspectrum analysis while enabling more effective layer-wise hyperparameter assignment. In one of the LLM pruning experiments, FARMS reduces the perplexity of the LLaMA-7B model by 17.3% when compared with state-of-the-art methods.

Figures

Figures reproduced from arXiv: 2506.06280 by the authors.

Figure 1
Figure 1. Comparing FARMS with common HT-SR methods in analyzing the weight matrices of a DNN. We use the weight matrix Wi of a well-trained layer with a large aspect ratio as an example (i.e., γ = m/n is much larger than 1). Due to the influence of the aspect ratio γ = m/n ≫ 1, the ESD is more concentrated and less HTed than other layers. As a result, previous methods for fitting a power-law distribution (shown as “PL Fittin… view at source ↗
Figure 2
Figure 2. ESD shapes can be biased by the aspect ratio. Here we visualize the eigenvalues of the symmetric [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Main Steps in FARMS. 3.3 Analyzing ESDs Using FARMS To mitigate the aspect ratio bias in analyzing ESDs, we use a block-wise sampling method when processing weight matrices. We partition each weight matrix into overlapping sub-matrices with a fixed aspect ratio (across all layers) following a predefined scheme described below. Consider the weight matrix Wi of the i-th layer, which has a shape of m × n. Without loss … view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Comparing our method to TempBalance at different layer usage settings in training ResNet and VGG series models on CIFAR100. Higher test accuracy values indicate better model performance. See [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Comparing the distribution of PL Alpha Hill of ResNet and VGG series models. The top blue￾shaded section in each subplot of the experiments (shown as ”CAL”) represents models trained without employing layer-wise optimization methods. The middle brown-shaded section in …
Figure 6
Figure 6. Figure 6: Comparing FARMS and previous HT-SR methods for measuring the randomly initialized ResNet 34 and VGG 19 weights. Figure 6a and Figure 6c show the PL Alpha Hill values for each layer in models (widen factor is 1.0) by using different methods. Figure 6b and Figure 6d show…
Figure 7
Figure 7. Figure 7: Comparing FARMS and the previous method for measuring the ESD of final layers in ResNet 34 and VGG 16 trained on CIFAR 100. B Marchenko–Pastur Distribution 0 2 4 Eigenvalue / n 0 2 Density m=4000, n=4000 Empirical Mar enko Pastur PDF 0 2 Eigenvalue / n 0 1 Density m=40…
Figure 8
Figure 8. Figure 8: The Marchenko-Pastur (MP) Law for different values of [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: Comparison of test accuracy across different architectures and learning rate scaling ranges. The [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: Comparing the distribution of PL Alpha Hill of DPOT-Tiny and DPOT-Small in different fine-tuning methods and data ratios. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_10.png]
Figure 11
Figure 11. Figure 11: Comparing the distribution of PL Alpha Hill of LLaMA-7B and LLaMA-13B in different layer￾wise strategies and sparsity ratios (shown as ”SR”). The pruning method is SparseGPT. small ones. This ultimately results in poor model performance or an imbalanced training proce…
Figure 12
Figure 12. Figure 12: Visualization of layer-wise Learning Rate and [PITH_FULL_IMAGE:figures/full_fig_p024_12.png]
Figure 13
Figure 13. Figure 13: Block-wise Sparsity Ratio for LLaMA-7B assigned by FARMS. We set up FARMS with four [PITH_FULL_IMAGE:figures/full_fig_p025_13.png]
Figure 14
Figure 14. Figure 14: Compare the PL Alpha Hill from FARMS and Baseline in measuring the training quality of a single layer. The Correlation Coefficient between FARMS and training quality is -0.89 and for baseline is -0.51. We can find that FARMS can measure the training quality more preci…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. S3LoRA: Safe Spectral Sharpness-Guided Pruning in Adaptation of Agent Planner

    cs.AI 2025-08 conditional novelty 5.0 of 10

    S3LoRA prunes LoRA layers with the sharpest spectral update concentration to improve safety in fine-tuned LLM agents without needing base models or extra data.

Reference graph

Works this paper leans on

74 extracted references · 31 canonical work pages · cited by 1 Pith paper

  1. [1]

    The neural tangent kernel in high dimensions: Triple descent and a multi-scale theory of generalization

    Ben Adlam and Jeffrey Pennington. The neural tangent kernel in high dimensions: Triple descent and a multi-scale theory of generalization. In International Conference on Machine Learning, pages 74--84. PMLR, 2020

    work page 2020

  2. [2]

    powerlaw: a python package for analysis of heavy-tailed distributions

    Jeff Alstott, Ed Bullmore, and Dietmar Plenz. powerlaw: a python package for analysis of heavy-tailed distributions. PloS one, 9 0 (1): 0 e85777, 2014

    work page 2014

  3. [3]

    High-dimensional asymptotics of feature learning: How one gradient step improves the representation

    Jimmy Ba, Murat A Erdogdu, Taiji Suzuki, Zhichao Wang, Denny Wu, and Greg Yang. High-dimensional asymptotics of feature learning: How one gradient step improves the representation. Advances in Neural Information Processing Systems, 35: 0 37932--37946, 2022

    work page 2022

  4. [4]

    Spectral analysis of large dimensional random matrices, volume 20

    Zhidong Bai and Jack W Silverstein. Spectral analysis of large dimensional random matrices, volume 20. Springer, 2010

    work page 2010

  5. [5]

    Neural architecture search on imagenet in four gpu hours: A theoretically inspired perspective

    Wuyang Chen, Xinyu Gong, and Zhangyang Wang. Neural architecture search on imagenet in four gpu hours: A theoretically inspired perspective. arXiv preprint arXiv:2102.11535, 2021

    arXiv 2021

  6. [6]

    Policy learning from tutorial books via understanding, rehearsing and introspecting

    Xiong-Hui Chen, Ziyan Wang, Yali Du, Shengyi Jiang, Meng Fang, Yang Yu, and Jun Wang. Policy learning from tutorial books via understanding, rehearsing and introspecting. In Advances in Neural Information Processing Systems, volume 37, 2024

    work page 2024

  7. [7]

    Boolq: Exploring the surprising difficulty of natural yes/no questions

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044, 2019

    arXiv 1905

  8. [8]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

    arXiv 2018

Show all 74 references
  1. [9]

    Power-law distributions in empirical data

    Aaron Clauset, Cosma Rohilla Shalizi, and Mark EJ Newman. Power-law distributions in empirical data. SIAM review, 51 0 (4): 0 661--703, 2009

    2009

  2. [10]

    Random Matrix Methods for Machine Learning

    Romain Couillet and Zhenyu Liao. Random Matrix Methods for Machine Learning. Cambridge University Press, 2022

    2022

  3. [11]

    Exact expressions for double descent and implicit regularization via surrogate random design

    Michal Derezinski, Feynman T Liang, and Michael W Mahoney. Exact expressions for double descent and implicit regularization via surrogate random design. Advances in neural information processing systems, 33: 0 5152--5164, 2020

    2020

  4. [12]

    High-dimensional asymptotics of prediction: Ridge regression and classification

    Edgar Dobriban and Stefan Wager. High-dimensional asymptotics of prediction: Ridge regression and classification. The Annals of Statistics, 46 0 (1): 0 247--279, 2018

    2018

  5. [13]

    Generalizable adversarial training via spectral normalization

    Farzan Farnia, Jesse M Zhang, and David Tse. Generalizable adversarial training via spectral normalization. arXiv preprint arXiv:1811.07457, 2018

    2018 arXiv

  6. [14]

    Sparsegpt: Massive language models can be accurately pruned in one-shot

    Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pages 10323--10337. PMLR, 2023

    2023

  7. [15]

    A framework for few-shot language model evaluation

    Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, et al. A framework for few-shot language model evaluation. Version v0. 0.1. Sept, 10: 0 8--9, 2021

    2021

  8. [16]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  9. [17]

    Weidenmüller

    Thomas Guhr, Axel Müller–Groeling, and Hans A. Weidenmüller. Random-matrix theories in quantum physics: common concepts. Physics Reports, 299 0 (4): 0 189--425, June 1998. ISSN 0370-1573. doi:10.1016/S0370-1573(97)00088-4. URL https://www.sciencedirect.com/science/article/pii/...

    1998 doi

  10. [18]

    The heavy-tail phenomenon in sgd

    Mert Gurbuzbalaban, Umut Simsekli, and Lingjiong Zhu. The heavy-tail phenomenon in sgd. In International Conference on Machine Learning, pages 3964--3975. PMLR, 2021

    2021

  11. [19]

    Learning both weights and connections for efficient neural network

    Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc....

    2015

  12. [20]

    Dpot: Auto-regressive denoising operator transformer for large-scale pde pre-training

    Zhongkai Hao, Chang Su, Songming Liu, Julius Berner, Chengyang Ying, Hang Su, Anima Anandkumar, Jian Song, and Jun Zhu. Dpot: Auto-regressive denoising operator transformer for large-scale pde pre-training. arXiv preprint arXiv:2403.03542, 2024

    2024 arXiv

  13. [21]

    Surprises in high-dimensional ridgeless least squares interpolation

    Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani. Surprises in high-dimensional ridgeless least squares interpolation. Annals of statistics, 50 0 (2): 0 949, 2022

    2022

  14. [22]

    Delving deep into rectifiers: Surpassing human-level performance on imagenet classification

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026--1034, 2015

    2015

  15. [23]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770--778, 2016

    2016

  16. [24]

    A simple general approach to inference about the tail of a distribution

    Bruce M Hill. A simple general approach to inference about the tail of a distribution. The annals of statistics, pages 1163--1174, 1975

    1975

  17. [25]

    Multiplicative noise and heavy tails in stochastic optimization

    Liam Hodgkinson and Michael Mahoney. Multiplicative noise and heavy tails in stochastic optimization. In International Conference on Machine Learning, pages 4262--4274. PMLR, 2021

    2021

  18. [26]

    Generalization bounds using lower tail exponents in stochastic optimizers

    Liam Hodgkinson, Umut Simsekli, Rajiv Khanna, and Michael Mahoney. Generalization bounds using lower tail exponents in stochastic optimizers. In International Conference on Machine Learning, pages 8774--8795. PMLR, 2022

    2022

  19. [27]

    Universality laws for high-dimensional learning with random features

    Hong Hu and Yue M Lu. Universality laws for high-dimensional learning with random features. IEEE Transactions on Information Theory, 69 0 (3): 0 1932--1964, 2022

    1932

  20. [28]

    Crafting heavy-tails in weight matrix spectrum without gradient noise, 2024

    Vignesh Kothapalli, Tianyu Pang, Shenyang Deng, Zongmin Liu, and Yaoqing Yang. Crafting heavy-tails in weight matrix spectrum without gradient noise, 2024. URL https://arxiv.org/abs/2406.04657

    2024 arXiv

  21. [29]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009

    2009

  22. [30]

    BaWA : Automatic optimizing pruning metric for large language models with balanced weight and activation

    Lian Liu, Xiandong Zhao, Guanchen Li, Dong Li, Wang, Yinhe Han, Xiaowei Li, and Ying Wang. BaWA : Automatic optimizing pruning metric for large language models with balanced weight and activation. In Proceedings of the 42nd International Conference on Machine Learning, ICML '2...

    2025

  23. [31]

    Model balancing helps low-data training and fine-tuning

    Zihang Liu, Yuanzhe Hu, Tianyu Pang, Yefan Zhou, Pu Ren, and Yaoqing Yang. Model balancing helps low-data training and fine-tuning. arXiv preprint arXiv:2410.12178, 2024

    2024 arXiv

  24. [32]

    Lift the veil for the truth: Principal weights emerge after rank reduction for reasoning-focused supervised fine-tuning

    Zihang Liu, Tianyu Pang, Oleg Balabanov, Chaoqun Yang, Tianjin Huang, Lu Yin, Yaoqing Yang, and Shiwei Liu. Lift the veil for the truth: Principal weights emerge after rank reduction for reasoning-focused supervised fine-tuning. In Proceedings of the 42nd International Confere...

    2025

  25. [33]

    Alphapruning: Using heavy-tailed self regularization theory for improved layer-wise pruning of large language models

    Haiquan Lu, Yefan Zhou, Shiwei Liu, Zhangyang Wang, Michael W Mahoney, and Yaoqing Yang. Alphapruning: Using heavy-tailed self regularization theory for improved layer-wise pruning of large language models. arXiv preprint arXiv:2410.10912, 2024

    arXiv

  26. [34]

    Traditional and heavy tailed self regularization in neural network models

    Michael Mahoney and Charles Martin. Traditional and heavy tailed self regularization in neural network models. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Lea...

    2019

  27. [35]

    Traditional and heavy-tailed self regularization in neural network models

    Charles H Martin and Michael W Mahoney. Traditional and heavy-tailed self regularization in neural network models. arXiv preprint arXiv:1901.08276, 2019

    1901 arXiv

  28. [36]

    Implicit self-regularization in deep neural networks: Evidence from random matrix theory and implications for learning

    Charles H Martin and Michael W Mahoney. Implicit self-regularization in deep neural networks: Evidence from random matrix theory and implications for learning. Journal of Machine Learning Research, 22 0 (165): 0 1--73, 2021

    2021

  29. [37]

    Martin, Tongsu Peng, and Michael W

    Charles H. Martin, Tongsu Peng, and Michael W. Mahoney. Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data. Nature Communications, 12 0 (1), July 2021. ISSN 2041-1723. doi:10.1038/s41467-021-24025-8. URL http://dx.do...

    2021 doi

  30. [38]

    The generalization error of random features regression: Precise asymptotics and the double descent curve

    Song Mei and Andrea Montanari. The generalization error of random features regression: Precise asymptotics and the double descent curve. Communications on Pure and Applied Mathematics, 75 0 (4): 0 667--766, 2022

    2022

  31. [39]

    Pointer sentinel mixture models

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016

    2016 arXiv

  32. [40]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789, 2018

    2018 arXiv

  33. [41]

    Spectral normalization for generative adversarial networks, 2018

    Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks, 2018. URL https://arxiv.org/abs/1802.05957

    2018 arXiv

  34. [42]

    Graph spectra and the detectability of community structure in networks

    Raj Rao Nadakuditi and Mark EJ Newman. Graph spectra and the detectability of community structure in networks. Physical review letters, 108 0 (18): 0 188701, 2012

    2012

  35. [43]

    Nonlinear random matrix theory for deep learning

    Jeffrey Pennington and Pratik Worah. Nonlinear random matrix theory for deep learning. In Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper/2017/hash/0f3d014eead934bbdbacb62a01dc4831-Abstract.html

    2017

  36. [44]

    Alphalora: Assigning lora experts based on layer training quality

    Peijun Qing, Chongyang Gao, Yefan Zhou, Xingjian Diao, Yaoqing Yang, and Soroush Vosoughi. Alphalora: Assigning lora experts based on layer training quality. arXiv preprint arXiv:2410.10054, 2024

    2024 arXiv

  37. [45]

    Winogrande: An adversarial winograd schema challenge at scale

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64 0 (9): 0 99--106, 2021

    2021

  38. [46]

    Stable rank normalization for improved generalization in neural networks and gans

    Amartya Sanyal, Philip HS Torr, and Puneet K Dokania. Stable rank normalization for improved generalization in neural networks and gans. arXiv preprint arXiv:1906.04659, 2019

    1906 arXiv

  39. [47]

    Very deep convolutional networks for large-scale image recognition

    Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014

    2014 arXiv

  40. [48]

    A tail-index analysis of stochastic gradient noise in deep neural networks

    Umut Simsekli, Levent Sagun, and Mert Gurbuzbalaban. A tail-index analysis of stochastic gradient noise in deep neural networks. In International Conference on Machine Learning, pages 5827--5837. PMLR, 2019

    2019

  41. [49]

    Hausdorff dimension, heavy tails, and generalization in neural networks

    Umut Simsekli, Ozan Sener, George Deligiannidis, and Murat A Erdogdu. Hausdorff dimension, heavy tails, and generalization in neural networks. Advances in Neural Information Processing Systems, 33: 0 5138--5151, 2020

    2020

  42. [50]

    A simple and effective pruning approach for large language models

    Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695, 2023

    2023 arXiv

  43. [51]

    Pdebench: An extensive benchmark for scientific machine learning

    Makoto Takamoto, Timothy Praditia, Raphael Leiteritz, Daniel MacKinlay, Francesco Alesiani, Dirk Pfl \"u ger, and Mathias Niepert. Pdebench: An extensive benchmark for scientific machine learning. Advances in Neural Information Processing Systems, 35: 0 1596--1611, 2022

    2022

  44. [52]

    Topics in random matrix theory, volume 132

    Terence Tao. Topics in random matrix theory, volume 132. American Mathematical Soc., 2012

    2012

  45. [53]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023 a

    2023 arXiv

  46. [54]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023 b

    2023 arXiv

  47. [55]

    Tulino and Sergio Verdú

    Antonia M. Tulino and Sergio Verdú. Random Matrix Theory and Wireless Communications . Foundations and Trends® in Communications and Information Theory, 1 0 (1): 0 1--182, June 2004. ISSN 1567-2190, 1567-2328. doi:10.1561/0100000001. URL https://www.nowpublishers.com/article/D...

    2004 doi

  48. [56]

    Glue: A multi-task benchmark and analysis platform for natural language understanding

    Alex Wang. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018

    2018 arXiv

  49. [57]

    Spectral Evolution and Invariance in Linear -width Neural Networks , November 2023

    Zhichao Wang, Andrew Engel, Anand Sarwate, Ioana Dumitriu, and Tony Chiang. Spectral Evolution and Invariance in Linear -width Neural Networks , November 2023. URL http://arxiv.org/abs/2211.06506. arXiv:2211.06506 [cs]

    2023 arXiv

  50. [58]

    Safe multi-agent reinforcement learning with natural language constraints

    Ziyan Wang, Meng Fang, Tristan Tomilin, Fei Fang, and Yali Du. Safe multi-agent reinforcement learning with natural language constraints. arXiv preprint arXiv:2405.20018, 2024

    2024 arXiv

  51. [59]

    M3hf: Multi-agent reinforcement learning from multi-phase human feedback of mixed quality

    Ziyan Wang, Zhicheng Zhang, Fei Fang, and Yali Du. M3hf: Multi-agent reinforcement learning from multi-phase human feedback of mixed quality. In The Twelfth International Conference on Learning Representations, 2025

    2025

  52. [60]

    Tensor programs iv: Feature learning in infinite-width neural networks

    Greg Yang and Edward J Hu. Tensor programs iv: Feature learning in infinite-width neural networks. In International Conference on Machine Learning, pages 11727--11737. PMLR, 2021

    2021

  53. [61]

    Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer

    Greg Yang, Edward J Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer. arXiv preprint arXiv:2203.03466, 2022

    2022 arXiv

  54. [62]

    Mitigating the backdoor effect for multi-task model merging via safety-aware subspace

    Jinluan Yang, Anke Tang, Didi Zhu, Zhengyu Chen, Li Shen, and Fei Wu. Mitigating the backdoor effect for multi-task model merging via safety-aware subspace. arXiv preprint arXiv:2410.13910, 2024

    2024 arXiv

  55. [63]

    Multimodal commonsense knowledge distillation for visual question answering (student abstract)

    Shuo Yang, Siwen Luo, and Soyeon Caren Han. Multimodal commonsense knowledge distillation for visual question answering (student abstract). In Proceedings of the AAAI conference on artificial intelligence, pages 29545--29547, 2025 a

    2025

  56. [64]

    Magic-vqa: Multimodal and grounded inference with commonsense knowledge for visual question answering

    Shuo Yang, Siwen Luo, Soyeon Caren Han, and Eduard Hovy. Magic-vqa: Multimodal and grounded inference with commonsense knowledge for visual question answering. arXiv preprint arXiv:2503.18491, 2025 b

    2025 arXiv

  57. [65]

    Gonzalez, Kannan Ramchandran, Charles H

    Yaoqing Yang, Ryan Theisen, Liam Hodgkinson, Joseph E. Gonzalez, Kannan Ramchandran, Charles H. Martin, and Michael W. Mahoney. Test accuracy vs. generalization gap: Model selection in nlp without accessing training or testing data. KDD '23, page 3011–3021, New York, NY, USA, ...

    2023

  58. [66]

    Outlier weighed layerwise sparsity (owl): A missing secret sauce for pruning llms to high sparsity

    Lu Yin, You Wu, Zhenyu Zhang, Cheng-Yu Hsieh, Yaqing Wang, Yiling Jia, Gen Li, Ajay Jaiswal, Mykola Pechenizkiy, Yi Liang, et al. Outlier weighed layerwise sparsity (owl): A missing secret sauce for pruning llms to high sparsity. arXiv preprint arXiv:2310.05175, 2023

    arXiv

  59. [67]

    Spectral norm regularization for improving the generalizability of deep learning, 2017

    Yuichi Yoshida and Takeru Miyato. Spectral norm regularization for improving the generalizability of deep learning, 2017. URL https://arxiv.org/abs/1705.10941

    2017 arXiv

  60. [68]

    Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

    1905 arXiv

  61. [69]

    Opt: Open pre-trained transformer language models

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022

    2022 arXiv

  62. [70]

    Temperature balancing, layer-wise weight analysis, and neural network training

    Yefan Zhou, Tianyu Pang, Keqin Liu, Michael W Mahoney, Yaoqing Yang, et al. Temperature balancing, layer-wise weight analysis, and neural network training. Advances in Neural Information Processing Systems, 36, 2024

    2024

  63. [71]

    Remedy: Recipe merging dynamics in large vision-language models

    Didi Zhu, Yibing Song, Tao Shen, Ziyu Zhao, Jinluan Yang, Min Zhang, and Chao Wu. Remedy: Recipe merging dynamics in large vision-language models. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  64. [72]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  65. [73]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  66. [74]

    training quality

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.