Pith. sign in

REVIEW 3 major objections 4 minor 67 references

Spurious Correlations in High Dimensional Regression: The Roles of Regularization, Simplicity Bias and Over-Parameterization

T0 review · 3 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Ridge regularization—not interpolation—makes high-dimensional regression learn spurious correlations, and the loss-optimal penalty sits exactly in the growing-spurious-correlation range.

desk verdict Solid concentration results, but the headline trade-off is proved only under Sigma_xx=I and a sample-ratio condition that the paper's own experiments violate; the RF equivalence is pointwise, not metric-level. read the letter →

arxiv 2502.01347 v2 pith:OXKN63NX submitted 2025-02-03 stat.ML cs.LG

Simone Bombari , Marco Mondelli This is my paper · ORCID
classification stat.MLcs.LG MSC 62J0762H2060B20
keywords spuriouscorrelationshigh-dimensionalregressionridgeregularizationsimplicitybiasrandomfeaturesover-parameterizationtestlosstrade-offSchurcomplement
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish a quantitative, first-principles account of when and why high-dimensional regression models learn spurious correlations. Its core claim is that for linear regression with Gaussian data and $n=\Theta(d)$, the covariance-based measure $C$ of spurious correlation learned by the ridge-regularized empirical risk minimizer concentrates around a deterministic function $C_\Sigma(\lambda)$ of the data covariance and the regularization strength. From that function the paper proves a trade-off: the ridge value that minimizes the in-distribution test loss lies in an interval where $C$ is positive and increasing, so the best in-distribution model necessarily relies on spurious features. It also shows that an over-parameterized random-features model is asymptotically equivalent to ridge regression with a positive effective regularization, which explains why over-parameterized models keep spurious correlations even with no explicit penalty. A sympathetic reader should care because this turns a phenomenon usually discussed qualitatively—shortcut learning and simplicity bias—into a computable statement about spectra of covariances.

What carries the argument

The load-bearing object is the deterministic functional $C_\Sigma(\lambda)$, the population analogue of the learned spurious correlation. It is a quadratic form in the projected covariance $P_y\Sigma$ and the resolvent $(\Sigma+\tau(\lambda)I)^{-1}$, with the auxiliary parameter $\tau(\lambda)$ fixed by the trace equation $1-\lambda/\tau=\frac1n\operatorname{tr}((\Sigma+\tau I)^{-1}\Sigma)$. The Schur complement $S_x^\Sigma$ enters as the conditional covariance of $y$ given $x$ and encodes how strongly the spurious feature is tied to the core feature, while $\lambda_{\max}(\Sigma_{yy})$ encodes spectral simplicity of the spurious feature. In the random-features half, the mechanism is the predictor-level equivalence $f_{\mathrm{RF}}(\hat{\theta}_{\mathrm{RF}}(\lambda),z)\approx f_{\mathrm{LR}}(\hat{\theta}_{\mathrm{LR}}(\tilde\lambda),z)$ with the explicit effective regularization $\tilde\lambda$, derived by splitting the activation into its first Hermite component and a negligible nonlinear remainder.

What would settle it

Simulate the paper's Gaussian linear model with $n=\Theta(d)$, a strongly non-isotropic core covariance $\Sigma_{xx}$ with eigenvalues spread over two decades, and a sample ratio $2d/n$ violating the bound (5.8); find the ridge that minimizes the in-distribution test loss by cross-validation and estimate $C(\hat{\theta}_{\mathrm{LR}}(\lambda))$. If the minimizing $\lambda$ falls outside the range where $C_\Sigma(\lambda)$ is increasing, or if $C$ is decreasing at the optimal $\lambda$, the trade-off claim is false in that regime.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that spurious correlations are created by regularization, not by interpolation. For unregularized linear regression with $n=\omega(d)$, the expected amount of spurious correlation is zero and the random fluctuations vanish as $O(\log d/\sqrt{d})$. With ridge regularization, however, the learned spurious correlation $C(\hat{\theta}_{\mathrm{LR}}(\lambda))$ concentrates around $C_\Sigma(\lambda)=\theta^{*\top}\Sigma(\Sigma+\tau(\lambda)I)^{-1}P_y\Sigma\theta^*$, where $\tau(\lambda)$ is the unique positive solution of $1-\lambda/\tau=\frac1n\operatorname{tr}((\Sigma+\tau I)^{-1}\Sigma)$. This deterministic proxy lets the paper read the data geometry directly: the cross-covariance $\Sigma_{yx}$ controls the bare correlation between features, the Schur complement $S_x^\Sigma=\Sigma_{yy}-\Sigma_{yx}\Sigma_{xx}^{-1}\Sigma_{xy}$ measures how much of $y$ is predictable from $x$, and the largest eigenvalue of $\Sigma_{yy}$ measures how spectrally simple the spurious feature is. Under the conditions of Proposition 5.3, $C_\Sigma(\lambda)$ is non-negative and increasing up to $\lambda_C$, while the in-distribution loss $L_\Sigma(\lambda)$ is minimized at some $\lambda^*_L\in(0,\lambda_C]$, so the loss-optimal regularized model is provably in the regime of growing spurious correlation. The random-features result completes the picture: $f_{\mathrm{RF}}(\hat{\theta}_{\mathrm{RF}}(\lambda),z)$ is asymptotically equal to $f_{\mathrm{LR}}(\hat{\theta}_{\mathrm{LR}}(\tilde\lambda),z)$ with $\tilde\lambda=\frac{2\tilde\mu_2 d}{\mu_1^2 n}+\frac{2d}{\mu_1^2 p}\lambda$, so over-parameterization itself acts as ridge regularization.

Load-bearing premise

The trade-off proof assumes the core-feature covariance is exactly the identity and that the sample-to-dimension ratio $2d/n$ is small enough to satisfy the bound in (5.8); outside that regime the paper does not prove that the loss-optimal ridge sits in the interval where spurious correlations are increasing.

Editorial extensions

If this is right

  • Unregularized linear interpolation with $n=\omega(d)$ does not learn spurious correlations: the expected $C$ is exactly zero and the magnitude is $O(\log d/\sqrt{d})$ with high probability.
  • Any ridge level that minimizes the in-distribution test loss lies in the interval $(0,\lambda_C]$ where the deterministic spurious correlation $C_\Sigma(\lambda)$ is positive and increasing, so in-distribution optimality and spurious reliance cannot be separated in this model.
  • Spurious features that are spectrally simpler, meaning larger $\lambda_{\max}(\Sigma_{yy})$, or more tightly correlated with the core feature, meaning smaller $\lambda_{\min}(S_x^\Sigma)$, produce larger learned spurious correlations.
  • A random-features model with zero explicit ridge is equivalent to ridge regression with effective regularization $\tilde\lambda>0$, so over-parameterization alone generates spurious correlations; the effect is stronger for activations with larger $\tilde\mu_2/\mu_1^2$.
  • Increasing $\lambda$ beyond the intermediate range suppresses $C$ but only by shrinking the whole predictor, which raises the in-distribution test loss.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because $C_\Sigma(\lambda)$ is computable from the covariance $\Sigma$, one could screen a dataset before training—estimate $\Sigma_{yy}$ and the Schur complement, compute the predicted spurious-correlation profile, and decide whether ERM will need group annotations or decorrelation.
  • Editorial inference: early stopping and ridge regularization should follow the same $C$-versus-$\lambda$ trade-off, so a network stopped early at the loss-optimal epoch should exhibit the same positive and increasing spurious correlation as the ridge-optimal linear predictor.
  • Editorial inference: the random-features equivalence suggests that the persistent spurious correlations seen in deep networks at small explicit $\lambda$ are an effective-regularization effect rather than a feature-learning effect; measuring the empirical NTK or Hessian regularization would yield a testable effective $\tilde\lambda$.
  • Editorial inference: if the trade-off is as sharp as stated, methods that reduce $C$ by reweighting or invariant objectives will generically incur an in-distribution loss penalty unless they also change the covariance structure that couples $y$ to $x$.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper studies spurious correlations in high-dimensional ridge regression and in a random-features model. It defines a metric C(θ) as the covariance between the model output on a fresh independent core feature together with the spurious feature and the label, and gives a non-asymptotic characterization of C for ridge-regularized linear regression: Theorem 1 shows that C(θLR(λ)) concentrates around a deterministic quantity CΣ(λ) defined through the data covariance Σ and an implicit regularization parameter τ(λ). Proposition 5.1 interprets CΣ(λ) in terms of the Schur complement of Σ and the spectrum of the spurious-feature covariance. Proposition 5.3 states a trade-off between CΣ(λ) and the in-distribution test loss LΣ(λ), claiming that the loss-minimizing λ lies in a region where CΣ(λ) is increasing. Theorem 2 gives a pointwise equivalence between a random-features predictor and a linear ridge predictor with an effective regularization. The theoretical results are complemented by experiments on synthetic Gaussian data, Color-MNIST, and CIFAR-10.

Significance. If fully established, the paper would provide a valuable quantitative handle on spurious correlations and simplicity bias in high-dimensional regression, connecting these phenomena to the spectrum of the data covariance and to ridge regularization. The concentration result Theorem 1 is a clean application of the Han–Xu machinery and appears correct; Proposition 5.2 is likewise a sound adaptation. The deterministic expression CΣ(λ) is elegant and falsifiable, and the experiments on multiple datasets support the qualitative trends. The paper also makes a useful contribution by proving a pointwise predictor-level equivalence for random features rather than only an equivalence of training/test losses. However, the headline trade-off claim is proved only under restrictive auxiliary assumptions, and the transfer from the random-features predictor equivalence to convergence of the C metric is not proved. The core concentration results are solid, but the advertised conclusions go beyond what is established.

major comments (3)
  1. [§5, Proposition 5.3 and abstract/introduction] The central trade-off claim—that the regularization minimizing the in-distribution loss lies in an interval where CΣ(λ) is increasing—is conditional on two restrictive hypotheses that are not reflected in the abstract or in the discussion following Proposition 5.3. Monotonicity and non-negativity of CΣ are proved only for Σxx = I (the derivative computation (B.35) is specific to that choice), and the inequality λC ≥ λL is proved only under the sample-ratio bound (5.8). For Σxx ≠ I, the representation (B.15) shows that the sign of CΣ(λ) depends on the alignment of θx* with Σxy, and no monotonicity argument is supplied. Moreover, (5.8) is not a generic n = Θ(d) condition: it forces 2d/n to be small relative to the spectral condition number of Σ, and in the paper's own synthetic example of Figure 2 (d = 400, n = 2000, λmax(Σyy) = 2, β = 0.5, σ² = 0.25) one has 2d/n = 0.4 while the right-hand side of (5.8) evaluates to roughly 3.5×10^{-3}. Thus the numerical demonstration of the trade-off lies outside the regime in which Proposition 5.3 is proved. The result may be true, but the manuscript as written does not establish it at the claimed level of generality; either the proof must be extended to general Σxx and to the full proportional regime, or the statements must be explicitly restricted.
  2. [§6, Theorem 2] Theorem 2 is a pointwise equivalence between the random-features predictor and the linear predictor on a single fresh input z. It does not imply convergence of the spurious-correlation metric C(θRF(λ)), because C is a covariance over the joint distribution of the core, spurious, and independent core features, and pointwise closeness at one draw does not control this second-order expectation. The sentences after Theorem 2—'Thus, we expect the presence of spurious correlations' and 'C(θRF(0)) approaches CΣ(λ~)'—are assertions rather than consequences of the stated theorem. To make the over-parameterization claim rigorous, the paper needs either a uniform version of the equivalence over the relevant population (for example, control of E_z[|fRF(z) - fLR(z)|²], or a direct bound on the covariance functional) or an explicit statement that the transfer to C is conjectural.
  3. [§5 and Appendix B, Proposition 5.3 proof] The proof of Proposition 5.3 itself relies on a sequence of inequalities that establish monotonicity of LΣ(λ) only for sufficiently large τ, and the final statement λC ≥ λL is obtained by requiring both the monotonicity threshold and the small-ratio condition (5.8). The paper's remark that (5.8) is 'implied by taking n = ω(d)' is asymptotic and does not cover the n = Θ(d) regime in which Theorem 1 and the experiments operate; for any fixed ratio n/d = c, the condition (5.8) fails when c is below a constant threshold. The manuscript should either weaken the claims to the asymptotic regime where it is proved or provide a finite-ratio analysis.
minor comments (4)
  1. [§4, proof of Proposition 4.1] The notation EE[C(θLR(0))] for the expectation over the label noise is easy to confuse with the noise vector E used in the same proof; a clearer notation such as E_E or E_ϵ would improve readability.
  2. [§5, discussion after Proposition 5.3] The sentence 'These trends in turn imply that the optimal value λ*_L that minimizes the test loss is s.t. λ*_L ∈ (0, λC]' is stated without repeating the hypotheses of Proposition 5.3; it should explicitly say 'under the assumptions of Proposition 5.3'.
  3. [Appendix F, synthetic Gaussian data] The synthetic data construction sets Σxy = (Σyy - βI)^{1/2}, which gives a Schur complement βI; however, the resulting full matrix Σ has a smallest eigenvalue strictly smaller than β. This affects any numerical check of condition (5.8) and should be stated explicitly so that readers can verify the regime.
  4. [Figures 2 and 5] The red-curve labels in the figure captions appear to omit the symbol C (for example, '(LR(λ)), n = 1000' should presumably read 'C(θLR(λ)), n = 1000'); the captions should be corrected for clarity.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central characterization is a genuine concentration result built on an external benchmark, and the trade-off is a direct calculus argument; the few self-citations in the random-features appendix are auxiliary, not load-bearing.

full rationale

The derivation chain is self-contained relative to its announced inputs. Theorem 1 and Proposition 5.2 are concentration results obtained by applying Han and Xu [19], an external non-asymptotic characterization of ridge estimators; the target statistic C(theta) = theta^T P_y Sigma theta* enters only as the linear functional being concentrated, and C_Sigma(lambda) is its expectation under the Gaussian sequence model (Eqs. B.9-B.13), not a quantity fitted to the data or defined in terms of the claim. The trade-off Proposition 5.3 is a direct calculus argument: C_Sigma is rewritten via the Schur-complement identity (B.15), and under Sigma_xx = I the derivative (B.35) is manifestly nonnegative for tau <= sqrt(lambda_min(Sigma_x)), giving monotonicity on [0, lambda_C]; the final lambda_C >= lambda_L claim follows from the explicit bound (5.8) and Lemma B.3, an independent matrix inequality. No step in this chain assumes the conclusion it is meant to prove. The random-features section derives a pointwise predictor equivalence (Theorem 2) through concentration lemmas; although Lemmas C.4 and C.5 borrow technical arguments from the authors' earlier work [6,8], those are auxiliary operator-norm and Hermite-expansion bounds whose assumptions do not include the RF equivalence and which are not the result being proved, so the citation is real evidence rather than circular import. The main caveat is one of scope, not circularity: the monotonicity/trade-off conclusion is proven only for Sigma_xx = I and under the small-ratio condition (5.8), while the paper's own synthetic experiment uses 2d/n = 0.4, outside (5.8). That is a limitation of the theorem's applicability, not a reduction of the prediction to its inputs.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

No free parameters are fitted: τ(λ) is the solution of an implicit equation determined by Σ and λ, not chosen to match data. The central theoretical claims rest on the external Han-Xu concentration theorem and on the stated Gaussian/Lipschitz/over-parameterization assumptions. The strongest restriction is the ad hoc Σxx=I condition in the trade-off result.

assumptions (6)
  • domain assumption Gaussian data with invertible well-conditioned covariance and trace 2d (Assumption 1)
    Used to invoke Han-Xu concentration results in Theorems 1 and Prop 5.2; the extension to sub-Gaussian needs the additional technical Assumption 6 on θ*.
  • domain assumption Odd, Lipschitz activation with nonzero first Hermite coefficient (Assumption 2)
    Required for the random-features expansion and for the concentration of ΦΦ^T to a linear kernel; excludes standard ReLU-like even activations.
  • domain assumption p=ω(n log^4 n) over-parameterization (Assumption 3)
    Needed in the random-features proof to suppress non-linear terms; much stronger than the proportional p=Θ(n) regime of prior equivalence results.
  • domain assumption Lipschitz-concentrated data distribution (Assumption 4)
    Background requirement for random-features concentration; covers Gaussian, sphere and hypercube but not arbitrary heavy-tailed data.
  • standard math Han-Xu [19] non-asymptotic distribution of ridge estimators (Theorems 2.3 and 3.1)
    External theorem that supplies the Gaussian sequence model and concentration for Lipschitz functions and prediction risk; the paper's linear-regression results are built directly on it.
  • ad hoc to paper Σxx=I for the monotonicity of CΣ(λ) (Prop 5.3)
    The derivative computation uses the identity covariance to simplify A(τ); the trade-off theorem is not proven for general core covariance.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Spurious Correlations in High Dimensional Regression: The Roles of Regularization, Simplicity Bias and Over-Parameterization." pith.science (2026). https://pith.science/paper/OXKN63NX

@misc{pith2026250201347,
  author       = {Pith},
  title        = {Pith review of: Spurious Correlations in High Dimensional Regression: The Roles of Regularization, Simplicity Bias and Over-Parameterization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OXKN63NX}},
  note         = {Machine review of arXiv:2502.01347}
}
abstract

Learning models have been shown to rely on spurious correlations between non-predictive features and the associated labels in the training data, with negative implications on robustness, bias and fairness. In this work, we provide a statistical characterization of this phenomenon for high-dimensional regression, when the data contains a predictive core feature $x$ and a spurious feature $y$. Specifically, we quantify the amount of spurious correlations $C$ learned via linear regression, in terms of the data covariance and the strength $\lambda$ of the ridge regularization. As a consequence, we first capture the simplicity of $y$ through the spectrum of its covariance, and its correlation with $x$ through the Schur complement of the full data covariance. Next, we prove a trade-off between $C$ and the in-distribution test loss $L$, by showing that the value of $\lambda$ that minimizes $L$ lies in an interval where $C$ is increasing. Finally, we investigate the effects of over-parameterization via the random features model, by showing its equivalence to regularized linear regression. Our theoretical results are supported by numerical experiments on Gaussian, Color-MNIST, and CIFAR-10 datasets.

Figures

Figures reproduced from arXiv: 2502.01347 by the authors.

Figure 1
Figure 1. Left two panels: pictorial representation of the core (spurious) feature x (y) and an inde￾pendent core feature x˜, taken from an image of a boat and a truck in the CIFAR-10 dataset. Right two panels: examples from a binary Color-MNIST dataset, where the labels correspond to the number shapes, and the zeros (ones) are colored in blue (red) with probability (1 + α)/2. Prior work has attempted to formalize the sim￾pli… view at source ↗
Figure 2
Figure 2. Test loss L( ˆθLR(λ)) (black) and spurious corre￾lations C( ˆθLR(λ) (red) as a function of the regularization term λ for two values of the number of samples n. Left: synthetic Gaussian dataset, with d = 400 (additional details in Appendix F); right: binary Color-MNIST dataset with correlation √ 1 − α2 = 0.25 between color and digit (see [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Test loss L( ˆθLR/RF(λ)) (black) and spurious corre￾lations C( ˆθLR/RF(λ)) (red) as a function of λmax (Σyy) (left) and λmin S Σ x  (right) on a synthetic Gaussian dataset, for both linear regression and random features, with λ = 1 (additional details in Appendix F). The connection between λmax (Σyy) and the simplicity bias of ERM can be illustrated via our initial image recog￾nition example. The (spurious) back￾gr… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Test loss L( ˆθLR/RF(λ)) (black) and spu￾rious correlations C( ˆθLR/RF(λ) (red) as a function of λmax (Σyy) / tr(Σyy) on a CIFAR-10 dataset for different levels of whitening (details on the whitening process in Appendix F). We restrict to the classes “boat” and “truck”…
Figure 6
Figure 6. Figure 6: Out-of-distribution test loss L( ˆθLR/RF(λ)) (black) and spurious correlations C( ˆθLR/RF(λ) (red) as a function of λmax (Σyy) (first panel) and λmin S Σ x  (second panel) on a Gaussian synthetic dataset, and for the CIFAR-10 experiment (third panel). We consider the …
Figure 7
Figure 7. Figure 7: Test loss L( ˆθNN/LR(λ)) (black) and spurious correlations C( ˆθNN/LR(λ) (red) as a function of λ. First and second panel: 2-layer fully connected ReLU network, trained on the multi-class color(C)-MNIST, for two different values of α. Third and fourth panel: Same setup…
Figure 8
Figure 8. Figure 8: Test loss L( ˆθNN/RF(λ)) (black) and spu￾rious correlations C( ˆθNN/RF(λ) (red) as a function of λ. Left: 2-layer fully connected ReLU network, trained on the Corrupted CIFAR-10 dataset (boats and trucks, with added textures “brightness” and “glass blur”). Right: RF mo…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

67 extracted references · 52 canonical work pages

  1. [19]

    The distribution of ridgeless least squares interpolators.arXiv preprint arXiv:2307.02044, 2023

    Qiyang Han and Xiaocong Xu. The distribution of ridgeless least squares interpolators.arXiv preprint arXiv:2307.02044, 2023

    arXiv 2023

  2. [1]

    Systematic generalisation with group invariant predictions

    Faruk Ahmed, Yoshua Bengio, Harm van Seijen, and Aaron Courville. Systematic generalisation with group invariant predictions. InInternational Conference on Learning Representations, 2021

    work page 2021

  3. [2]

    Invariant risk minimization

    Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02893, 2020

    arXiv 1907

  4. [3]

    High-dimensional asymptotics of feature learning: How one gradient step improves the representation

    Jimmy Ba, Murat A Erdogdu, Taiji Suzuki, Zhichao Wang, Denny Wu, and Greg Yang. High-dimensional asymptotics of feature learning: How one gradient step improves the representation. InAdvances in Neural Information Processing Systems (NeurIPS), 2022

    work page 2022

  5. [4]

    Deep learning: a statistical viewpoint

    Peter L Bartlett, Andrea Montanari, and Alexander Rakhlin. Deep learning: a statistical viewpoint. Acta numerica, 30:87–201, 2021. 14

    work page 2021

  6. [5]

    Reconciling modern machine-learning practice and the classical bias–variance trade-off.Proceedings of the National Academy of Sciences, 116(32):15849–15854, 2019

    Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine-learning practice and the classical bias–variance trade-off.Proceedings of the National Academy of Sciences, 116(32):15849–15854, 2019

    2019

  7. [6]

    Memorization and optimization in deep neural networks with minimum over-parameterization

    Simone Bombari, Mohammad Hossein Amani, and Marco Mondelli. Memorization and optimization in deep neural networks with minimum over-parameterization. InAdvances in Neural Information Processing Systems (NeurIPS), 2022

    work page 2022

  8. [7]

    Beyond the universal law of robustness: Sharper laws for random features and neural tangent kernels

    Simone Bombari, Shayan Kiyani, and Marco Mondelli. Beyond the universal law of robustness: Sharper laws for random features and neural tangent kernels. InProceedings of the 40th International Conference on Machine Learning, 2023

    work page 2023

Show all 67 references
  1. [8]

    Privacy for free in the over-parameterized regime.arXiv preprint arXiv:2410.14787, 2024

    Simone Bombari and Marco Mondelli. Privacy for free in the over-parameterized regime.arXiv preprint arXiv:2410.14787, 2024

    2024 arXiv

  2. [9]

    A universal law of robustness via isoperimetry

    Sebastien Bubeck and Mark Sellke. A universal law of robustness via isoperimetry. InAdvances in Neural Information Processing Systems (NeurIPS), 2021

    2021

  3. [10]

    Chang, G

    C. Chang, G. Adam, and A. Goldenberg. Towards robust classification model by counterfactual and invariant data generation. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021

    2021

  4. [11]

    Provable benefits of overparameterization in model compression: From double descent to pruning neural networks

    Xiangyu Chang, Yingcong Li, Samet Oymak, and Christos Thrampoulidis. Provable benefits of overparameterization in model compression: From double descent to pruning neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 6974–6983, 2021

    2021

  5. [12]

    Dimension free ridge regression.The Annals of Statistics, 52(6):2879 – 2912, 2024

    Chen Cheng and Andrea Montanari. Dimension free ridge regression.The Annals of Statistics, 52(6):2879 – 2912, 2024

    2024

  6. [13]

    Neural networks can learn representations with gradient descent

    Alexandru Damian, Jason Lee, and Mahdi Soltanolkotabi. Neural networks can learn representations with gradient descent. InConference on Learning Theory (COLT), 2022

    2022

  7. [14]

    On the (non-) robustness of two-layer neural networks in different learning regimes

    Elvis Dohmatob and Alberto Bietti. On the (non-) robustness of two-layer neural networks in different learning regimes. arXiv preprint arXiv:2203.11864, 2022

    2022 arXiv

  8. [15]

    Wichmann

    Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. Shortcut learning in deep neural networks.Nature Machine Intelligence, 2(11):665–673, 2020

    2020

  9. [16]

    Wichmann, and Wieland Brendel

    Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. InInternational Conference on Learning Representations (ICLR), 2019

    2019

  10. [17]

    The gaussian equivalence of generative models for learning with shallow neural networks

    Sebastian Goldt, Bruno Loureiro, Galen Reeves, Florent Krzakala, Marc Mézard, and Lenka Zdeborová. The gaussian equivalence of generative models for learning with shallow neural networks. InMathematical and Scientific Machine Learning, pages 426–471. PMLR, 2022

    2022

  11. [18]

    Modeling the influence of data structure on learning in neural networks: The hidden manifold model.Physical Review X, 10(4):041044, 2020

    Sebastian Goldt, Marc Mézard, Florent Krzakala, and Lenka Zdeborová. Modeling the influence of data structure on learning in neural networks: The hidden manifold model.Physical Review X, 10(4):041044, 2020

    2020

  12. [20]

    The curse of overparametrization in adversarial training: Precise analysis of robust generalization for random features regression.The Annals of Statistics, 52(2):441 – 465, 2024

    Hamed Hassani and Adel Javanmard. The curse of overparametrization in adversarial training: Precise analysis of robust generalization for random features regression.The Annals of Statistics, 52(2):441 – 465, 2024

    2024

  13. [21]

    Hastie, Andrea Montanari, Saharon Rosset, and Ryan J

    Trevor J. Hastie, Andrea Montanari, Saharon Rosset, and Ryan J. Tibshirani. Surprises in high- dimensional ridgeless least squares interpolation.Annals of statistics, 50 2:949–986, 2022. 15

    2022

  14. [22]

    What shapes feature representations? Exploring datasets, architectures, and training

    Katherine Hermann and Andrew Lampinen. What shapes feature representations? Exploring datasets, architectures, and training. InAdvances in Neural Information Processing Systems (NeurIPS), 2020

    2020

  15. [23]

    On the foundations of shortcut learning

    Katherine Hermann, Hossein Mobahi, Thomas FEL, and Michael Curtis Mozer. On the foundations of shortcut learning. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  16. [24]

    Hong Hu and Yue M. Lu. Universality laws for high-dimensional learning with random features.IEEE Transactions on Information Theory, 69(3):1932–1964, 2023

    1932

  17. [25]

    On feature learning in the presence of spurious correlations

    Pavel Izmailov, Polina Kirichenko, Nate Gruver, and Andrew Gordon Wilson. On feature learning in the presence of spurious correlations. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors,Advances in Neural Information Processing Systems, 2022

    2022

  18. [26]

    Sgd on neural networks learns functions of increasing complexity

    Dimitris Kalimeris, Gal Kaplun, Preetum Nakkiran, Benjamin Edelman, Tristan Yang, Boaz Barak, and Haofeng Zhang. Sgd on neural networks learns functions of increasing complexity. InAdvances in Neural Information Processing Systems, 2019

    2019

  19. [27]

    Last layer re-training is sufficient for ro- bustness to spurious correlations

    Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re-training is sufficient for ro- bustness to spurious correlations. InThe Eleventh International Conference on Learning Representations, 2023

    2023

  20. [28]

    Demystifying disagreement-on-the-line in high dimensions

    Donghwan Lee, Behrad Moniri, Xinmeng Huang, Edgar Dobriban, and Hamed Hassani. Demystifying disagreement-on-the-line in high dimensions. InProceedings of the 40th International Conference on Machine Learning, 2023

    2023

  21. [29]

    Just train twice: Improving group robustness without training group information

    Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In Proceedings of the 38th International Conference on Machine Learning, 2021

    2021

  22. [30]

    Avoiding spurious correlations via logit correction

    Sheng Liu, Xu Zhang, Nitesh Sekhar, Yue Wu, Prateek Singhal, and Carlos Fernandez-Granda. Avoiding spurious correlations via logit correction. In The Eleventh International Conference on Learning Representations, 2023

    2023

  23. [31]

    Learning curves of generic features maps for realistic datasets with a teacher-student model

    Bruno Loureiro, Cedric Gerbelot, Hugo Cui, Sebastian Goldt, Florent Krzakala, Marc Mezard, and Lenka Zdeborová. Learning curves of generic features maps for realistic datasets with a teacher-student model. In Advances in Neural Information Processing Systems, 2021

    2021

  24. [32]

    Minimum-norm interpolation under covariate shift

    Neil Rohit Mallinar, Austin Zane, Spencer Frei, and Bin Yu. Minimum-norm interpolation under covariate shift. InForty-first International Conference on Machine Learning, 2024

    2024

  25. [33]

    Generalization error of random feature and kernel methods: Hypercontractivity and kernel matrix concentration.Applied and Computational Harmonic Analysis, 59:3–84, 2022

    Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Generalization error of random feature and kernel methods: Hypercontractivity and kernel matrix concentration.Applied and Computational Harmonic Analysis, 59:3–84, 2022. Special Issue on Harmonic Analysis and Machine Learning

    2022

  26. [34]

    The generalization error of random features regression: Precise asymptotics and the double descent curve.Communications on Pure and Applied Mathematics, 75(4):667– 766, 2022

    Song Mei and Andrea Montanari. The generalization error of random features regression: Precise asymptotics and the double descent curve.Communications on Pure and Applied Mathematics, 75(4):667– 766, 2022

    2022

  27. [35]

    Hard imagenet: Segmentations for objects with strong spurious cues

    Mazda Moayeri, Sahil Singla, and Soheil Feizi. Hard imagenet: Segmentations for objects with strong spurious cues. InAdvances in Neural Information Processing Systems, 2022

    2022

  28. [36]

    A theory of non-linear feature learning with one gradient step in two-layer neural networks

    Behrad Moniri, Donghwan Lee, Hamed Hassani, and Edgar Dobriban. A theory of non-linear feature learning with one gradient step in two-layer neural networks. InInternational Conference on Machine Learning (ICML), 2024

    2024

  29. [37]

    The generalization error of max-margin linear classifiers: Benign overfitting and high dimensional asymptotics in the overparametrized regime

    Andrea Montanari, Feng Ruan, Youngtak Sohn, and Jun Yan. The generalization error of max-margin linear classifiers: Benign overfitting and high dimensional asymptotics in the overparametrized regime. arXiv preprint arXiv:1911.01544, 2019. 16

    1911 arXiv

  30. [38]

    Universality of empirical risk minimization

    Andrea Montanari and Basil N Saeed. Universality of empirical risk minimization. InConference on Learning Theory, pages 4310–4312. PMLR, 2022

    2022

  31. [39]

    Simplicity bias in 1-hidden layer neural networks

    Depen Morwani, jatin batra, Prateek Jain, and Praneeth Netrapalli. Simplicity bias in 1-hidden layer neural networks. InThirty-seventh Conference on Neural Information Processing Systems, 2023

    2023

  32. [40]

    Tight bounds on the smallest eigenvalue of the neural tangent kernel for deep ReLU networks

    Quynh Nguyen, Marco Mondelli, and Guido Montufar. Tight bounds on the smallest eigenvalue of the neural tangent kernel for deep ReLU networks. InInternational Conference on Machine Learning (ICML), 2021

    2021

  33. [41]

    Analysis of Boolean Functions

    Ryan O’Donnell. Analysis of Boolean Functions. Cambridge University Press, 2014

    2014

  34. [42]

    Gradient starvation: A learning proclivity in neural networks

    Mohammad Pezeshki, Sékou-Oumar Kaba, Yoshua Bengio, Aaron Courville, Doina Precup, and Guil- laume Lajoie. Gradient starvation: A learning proclivity in neural networks. InAdvances in Neural Information Processing Systems, 2021

    2021

  35. [43]

    Finding and fixing spurious patterns with explanations

    Gregory Plumb, Marco Tulio Ribeiro, and Ameet Talwalkar. Finding and fixing spurious patterns with explanations. Transactions on Machine Learning Research, 2022

    2022

  36. [44]

    Complexity matters: Dynamics of feature learning in the presence of spurious correlations

    GuanWen Qiu, Da Kuang, and Surbhi Goel. Complexity matters: Dynamics of feature learning in the presence of spurious correlations. InInternational Conference on Machine Learning (ICML), 2024

    2024

  37. [45]

    On the spectral bias of neural networks

    Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. InProceedings of the 36th International Conference on Machine Learning, 2019

    2019

  38. [46]

    Random features for large-scale kernel machines

    Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. InAdvances in Neural Information Processing Systems, 2007

    2007

  39. [47]

    Early stopping and non-parametric regression: an optimal data-dependent stopping rule.The Journal of Machine Learning Research, 15(1):335–366, 2014

    Garvesh Raskutti, Martin J Wainwright, and Bin Yu. Early stopping and non-parametric regression: an optimal data-dependent stopping rule.The Journal of Machine Learning Research, 15(1):335–366, 2014

    2014

  40. [48]

    Hashimoto, and Percy Liang

    Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. In International Conference on Learning Representations, 2020

    2020

  41. [49]

    An investigation of why overpa- rameterization exacerbates spurious correlations

    Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. An investigation of why overpa- rameterization exacerbates spurious correlations. InProceedings of the 37th International Conference on Machine Learning, volume 119 ofProceedings of Machine Learning Research, pag...

    2020

  42. [50]

    Information-theoretic bias reduction via causal view of spurious correlation

    Seonguk Seo, Joon-Young Lee, and Bohyung Han. Information-theoretic bias reduction via causal view of spurious correlation. InAAAI Conference on Artificial Intelligence, 2022

    2022

  43. [51]

    The pitfalls of simplicity bias in neural networks

    Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. InAdvances in Neural Information Processing Systems, 2020

    2020

  44. [52]

    Salient imagenet: How to discover spurious features in deep learning? In International Conference on Learning Representations, 2022

    Sahil Singla and Soheil Feizi. Salient imagenet: How to discover spurious features in deep learning? In International Conference on Learning Representations, 2022

    2022

  45. [53]

    Generalization error of min-norm interpolators in transfer learning

    Yanke Song, Sohom Bhattacharya, and Pragya Sur. Generalization error of min-norm interpolators in transfer learning. arXiv preprint arXiv:2406.13944, 2024

    2024 arXiv

  46. [54]

    Regularized linear regression: A precise analysis of the estimation error

    Christos Thrampoulidis, Samet Oymak, and Babak Hassibi. Regularized linear regression: A precise analysis of the estimation error. InConference on Learning Theory, pages 1683–1709. PMLR, 2015

    2015

  47. [55]

    Overcoming simplicity bias in deep networks using a feature sieve

    Rishabh Tiwari and Pradeep Shenoy. Overcoming simplicity bias in deep networks using a feature sieve. In Proceedings of the 40th International Conference on Machine Learning, 2023

    2023

  48. [56]

    Overparameterization improves robustness to covariate shift in high dimensions

    Nilesh Tripuraneni, Ben Adlam, and Jeffrey Pennington. Overparameterization improves robustness to covariate shift in high dimensions. InAdvances in Neural Information Processing Systems, 2021. 17

    2021

  49. [57]

    Counterfactual invariance to spurious correlations in text classification

    Victor Veitch, Alexander D’Amour, Steve Yadlowsky, and Jacob Eisenstein. Counterfactual invariance to spurious correlations in text classification. InAdvances in Neural Information Processing Systems, 2021

    2021

  50. [58]

    Introduction to the non-asymptotic analysis of random matrices, page 210–268

    Roman Vershynin. Introduction to the non-asymptotic analysis of random matrices, page 210–268. Cambridge University Press, 2012

    2012

  51. [59]

    High-dimensional probability: An introduction with applications in data science

    Roman Vershynin. High-dimensional probability: An introduction with applications in data science. Cambridge university press, 2018

    2018

  52. [60]

    Noise or signal: The role of image backgrounds in object recognition

    Kai Yuanqing Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. Noise or signal: The role of image backgrounds in object recognition. InInternational Conference on Learning Representations (ICLR), 2021

    2021

  53. [61]

    Zhang, Sen Wu, Christopher Ré, and Weijie J

    Fan Yang, Hongyang R. Zhang, Sen Wu, Christopher Ré, and Weijie J. Su. Precise high-dimensional asymptotics for quantifying heterogeneous transfers.arXiv preprint arXiv:2010.11750, 2023

    2010 arXiv

  54. [62]

    Spurious correlations in machine learning: A survey.arXiv preprint arXiv:2402.12715, 2024

    Wenqian Ye, Guangtao Zheng, Xu Cao, Yunsheng Ma, and Aidong Zhang. Spurious correlations in machine learning: A survey.arXiv preprint arXiv:2402.12715, 2024

    2024

  55. [63]

    Coping with label shift via distributionally robust optimisation

    Jingzhao Zhang, Aditya Krishna Menon, Andreas Veit, Srinadh Bhojanapalli, Sanjiv Kumar, and Suvrit Sra. Coping with label shift via distributionally robust optimisation. InInternational Conference on Learning Representations, 2021

    2021

  56. [64]

    Examining and combating spurious features under distribution shift

    Chunting Zhou, Xuezhe Ma, Paul Michel, and Graham Neubig. Examining and combating spurious features under distribution shift. InInternational Conference on Machine Learning (ICML), 2021

    2021

  57. [65]

    On the relation between accuracy and fairness in binary classification

    Indre Zliobaite. On the relation between accuracy and fairness in binary classification. In2nd ICML Workshop on Fairness, Accountability, and Transparency in Machine Learning (FATML), 2015. 18 A Additional Notation We define a sub-Gaussian random variable according to Proposit...

    2015

  58. [66]

    its covarianceΣ ∈ R2d×2d is invertible, withλmax (Σ) = O (1), λmin (Σ) = Ω(1), and tr(Σ) = 2d

  59. [67]

    orthogonal features

    for z ∼ PZ, the random variableΣ−1/2z has independent, mean-0, unit variance, sub-Gaussian entries. This assumption resembles the requirements A-B in Section 2.2 in [19], where we also included the scaling of the trace. To formally state the equivalent of Theorem 1 and Proposi...

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.