Pith. sign in

REVIEW 3 major objections 4 minor 2 cited by

Pretraining sets a per-feature penalty that decides whether fine-tuning reuses or rewrites features.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

In diagonal linear networks, fine-tuning generalization is governed by a tunable per-dimension penalty whose sparsity and pretraining dependence define four regimes and a trade-off between feature reuse and new-feature learning.

T0 review reviewed 2026-08-02 challenge →

load-bearing objection Strong extension of Lippl & Lindsey with a useful regime taxonomy, but the central k_d formula doesn't match its own proof—worth refereeing once that's fixed. the 3 major comments →

arxiv 2602.20062 v2 pith:YKLK3NXO submitted 2026-02-23 cs.LG stat.ML

A Theory of How Pretraining Shapes Inductive Bias in Fine-Tuning

Nicolas Anguita , Francesco Locatello , Andrew M. Saxe , Marco Mondelli , Flavia Mancini , Samuel Lippl , Clementine Domine This is my paper
classification cs.LG stat.ML
keywords pretrainingfine-tuninginductive biasdiagonal linear networksfeature reusegeneralization errorreplica methodinitialization scale
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that after pretraining, fine-tuning a diagonal linear network inherits a coordinate-wise penalty written by the pretraining stage. The size of the penalty on each coordinate is controlled by three initialization knobs — overall weight scale, relative scale between first and second layer, and readout re-initialization scale — together with whether pretraining activated that coordinate. From this penalty the authors prove that every initialization produces an inductive bias on a universal line segment: sparsity preference between ℓ1 and ℓ2, and pretraining dependence between zero and inverse proportionality to the pretrained feature. These extremes define four learning regimes, one of which is new. The paper's practical claim is that a smaller early-layer scale selects the intermediate regime that reuses pretrained features and still learns sparse new ones, which generalizes best when the fine-tuning task draws its features from a subset of pretrained features.

Core claim

The paper's central claim, Theorem 4.1, is that gradient flow through pretraining and fine-tuning converges to the minimizer of a separable penalty Q_k(β)=Σ_d q_{k_d}(β_d) subject to interpolation on the fine-tuning set. The per-coordinate coefficient k_d depends on c_PT, λ_PT, γ_FT and on the pretrained function value β̂_PT,d, so the fine-tuning solution genuinely depends on what was learned. From this formula the paper derives universal ranges ℓ-order∈[1,2], pretraining dependence PD∈[-1,0], and ℓ-order+PD∈[1,2], four regimes including the new intermediate rich/pretraining-dependent one. The practical claim: making the early-layer initialization relatively small (negative λ_PT) while keepi

What carries the argument

The central object is the coordinate-wise penalty Q_k(β)=Σ_d q_{k_d}(β_d), where q_k(z)=(√k/4)(1−√(1+4z²/k)+(2z/√k) arcsinh(2z/√k)). Each coordinate's stiffness k_d carries all of the pretraining information through the identity in Eq. (14); it interpolates between ℓ1-like and ℓ2-like behavior, with the two conserved quantities of the network (absolute scale c_PT and relative scale λ_PT) determining the interpolation. The logarithmic derivative of q_k with respect to |β_FT,d| defines the paper's ℓ-order (sparsity preference), and its logarithmic derivative with respect to |β_PT,d| defines pretraining dependence PD. The theorem works because c_PT and λ_PT are conserved during pretraining, so

Load-bearing premise

The quantitative predictions rest on the replica-symmetric calculation being exact in the high-dimensional limit — the paper itself flags the replica method as non-rigorous — and on pretraining perfectly recovering the pretraining teacher (α_PT≥1), leaving the finite-pretraining regime, where most real fine-tuning happens, outside the theory.

What would settle it

Train a diagonal linear network at D=5000 with the paper's pretrain-then-finetune protocol, sweep α_FT and the three initialization parameters, and compare the measured generalization error to the fixed-point prediction of Proposition 4.2. Any systematic mismatch that does not shrink with D, or a measured (ℓ-order, PD) pair outside the claimed triangle ℓ-order∈[1,2], PD∈[-1,0], would refute the central claim.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Fine-tuning inherits an implicit regularization written by pretraining, so the same fine-tuning data can lead to different solutions — and different generalization — depending only on how the network was initialized before pretraining.
  • The relative scale between layers (λ_PT) is a genuine control knob: negative values push toward the rich pretraining-dependent regime, positive values toward the lazy pretraining-dependent regime.
  • The four regimes have distinct task preferences: full task overlap favors lazy reuse, no overlap favors learning new sparse features, and subset overlap favors the intermediate rich pretraining-dependent regime.
  • Very large absolute scale c_PT or readout scale γ_FT erases the benefit of pretraining, putting the network in a lazy pretraining-independent regime no matter what λ_PT was.
  • The same qualitative signatures appear in ResNets: down-scaling early blocks improves fine-tuning accuracy on CIFAR-100, up-scaling all weights hurts, and smaller readout re-initialization helps at intermediate sample sizes.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Inference: the theory implies an optimal λ_PT that moves with the overlap fraction ρ_shared/ρ_new; practitioners could tune only the early-layer scale and expect the crossovers shown in the paper's generalization curves, but the paper does not turn this into a tuning recipe.
  • Inference: because k_d depends on |β̂_PT,d/c_PT|, coordinates that were weakly represented during pretraining should behave like new coordinates even if technically shared; that predicts a smooth degradation in reuse benefit with pretraining signal strength, which the paper does not analyze because it assumes α_PT≥1.
  • Inference: the per-coordinate penalty view suggests that pretraining data quality and initialization scale are interchangeable to some degree — a poorer pretraining fit could be partially compensated by a more negative λ_PT — an exchange the paper leaves untested.
  • Inference: if the effect survives nonlinearity, it gives a principled reason to use depth-wise scaled initialization in transfer learning rather than global scaling, and suggests that the common choice to reinitialize only the final layer leaves the reuse/refinement balance unexplored.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper develops an analytical theory of pretraining followed by fine-tuning (PT+FT) in diagonal linear networks. Theorem 4.1 characterizes the implicit bias of fine-tuning as a coordinate-wise minimization with penalty Q_k, whose coefficient k_d is claimed to depend on the pretraining initialization (c_PT, λ_PT) and the fine-tuning readout reinitialization scale γ_FT. Proposition 4.2 then uses replica theory to turn this implicit bias into fixed-point equations for the generalization error. The paper identifies four learning regimes—rich/pretraining-independent, lazy/pretraining-dependent, lazy/pretraining-independent, and an intermediate rich/pretraining-dependent regime—and argues that a smaller early-layer scale (negative λ_PT) places the network in the intermediate regime, improving generalization when fine-tuning relies on a subset of pretrained features. The theoretical predictions are compared with simulations on diagonal networks and with ResNet experiments on CIFAR-100.

Significance. If the central derivation were correct, this would be a useful and fairly complete treatment of an important problem. It extends prior work by Lippl & Lindsey from the infinitesimal-scale limit to a full two-parameter initialization plane, and it is one of the few papers to connect an implicit-bias characterization of PT+FT to replica-theoretic generalization curves. The paper is also commendably transparent about the numerical replica solver: Appendix E documents damping, forward/backward continuation, Monte Carlo error bars, and branch-selection diagnostics, and the diagonal-network simulations are compared with the predictions without fitting constants. The four-regime taxonomy and the proposal that relative layer scale controls feature reuse versus refinement are conceptually appealing. However, the manuscript contains internal algebraic inconsistencies in the equations that drive all downstream quantitative claims. These must be resolved before the paper can be judged on its merits.

major comments (3)
  1. [Section 4.1, Eq. (14); Appendix B.1] Theorem 4.1 as stated does not follow from the proof. Appendix B.1 derives c_FT=(λ̃_PT+c_PT)(1+√(1+(β̂_PT,d/c_PT)^2))+γ_FT^2 and then invokes k_d=4c_FT^2. Substitution gives k_d=[2c_PT(1+λ_PT)(1+√(1+(β̂_PT,d/c_PT)^2))+2γ_FT^2]^2, not the expression in Eq. (14), where the γ_FT^2 term appears without the factor of 2. The only way to reconcile Eq. (14) with k=4c^2 would be to set v_FT(0)=γ_FT/√2, which contradicts Eq. (5). Since Eq. (14) is used for the phase portraits (Fig. 2b) and all replica generalization curves (Fig. 3), every γ_FT-dependent quantitative prediction in the paper would need to be recomputed. This is not a presentational issue but an internal algebraic inconsistency in the central theorem.
  2. [Section 4.2, Eq. (20); Appendix C, Eq. (39); Appendix E.2.1] The fixed-point equation for χ is written with a prefactor θ in Eq. (20), but with prefactor θ_0 in Appendix C, Eq. (39), and in the numerical solver described in Appendix E.2.1. These are not equivalent under the closure relations θ=(χ+λ)/α and θ_0=(σ_0^2+p)/α. The reader cannot determine whether Fig. 3 was generated with the equation stated in the main text or with the equation stated in the appendix. Please reconcile the displayed theorem with the derivation and with the code, and state which form was used for the reported curves.
  3. [Abstract and Section 4; Appendix D.5] The abstract and Section 4 describe the generalization error as 'exact', but Proposition 4.2 is conditional on the replica-symmetric ansatz, which the paper itself calls non-rigorous (Section 2 and Appendix C.1), and on the assumption α_PT≥1, i.e. perfect pretraining recovery. The Appendix also lists explicit reservations about the ResNet mapping. These qualifications should be reflected in the abstract and in any statement of exactness; as written, the claims overstate what is proved. This is partly a framing issue, but it is load-bearing because the paper's central advertised contribution is an 'exact expression for the generalization error'.
minor comments (4)
  1. [Abstract (arXiv metadata)] The abstract claims experiments on ResNets trained on CIFAR-100 and SVHN as well as Transformers trained on modular arithmetic tasks, but the full text contains only ResNet experiments on CIFAR-100. Either the missing experiments should be added or the abstract should be corrected.
  2. [Section 6 and Figure 4] The text appears to swap the roles of Fig. 4a and Fig. 4b: the text says 'In Fig. 4a, we observe that a ResNet initialized with a non-standard small value of κ', but the caption assigns κ/λ_PT to Fig. 4b and cP_T to Fig. 4a. Please align the text with the figure.
  3. [Appendix D.3, D.4] There are typographical errors ('discribed') and inconsistent use of symbols (c_FT vs. c_PT in D.4) that should be cleaned up.
  4. [Section 5.2] The phase portrait in Fig. 2b is evaluated at a single representative point β_FT,d=1/√ρ_FT. Since ℓ-order and PD depend on β_FT,d, the portrait may depend on this choice. A brief sensitivity analysis or an explicit statement of this limitation would help.

Circularity Check

0 steps flagged

No significant circularity; the central derivation is self-contained, though Theorem 4.1's proof has an internal factor-of-2 inconsistency that is a correctness issue, not a circularity issue.

full rationale

The paper's central claim is that the PT+FT implicit bias is argmin Q_k with k_d given by Eq. (14), and that generalization error follows from the replica fixed-point equations (Prop. 4.2). The derivation of k_d is an explicit function of initialization parameters and the pretrained function; no parameter in Eq. (14) is fitted to the generalization curves. Proposition 4.2 imports the scalar denoiser and fixed-point equations from Bereyhi & Müller (2018) and Bereyhi et al. (2019), which are external, non-self references; the Monte-Carlo solver in Appendix E is used to evaluate, not fit, the resulting expressions. The regime ranges ℓ-order∈[1,2], PD∈[−1,0], ℓ-order+PD∈[1,2] are proved analytically in Appendix B.2 from the q_k formula and the definition of k_d; they do not assume the theorem's conclusion. The empirical diagonal-network simulations and ResNet experiments test the theory's parameter directions and do not tune theory constants to match data. Self-citations to Lippl & Lindsey (2024), Dominé et al. (2024), and Kunin et al. (2024) are contextual and non-load-bearing: Theorem 4.1 rests on Azulay et al. (2021), an external result, and on the paper's own Appendix B.1 computation; Lippl & Lindsey is used only as a limit case and as the origin of the ℓ-order/PD metrics, with the generalization here derived independently. Thus there is no circular step. I note, however, a separate correctness concern, not a circularity: in Appendix B.1, c_FT is computed as (λ̃_PT+c_PT)(1+sqrt(1+(β_PT/c_PT)^2))+γ_FT^2 and then k_i is set to 4 c_FT^2, which would make the γ_FT^2 term enter with coefficient 2, whereas Theorem 4.1 Eq. (14) has coefficient 1 on γ_FT^2 inside the square. This internal algebraic inconsistency affects the proof of Theorem 4.1 as written, but it is not an equivalence-by-construction or fit-renamed-as-prediction; it is a mathematical correctness issue outside the circularity score.

Axiom & Free-Parameter Ledger

1 free parameters · 6 axioms · 0 invented entities

The paper is mostly analytic; no parameters are fitted to the generalization outcomes. The main ledger items are mathematical and modeling assumptions: validity of the prior implicit-bias result, exactness of the replica method, perfect pretraining, Gaussian high-dimensional design, and a nonstandard reinitialization protocol.

free parameters (1)
  • Phase-portrait evaluation point β_FT,d = 1/sqrt(ρ_FT) with ρ_FT=0.1
    ℓ-order and PD are local elasticities evaluated at a representative fine-tuning coefficient; the phase portrait in Fig. 2b uses this hand-chosen value, so regime boundaries are not global statements about the penalty.
axioms (6)
  • standard math Gradient flow in overparameterized diagonal linear networks converges to the Q_k minimizer (Azulay et al. 2021)
    Invoked in Theorem 4.1; the proof relies on prior work and is not re-derived in full.
  • domain assumption The replica-symmetric ansatz is exact in the high-dimensional limit for estimators with anisotropic penalties Q_k
    Explicitly non-rigorous (Section 2); used in Prop 4.2 to derive the scalar channel and fixed-point equations.
  • domain assumption Pretraining with α_PT ≥ 1 perfectly recovers the teacher β_PT
    Section 3.3; excludes finite-pretraining generalization, misspecified teachers, and the common low-pretraining-data regime.
  • domain assumption Input features are i.i.d. Gaussian with variance 1/D in the D→∞ limit
    Data model in Section 3.3; needed for the R-transform and the replica calculation.
  • ad hoc to paper The post-pretraining rebalancing/reinitialization protocol (Eqs. 4–5) faithfully models fine-tuning
    The protocol sets β_FT(0)=0 by rebalancing positive/negative pathways and reinitializes the readout to γ_FT; this is not standard fine-tuning practice, and the ResNet extrapolation depends on this analogy.
  • domain assumption Fine-tuning task structure follows the spike-and-slab overlap model (Eq. 10)
    Restricts task statistics to sparse-overlap settings; many real transfer settings have dense features or continuous sharing.

reviewed 2026-08-02 · how reviews work

0 comments
Cite this review

Pith. "Pith review of A Theory of How Pretraining Shapes Inductive Bias in Fine-Tuning." pith.science (2026). https://pith.science/paper/YKLK3NXO

@misc{pith2026260220062,
  author       = {Pith},
  title        = {Pith review of: A Theory of How Pretraining Shapes Inductive Bias in Fine-Tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YKLK3NXO}},
  note         = {Machine review of arXiv:2602.20062}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Pretraining and fine-tuning are central stages in modern machine learning systems. In practice, feature learning plays an important role across both stages: deep neural networks learn a broad range of useful features during pretraining and further refine those features during fine-tuning. However, an end-to-end theoretical understanding of how choices of initialization impact the ability to reuse and refine features during fine-tuning has remained elusive. Here we develop an analytical theory of the pretraining fine-tuning pipeline in diagonal linear networks, deriving exact expressions for the generalization error as a function of initialization parameters and task statistics. We find that different initialization choices place the network into four distinct fine-tuning regimes that are distinguished by their ability to support feature learning and reuse and therefore by the task statistics for which they are beneficial. In particular, a smaller initialization scale in earlier layers enables the network to both reuse and refine its features, leading to superior generalization on fine-tuning tasks that rely on a subset of pretraining features. We demonstrate empirically that the same initialization parameters impact generalization in ResNets trained on CIFAR-100 and SVHN as well as Transformers trained on modular arithmetic tasks. Overall, our results demonstrate an alytically how data and network initialization interact to shape fine-tuning generalization, highlighting an important role for the relative scale of initialization across different layers in enabling continued feature learning during fine-tuning.

Figures

Figures reproduced from arXiv: 2602.20062 by Andrew M. Saxe, Clementine Domine, Flavia Mancini, Francesco Locatello, Marco Mondelli, Nicolas Anguita, Samuel Lippl.

Figure 1
Figure 1. Figure 1: Setup. Schematic illustration of the theoretical setup: dependence on the initialization parameters cP T , λP T , and γF T (shown in pink); and dependence on the data parameters ρP T , ρ shared F T , and ρ new F T (shown in grey). diagonal linear networks for infinitesimal initial weights in pretraining and studied the resulting generalization behavior using simulations in a teacher-student setup. Here we … view at source ↗
Figure 2
Figure 2. Figure 2: Implicit bias and learning regimes of fine-tuning. (a) ℓ-order and pretraining dependence jointly define four different learning regimes in PT+FT. Different initialization parameters induce changes between these learning regimes, as indicated by the arrows. (b) We can interpolate between these four regimes (for a color legend, see panel (a)) by shifting the initialization parameters (βˆF T ,d = 1/ √ρF T , … view at source ↗
Figure 3
Figure 3. Figure 3: Generalization curves for different initialization parameters and task parameters. The generalization error E as a function of the data scale αF T . Lines depict replica predictions and points depict the results of our empirical simulations (±2 standard errors). In all cases, ρP T = 0.1. We consider cP T = 10−3 , λP T = 0, and γF T = 0, varying one initialization parameter for each panel. Simulation detail… view at source ↗
Figure 4
Figure 4. Figure 4: ResNet CIFAR-100. Generalization performance as a function of the number of samples and initialization parameters. We vary (a) the absolute scale of initialization by multiplying all weights in the network by cP T , (b) the relative scale of initial￾ization by multiplying the first three blocks of the ResNet by κ (equivalent to λP T ), and (c) the readout initialization by multiply￾ing the readout by γF T … view at source ↗
Figure 5
Figure 5. Figure 5: Comparison of the four fine-tuning learning regimes. We show three task parameter settings: 1) no overlap between pretraining and fine-tuning dimensions; 2) identical pretraining and fine-tuning dimensions; 3) fine-tuning dimension as a subset of pretraining dimensions. We show for different initialization parameter settings: a lazy, pretraining-dependent regime (II, shown in purple), a lazy, pretraining-i… view at source ↗
Figure 6
Figure 6. Figure 6: ResNet experiments on CIFAR-100. Resnet layers before and after fine-tuning (PR Pre and PR Post) as well as their ENSD as a function of the κF T re-initialization. D.2. Experiments varying cP T To implement an equivalent notion of cP T scaling in this architecture, we scale all network parameters by a constant factor κ. In this experiment, the overall scaling is applied during pretraining, while the networ… view at source ↗
Figure 7
Figure 7. Figure 7: ResNet experiments on CIFAR-100. Resnet layers before and after fine-tuning (PR Pre and PR Post) as well as their ENSD as a function of the cP T re-initialization. D.3. Experiments varying γF T To implement an equivalent notion of γF T rescaling in this architecture, we scale the last layer parameter. In this experiment, the overall scaling is applied during fine-tuning, while the network remains unchanged… view at source ↗
Figure 8
Figure 8. Figure 8: ResNet experiments on CIFAR-100. Resnet layers before and after fine-tuning (PR Pre and PR Post) as well as their ENSD as a function of the γF T re-initialization. D.4. Experiments varying cF T For completeness, we include the heuristic proposed by (Lippl & Lindsey, 2024) for inducing a pretraining-dependent rich regime, which consists of rescaling all network weights by a constant cF T < 1 during fine-tun… view at source ↗
Figure 9
Figure 9. Figure 9: ResNet experiments on CIFAR-100. (a) Generalization performance as a function of the number of samples and initalization parameters. We vary cF T . (b) Resnet layers before and after fine-tuning (PR Pre and PR Post) as well as their ENSD as a function of the γF T re-initialization. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. A mathematical theory of balancing relational generalization and memorization

    cs.LG 2026-05 unverdicted novelty 7.0

    Introduces transitive inference with exceptions task and analytically shows kernel ridge regression balances relational generalization and memorization depending on representational geometry, with validation in finetu...

  2. Optimal Representation Size: High-Dimensional Analysis of Pretraining and Linear Probing

    cs.LG 2026-05 unverdicted novelty 6.0

    In high-dimensional analysis, pretrained PCA representations for linear probing generalize best at low dimensionality when pretraining data is plentiful but labeled data scarce, with an exact trade-off showing how muc...

Reference graph

Works this paper leans on

55 extracted references · 15 linked inside Pith · cited by 2 Pith papers

  1. [1]

    Neural networks as kernel learners: The silent alignment effect, 10 2021

    Atanasov, A., Bordelon, B., and Pehlevan, C. Neural networks as kernel learners: The silent alignment effect, 10 2021. URL https://openreview.net/forum?id=1NvflqAdoom

    2021

  2. [2]

    M., Cholakkal, H., Shah, M., Yang, M.-H., and Khan, F

    Awais, M., Naseer, M., Khan, S., Anwer, R. M., Cholakkal, H., Shah, M., Yang, M.-H., and Khan, F. S. Foundation models defining a new era in vision: a survey and outlook. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025

    2025

  3. [3]

    S., Woodworth, B

    Azulay, S., Moroshko, E., Nacson, M. S., Woodworth, B. E., Srebro, N., Globerson, A., and Soudry, D. On the implicit bias of initialization shape: Beyond infinitesimal mirror descent. In International Conference on Machine Learning, pp.\ 468--477. PMLR, 2021

    2021

  4. [4]

    and Montanari, A

    Bayati, M. and Montanari, A. The dynamics of message passing on dense graphs, with applications to compressed sensing. IEEE Transactions on Information Theory, 57 0 (2): 0 764--785, 2011 a

    2011

  5. [5]

    and Montanari, A

    Bayati, M. and Montanari, A. The lasso risk for gaussian matrices. IEEE Transactions on Information Theory, 58 0 (4): 0 1997--2017, 2011 b

    1997

  6. [6]

    and M \"u ller, R

    Bereyhi, A. and M \"u ller, R. R. Maximum-a-posteriori signal recovery with prior information: Applications to compressive sensing. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.\ 4494--4498. IEEE, 2018

    2018

  7. [7]

    R., and Schulz-Baldes, H

    Bereyhi, A., M \"u ller, R. R., and Schulz-Baldes, H. Statistical mechanics of map estimation: General replica ansatz. IEEE Transactions on Information Theory, 65 0 (12): 0 7896--7934, 2019

    2019

  8. [8]

    Incremental learning in diagonal linear networks

    Berthier, R. Incremental learning in diagonal linear networks. Journal of Machine Learning Research, 24 0 (171): 0 1--26, 2023

    2023

  9. [9]

    On the opportunities and risks of foundation models

    Bommasani, R. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021

    Pith/arXiv arXiv 2021

  10. [10]

    Exact learning dynamics of deep linear networks with prior knowledge

    Braun, L., Dominé, C., Fitzgerald, J., and Saxe, A. Exact learning dynamics of deep linear networks with prior knowledge. Advances in Neural Information Processing Systems, 35: 0 6615–6629, 12 2022

    2022

  11. [11]

    and Bach, F

    Chizat, L. and Bach, F. Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss. In Conference on learning theory, pp.\ 1305--1338. PMLR, 2020

    2020

  12. [12]

    On lazy training in differentiable programming

    Chizat, L., Oyallon, E., and Bach, F. On lazy training in differentiable programming. Advances in neural information processing systems, 32, 2019

    2019

  13. [13]

    Ask your distribution shift if pre-training is right for you

    Cohen-Wang, B., Vendrow, J., and Madry, A. Ask your distribution shift if pre-training is right for you. arXiv preprint arXiv:2403.00194, 2024

    Pith/arXiv arXiv 2024

  14. [14]

    C., Anguita, N., Proca, A

    Domin \'e , C. C., Anguita, N., Proca, A. M., Braun, L., Kunin, D., Mediano, P. A., and Saxe, A. M. From lazy to rich: Exact learning dynamics in deep linear networks. arXiv preprint arXiv:2409.14623, 2024

    Pith/arXiv arXiv 2024

  15. [15]

    Edwards, S. F. and Anderson, P. W. Theory of spin glasses. Journal of Physics F: Metal Physics, 5 0 (5): 0 965, 1975

    1975

  16. [16]

    K., Paul, M., Kharaghani, S., Roy, D

    Fort, S., Dziugaite, G. K., Paul, M., Kharaghani, S., Roy, D. M., and Ganguli, S. Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel. Advances in Neural Information Processing Systems, 33: 0 5850--5861, 2020

    2020

  17. [17]

    A theory of multineuronal dimensionality, dynamics and measurement

    Gao, P., Trautmann, E., Yu, B., Santhanam, G., Ryu, S., Shenoy, K., and Ganguli, S. A theory of multineuronal dimensionality, dynamics and measurement. BioRxiv, pp.\ 214262, 2017

    2017

  18. [18]

    R., and Aoi, M

    Giaffar, H., Bux \'o , C. R., and Aoi, M. The effective number of shared dimensions between paired datasets. In International Conference on Artificial Intelligence and Statistics, pp.\ 4249--4257. PMLR, 2024

    2024

  19. [19]

    Characterizing implicit bias in terms of optimization geometry

    Gunasekar, S., Lee, J., Soudry, D., and Srebro, N. Characterizing implicit bias in terms of optimization geometry. In International Conference on Machine Learning, pp.\ 1832--1841. PMLR, 2018

    2018

  20. [20]

    and Verd \'u , S

    Guo, D. and Verd \'u , S. Randomly spread cdma: Asymptotics via statistical physics. IEEE Transactions on Information Theory, 51 0 (6): 0 1983--2010, 2005

    1983

  21. [21]

    Huh, M., Agrawal, P., and Efros, A. A. What makes imagenet good for transfer learning? arXiv preprint arXiv:1608.08614, 2016

    Pith/arXiv arXiv 2016

  22. [22]

    Neural tangent kernel: Convergence and generalization in neural networks

    Jacot, A., Gabriel, F., and Hongler, C. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018

    2018

  23. [23]

    Train on Validation (ToV): Fast data selection with applications to fine-tuning

    Jain, A., Montanari, A., and Sasoglu, E. Train on Validation (ToV): Fast data selection with applications to fine-tuning . arXiv preprint arXiv:2510.00386, 2025

    arXiv 2025

  24. [24]

    S., Dick, R

    Jain, S., Kirk, R., Lubana, E. S., Dick, R. P., Tanaka, H., Grefenstette, E., Rockt \"a schel, T., and Krueger, D. S. Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks. arXiv preprint arXiv:2311.12786, 2023

    Pith/arXiv arXiv 2023

  25. [25]

    A., Sun, Y., Jahagirdar, H., Zhang, Y., Du, R., Sahu, A

    Kang, F., Just, H. A., Sun, Y., Jahagirdar, H., Zhang, Y., Du, R., Sahu, A. K., and Jia, R. Get more for less: Principled data selection for warming up fine-tuning in llms. arXiv preprint arXiv:2405.02774, 2024

    Pith/arXiv arXiv 2024

  26. [26]

    A., Xu, W., Avestimehr, A

    Khajehnejad, M. A., Xu, W., Avestimehr, A. S., and Hassibi, B. Weighted _1 minimization for sparse recovery with prior information. In 2009 IEEE international symposium on information theory, pp.\ 483--487. IEEE, 2009

    2009

  27. [27]

    Koh, P. W. and Liang, P. Understanding black-box predictions via influence functions. In International conference on machine learning, pp.\ 1885--1894. PMLR, 2017

    2017

  28. [28]

    Fine-tuning can distort pretrained features and underperform out-of-distribution

    Kumar, A., Raghunathan, A., Jones, R., Ma, T., and Liang, P. Fine-tuning can distort pretrained features and underperform out-of-distribution. arXiv preprint arXiv:2202.10054, 2022

    Pith/arXiv arXiv 2022

  29. [29]

    Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learning, 06 2024

    Kunin, D., Raventós, A., Dominé, C., Chen, F., Klindt, D., Saxe, A., and Ganguli, S. Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learning, 06 2024. URL https://arxiv.org/abs/2406.06158

    Pith/arXiv arXiv 2024

  30. [30]

    Lampinen, A. K. and Ganguli, S. An analytic theory of generalization dynamics and transfer learning in deep linear networks. arXiv preprint arXiv:1809.10374, 2018

    Pith/arXiv arXiv 2018

  31. [31]

    and Lindsey, J

    Lippl, S. and Lindsey, J. Inductive biases of multi-task learning and finetuning: multiple regimes of feature reuse. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  32. [32]

    and Li, J

    Lyu, K. and Li, J. Gradient descent maximizes the margin of homogeneous neural networks. arXiv preprint arXiv:1906.05890, 2019

    Pith/arXiv arXiv 1906

  33. [33]

    A kernel-based view of language model fine-tuning

    Malladi, S., Wettig, A., Yu, D., Chen, D., and Arora, S. A kernel-based view of language model fine-tuning. In International Conference on Machine Learning, pp.\ 23610--23641. PMLR, 2023

    2023

  34. [34]

    Abide by the law and follow the flow: conservation laws for gradient flows, 12 2023

    Marcotte, S., Gribonval, R., and Peyré, G. Abide by the law and follow the flow: conservation laws for gradient flows, 12 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/hash/c7bee9b76be21146fd592fc2b46614d5-Abstract-Conference.html

    2023

  35. [35]

    M \'e zard, M., Parisi, G., and Virasoro, M. A. Spin glass theory and beyond: An Introduction to the Replica Method and Its Applications, volume 9. World Scientific Publishing Company, 1987

    1987

  36. [36]

    R., Alfano, G., Zaidel, B

    M \"u ller, R. R., Alfano, G., Zaidel, B. M., and de Miguel, R. Applications of large random matrices in communications engineering. arXiv preprint arXiv:1310.5479, 2013

    Pith/arXiv arXiv 2013

  37. [37]

    S., Gunasekar, S., Lee, J., Srebro, N., and Soudry, D

    Nacson, M. S., Gunasekar, S., Lee, J., Srebro, N., and Soudry, D. Lexicographic and depth-sensitive margins in homogeneous and non-homogeneous deep models. In International Conference on Machine Learning, pp.\ 4683--4692. PMLR, 2019

    2019

  38. [38]

    S., Ravichandran, K., Srebro, N., and Soudry, D

    Nacson, M. S., Ravichandran, K., Srebro, N., and Soudry, D. Implicit bias of the step size in linear diagonal neural networks. In International Conference on Machine Learning, pp.\ 16270--16295. PMLR, 2022

    2022

  39. [39]

    B., Zafar, A., Khan, A., and Shahid, A

    Parthasarathy, V. B., Zafar, A., Khan, A., and Shahid, A. The ultimate guide to fine-tuning llms from basics to breakthroughs: An exhaustive review of technologies, research, best practices, applied research challenges and opportunities. arXiv preprint arXiv:2408.13296, 2024

    Pith/arXiv arXiv 2024

  40. [40]

    and Flammarion, N

    Pesme, S. and Flammarion, N. Saddle-to-saddle dynamics in diagonal linear networks. Advances in Neural Information Processing Systems, 36: 0 7475--7505, 2023

    2023

  41. [41]

    Implicit bias of sgd for diagonal linear networks: a provable benefit of stochasticity

    Pesme, S., Pillaud-Vivien, L., and Flammarion, N. Implicit bias of sgd for diagonal linear networks: a provable benefit of stochasticity. Advances in Neural Information Processing Systems, 34: 0 29218--29230, 2021

    2021

  42. [42]

    Rangan, S., Goyal, V., and Fletcher, A. K. Asymptotic analysis of map estimation via the replica method and compressed sensing. Advances in Neural Information Processing Systems, 22, 2009

    2009

  43. [43]

    How do infinite width bounded norm networks look in function space? In Conference on Learning Theory, pp.\ 2667--2690

    Savarese, P., Evron, I., Soudry, D., and Srebro, N. How do infinite width bounded norm networks look in function space? In Conference on Learning Theory, pp.\ 2667--2690. PMLR, 2019

    2019

  44. [44]

    L., and Ganguli, S

    Saxe, A., McClelland, J. L., and Ganguli, S. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. openreview.net, 12 2013. URL https://openreview.net/forum?id=_wzZwKpTDF_9C

    2013

  45. [45]

    M., McClelland, J

    Saxe, A. M., McClelland, J. L., and Ganguli, S. A mathematical theory of semantic development in deep neural networks. Proceedings of the National Academy of Sciences, 116 0 (23): 0 11537--11546, 2019

    2019

  46. [46]

    A theoretical analysis of fine-tuning with linear teachers

    Shachaf, G., Brutzkus, A., and Globerson, A. A theoretical analysis of fine-tuning with linear teachers. Advances in Neural Information Processing Systems, 34: 0 15382--15394, 2021

    2021

  47. [47]

    S., Gunasekar, S., and Srebro, N

    Soudry, D., Hoffer, E., Nacson, M. S., Gunasekar, S., and Srebro, N. The implicit bias of gradient descent on separable data. Journal of Machine Learning Research, 19 0 (70): 0 1--57, 2018

    2018

  48. [48]

    Tahir, J., Ganguli, S., and Rotskoff, G. M. Features are fate: a theory of transfer learning in high-dimensional regression. arXiv preprint arXiv:2410.08194, 2024

    Pith/arXiv arXiv 2024

  49. [49]

    and Sato, I

    Tomihari, A. and Sato, I. Understanding linear probing then fine-tuning language models from ntk perspective. Advances in Neural Information Processing Systems, 37: 0 139786--139822, 2024

    2024

  50. [50]

    and Lu, W

    Vaswani, N. and Lu, W. Modified-cs: Modifying compressive sensing for problems with partially known support. IEEE Transactions on Signal Processing, 58 0 (9): 0 4595--4607, 2010. doi:10.1109/TSP.2010.2051150

    arXiv 2010

  51. [51]

    Limitations of the ntk for understanding generalization in deep learning

    Vyas, N., Bansal, Y., and Nakkiran, P. Limitations of the ntk for understanding generalization in deep learning. arXiv preprint arXiv:2206.10012, 2022

    Pith/arXiv arXiv 2022

  52. [52]

    D., Moroshko, E., Savarese, P., Golan, I., Soudry, D., and Srebro, N

    Woodworth, B., Gunasekar, S., Lee, J. D., Moroshko, E., Savarese, P., Golan, I., Soudry, D., and Srebro, N. Kernel and rich regimes in overparametrized models. In Conference on Learning Theory, pp.\ 3635--3673. PMLR, 2020

    2020

  53. [53]

    How transferable are features in deep neural networks? Advances in neural information processing systems, 27, 2014

    Yosinski, J., Clune, J., Bengio, Y., and Lipson, H. How transferable are features in deep neural networks? Advances in neural information processing systems, 27, 2014

    2014

  54. [54]

    Understanding deep learning requires rethinking generalization

    Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530, 2016

    Pith/arXiv arXiv 2016

  55. [55]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

This paper was first reviewed by deepseek-v4-flash on August 2, 2026.