Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods

Chenlu Ye; Hanning Zhang; Rui Pan; Tong Zhang; Xingyuan Pan; Yifan Hao

REVIEW 3 major objections 5 minor 52 references

Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper proves that a weighted average of pretrained and fine-tuned weights beats the fine-tuned model alone, even on the fine-tuning task, under a sparse-task condition.

desk verdict A genuinely new linear-model theorem for why weight ensembling helps, wrapped in an empirical story that overclaims the LLM connection. read the letter →

arxiv 2506.01901 v1 pith:M4CVSHAW submitted 2025-06-02 cs.AI

Yifan Hao , Xingyuan Pan , Hanning Zhang , Chenlu Ye , Rui Pan , Tong Zhang This is my paper

classification cs.AI

keywords overadaptationmodelensemblingsupervisedfine-tuningcatastrophicforgettingbias-variancetrade-offoverparameterizedlinearregressionbenignoverfitting

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Supervised fine-tuning makes a model specialize to a downstream task, but it also forgets pretraining knowledge and can overfit the fine-tuning data. This paper tries to establish that averaging the weights of the pretrained model and the fine-tuned model does more than recover lost general knowledge: with the right mixing weight, the ensemble has strictly lower excess risk on the fine-tuning task itself, a phenomenon the authors call overadaptation. The theoretical core is an overparameterized linear-regression setting in which pretraining is benign overfitting on a high-dimensional task and fine-tuning lives on a sparse, low-rank task. The paper proves that a weighted average of the two estimators improves both the fine-tuning loss and the combined pretraining-plus-fine-tuning loss, and shows that this matches LLM instruction-following experiments.

What carries the argument

The load-bearing object is the convex interpolated estimator $\hat\theta^\tau_\lambda=(1-\tau)\hat\theta_1+\tau\hat\theta_\lambda$ in an overparameterized linear regression, where $\hat\theta_1$ is the min-norm (ridgeless) pretrained estimator on Task 1 and $\hat\theta_\lambda$ is the ridge-fine-tuned estimator initialized at $\hat\theta_1$. The covariance of Task 2 is assumed sparse: $\tilde p\asymp n$ nonzero small eigenvalues and roughly $p-\tilde p$ zero eigenvalues, so fine-tuning is effective but overfitting is harmful, while Task 1 is high-dimensional with benign overfitting. The proof computes excess-risk approximations for both tasks, shows the loss is dominated by a bias term proportional to $\zeta_2$ (task difference) and a variance term proportional to $\tilde\sigma^2$ (fine-tuning noise), and differentiates through the ensemble weight to show the optimal $\tau$ is $\tau'(\lambda)$. The identity $g'(\tau)$ being negative for $\tau\ge\tau'(\lambda)$ is what makes the ensemble strictly beat its fine-tuned component.

What would settle it

Take an instruction-following fine-tuning setup and measure the spectrum of the fine-tuning feature or Hessian covariance, then sweep the ensemble weight $\tau$ from 0 to 1 on the fine-tuning validation set across several seeds. If the spectrum has no mass of near-zero eigenvalues, or if no $\tau$ beats the best fine-tuned checkpoint on the fine-tuning task, the overadaptation regime described by Theorem 5.1 is absent and the central claim would not transfer.

Watch

Extended reading notes

Core claim

On its own terms, the central discovery is Theorem 5.1: if the fine-tuning task has a sparse covariance structure (many zero eigenvalues, dimension comparable to sample size) and the task difference between pretraining and fine-tuning is the same order as the fine-tuning noise, then ridge-regularized fine-tuning beats ridgeless fine-tuning, and the ensemble $\hat\theta^\tau_\lambda=(1-\tau)\hat\theta_1+\tau\hat\theta_\lambda$ beats the fine-tuned estimator alone. For $0\le\lambda<\lambda'$ and $\tau'(\lambda)\le\tau<1$, the ensemble's excess risk on Task 2 is strictly smaller than that of $\hat\theta_\lambda$; for $\tau'(\lambda)/2\le\tau<1$, the sum of pretraining and fine-tuning excess risks is strictly smaller. The mechanism is a bias-variance trade-off: the pretrained model carries a large bias on the new task but little variance, the ridgeless fine-tuned model has little bias but large variance from fitting noise, and interpolation with the right weight balances the two. Because the fine-tuning task's sparse structure makes overfitting harmful rather than benign, the ensemble outperforms even the model trained specifically for that task.

Load-bearing premise

The theorem's conditions require that the fine-tuning task have a sparse, low-rank covariance with dimension comparable to sample size and that fine-tuning noise be the same order as the task difference; if instruction-following fine-tuning in real LLMs does not have this eigenvalue structure, the overadaptation regime that makes ensembling beneficial need not appear.

Editorial extensions

If this is right

When the sparse-task conditions hold, early stopping and ridge regularization are not enough: a tuned weight average of the pretrained and fine-tuned models strictly improves downstream excess risk.
Forgetting is reduced for free: with $\tau$ at least $\tau'(\lambda)/2$, the sum of pretraining and fine-tuning losses improves, so a practitioner can pick one mixing weight to serve both goals.
The optimal ensemble weight $\tau'(\lambda)$ is determined by the ratio of fine-tuning noise to task difference, and no additional training is needed once both checkpoints exist.
Empirically, the paper's LLM results indicate the same ordering holds on MT-Bench, MMLU, and Commonsense-QA for Llama-3-8B, Qwen2-7B, and Gemma-2-9B.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable extension the paper leaves implicit is a phase transition in $\tau$: plotting validation error against $\tau$ on tasks with different noise-to-task-difference ratios should show the optimal mixing weight shifting to smaller $\tau$ as fine-tuning noise grows.
The same bias-variance argument should apply to parameter-efficient fine-tuning such as LoRA, where the pretrained weights stay fixed and only low-rank adapters are averaged; this can be tested directly without retraining.
If SFT data are made gradually more diverse so that the fine-tuning covariance becomes dense, Theorem 5.1's conditions no longer hold, predicting that ensembling will stop beating the fine-tuned model on the fine-tuning domain; this is a sharp, checkable boundary for when overadaptation matters.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The theorem is the real thing. Thm 5.1, with the bias-variance decomposition in Lemma 6.1, gives a clean, self-contained proof that interpolating pretrained and ridge/ridgeless estimators can reduce both fine-tuning error and the combined pretraining-plus-fine-tuning error, under a sparse covariance structure. I checked for circularity and did not find any; the conditions are assumptions, not fitted targets. That is a genuine contribution, and the appendix bounds are detailed enough to be checkable. The paper earns its keep on the theory alone.

The main soft spot is the bridge to LLMs. The proof genuinely needs the fine-tuning covariance to have a near-zero block of eigenvalues (p−p̃ zeros, p̃γ≍1). The paper says this can be validated with PyHessian, but no Hessian eigenvalue analysis appears anywhere. That is a missing load-bearing check, and the stress-test note is right: with a continuous spectrum, the Lemma 6.1 bounds change and the overadaptation inequality can fail. So the LLM experiments are suggestive, not confirmatory.

Second, the headline claim that ensembling beats regularization is overstretched. Theorem 5.1 only shows ensemble beats ridge for λ<λ′, i.e., suboptimal ridge. No comparison to optimally tuned ridge. The abstract says ensembling is "more effective" than regularization without that caveat. That should be fixed.

Third, the empirics are thin: one instruction-following dataset (Dolly), MT-Bench as the only downstream benchmark, modest gains (5.68→5.96 on Llama-3), and error bars only for one configuration. The claim of "consistently outperforms" needs variance estimates across the whole table.

Minor but real: there is a citation error (LoRA is Hu et al.; Huang et al. is about continual learning), and a few proof typos (e.g., curly braces in Lpre(ˆθ2}, a stray "4τeσ3" in C.5). These are fixable.

Who is this for? People working on weight averaging, model soups, or the theory of fine-tuning. A serious referee should engage with it; the theorem is worth publishing after the claims are narrowed and the Hessian check either added or dropped. I would send it to review, but I would advise the authors to soften the LLM claims and add the missing validation.

Referee Report

3 major / 5 minor

Summary. The paper studies 'overadaptation' in supervised fine-tuning, presenting empirical evidence on instruction-following LLMs (Llama-3-8B, Qwen2-7B, Gemma-2-9B fine-tuned on Dolly) that weight-averaging the pretrained model with the fine-tuned model improves MT-Bench scores and mitigates forgetting on MMLU and Commonsense-QA. It then develops a theory in over-parameterized linear regression: the pretrained model is a ridgeless estimator on Task 1, fine-tuning is ridge or ridgeless on Task 2, and the ensemble is a convex combination of the two. Under Conditions 1 and 2 (block-diagonal covariances, a sparse spectrum for the fine-tuning task, and specific noise/task-difference scales), Theorem 5.1 claims three sets of strict inequalities on excess risks Lft and Lpre+Lft, showing that a ridge penalty helps and that adding an ensemble with a suitable weight improves both the fine-tuning task and the combined pretraining/fine-tuning objective. Proof sketches and detailed bounds appear in Section 6 and Appendices C-D.

Significance. If the central claims are fully supported, the paper would provide a formal bias-variance explanation for a phenomenon that has so far been understood mainly empirically, and it would give guidance on when weight ensembling after SFT should help. Strengths of the manuscript include a clearly stated conditional theorem with explicit assumptions, a substantial appendix of concentration bounds, and public code and experimental details. The significance is tempered by two gaps: the theorem compares ensembling only against suboptimal ridge penalties rather than the optimal ridge, and the translation from the linear result to LLMs relies on an NTK linearization and a sparse-spectrum condition that the paper asserts but does not verify in the models it experiments on. As a result, the theoretical contribution is a solid conditional result whose scope of application to the empirical setting remains open.

major comments (3)

[Section 5, Theorem 5.1 item 3; Section 6.2, Eq. (11)] The paper's headline claim that ensembling is 'a more effective solution' than regularization is not established by the theorem. Item 3 shows Lft(θ^τ_λ) < Lft(θ_λ) only for 0 ≤ λ < λ' = σ~²/(nζ2), i.e., for ridge penalties strictly below the value that the paper itself identifies as optimal (Appendix C.4 states 'the optimal choice of λ nearly equals λ''). No inequality compares the ensemble to the optimal ridge estimator θ_{λ'}. Consequently, the abstract's statement that 'while regularization techniques aim to address this trade-off, we show that ensembling provides a more effective solution' overreaches; the theorem is consistent with ensembling helping only when the regularizer is misspecified. Please either extend the analysis to cover λ = λ' (or explain the obstruction), or revise the abstract and conclusion to state that the comparison is against suboptimal ridge regularization.
[Section 4, 'The connection between theoretical and empirical results'; Section 5, 'Discussion on the conditions'] The translation of the linear theory to the LLM experiments relies on two unverified assumptions. First, the NTK linearization f(x,ϑ) ≈ f(x,ϑ0)+∇f(x,ϑ0)^T(ϑ−ϑ0) is asserted with the remark that 'fine-tuning results in parameters close to pretraining points,' but no evidence is provided that full SFT of 7-9B parameter models stays in such a regime. Second, Condition 2.2's sparse spectrum (p~≍n, p~γ≍1, and p−p~ zero eigenvalues in eΣ) is said in Section 5 to be validated by PyHessian/Lanczos eigenvalue analysis, but no such analysis appears in the paper or appendices. If the fine-tuning feature covariance has a continuous spectrum without a large zero block, the trace bounds in Lemma 6.1 (especially Eqs. (38)-(39) and (51)-(52)) change, and the overadaptation inequality may fail. Please add the promised Hessian eigenvalue analysis and a direct accuracy check of the linearization, or clearly brand the LLM results as empirical phenomena not covered by the theorem.
[Section 3.1 and Section 3.3, Table 1, Figures 2-3] The paper repeatedly states that ensembling outperforms the fine-tuned model 'even on the fine-tuning domain itself,' but the only downstream evaluation is MT-Bench, a general instruction-following benchmark that is not the same distribution as the Dolly training data. The models are fine-tuned on Dolly, and no held-out Dolly split is used. This weakens the empirical grounding of the overadaptation phenomenon as stated; please either evaluate on a held-out portion of the fine-tuning distribution or rephrase the claim as 'on the evaluated instruction-following benchmark.'

minor comments (5)

[Section 4, Eq. (5)] Equation (5) appears to be missing the term +θ1; the ridge solution should be θλ = θ1 + eX^T(eX eX^T + nλI)^{-1}(eY − eX θ1), as written correctly in Appendix C.2.
[Section 2, Related Works] The text attributes the proposal of LoRA to 'Huang et al. (2021)', but the cited Huang et al. (2021) reference is a continual-learning paper; the correct LoRA reference is Hu et al. (2021), which is already in the bibliography.
[Section 5, Condition 2.3] Condition 2.3 contains a stray double comma ('ζ1 = O(n^{-ξ}), , ζ2 ≍ ...'); please clean up the punctuation.
[Appendix B, 'Empirical Validation for Theorem 5.1'] The simulation setting text is inconsistent: it lists (a) γ=n^{-1.0}, n=40 and (c) γ=n^{-1.0}, n=40, while the captions of Figures 4 and 5 list (a) γ=0.025, n=40; (b) γ=0.004, n=40; (c) γ=0.017, n=60; (d) γ=0.0022, n=60. Please correct the text and make explicit that the simulated covariances exactly satisfy Condition 2.2, so the simulations validate the theorem but not the LLM bridge.
[Section 6.1, Lemma 6.1] The statement of Lemma 6.1 uses '≈' without an explicit error tolerance; the appendix provides bounds, but the main-text proof sketch would be clearer if the approximation were defined as holding up to lower-order terms that are negligible under Condition 2.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: Theorem 5.1 is proved from stated conditions, and the empirical results are not used as inputs to the proof.

full rationale

The central claim, Theorem 5.1, is a conditional mathematical statement proved in Appendix C from Conditions 1 and 2 and standard concentration results (Bartlett et al. 2019; Koltchinskii and Lounici 2017; Vershynin 2018). The excess-risk expressions in Lemma 6.1 are derived from the model, and the inequalities such as Lft(θhatτλ) < Lft(θhatλ) follow by differentiating those expressions; the thresholds τ'(λ) and λ' are computed quantities, not fitted to the LLM experiments. The empirical Section 3 is presented as motivation and independent observation, and no parameter of the theorem is fitted to the LLM data. The cited works with overlapping authorship (Lin et al. 2023, 2024; Hao et al. 2024) are contextual and are not used to justify the theorem; no uniqueness or modeling assumption is imported from those citations. The only weakly supported bridge to LLMs is the NTK linearization and the asserted but not shown Hessian-based validation of Condition 2, but an unverified assumption is a correctness risk, not circularity. Hence no step in the derivation chain reduces to its own inputs.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

No invented entities and no fitted parameters inside the theorem itself. The central claim rests on a set of domain assumptions about the structure of pretraining and fine-tuning tasks, and the empirical support rests on hyperparameter selection. The theory is conditional on those assumptions, which is a modeling choice rather than evidence of circularity.

free parameters (4)

penalty coefficient λ in LLM experiments = searched over {1e-3, 2e-3, 5e-3, 1e-2}
Selected on a custom validation set that mimics MT-Bench; Table 1 reports the resulting best values without stating them.
ensemble weight τ in LLM experiments = searched over {0.1,...,0.9}
Chosen on validation set; figures show tradeoff curves. The central empirical claim depends on a favorable τ selection.
learning rate in LLM fine-tuning = searched over {1e-6, 2e-6, 5e-6}
Part of the hyperparameter search for fine-tuning on Dolly.
ridge λ in simulations = 0.0001 (best) and 1e-7 (small)
Appendix B tunes λ to the best excess risk to validate Theorem 5.1; selection strengthens the empirical validation but not the theorem.

assumptions (6)

domain assumption Two-task linear model: y = x^Tθ + noise, with θ = θc + α1 and θtilde = θc + α2 (Section 4).
Assumes tasks share a common component θc and differ by isotropic random components; this structure drives the bias-variance tradeoff. No independent evidence is given that LLM tasks share this decomposition.
domain assumption Condition 1 and Condition 2 eigenvalue structures (Section 5).
The proof only works when Task 1 has a handful of large eigenvalues and Task 2 is sparse (ptilde≍n, ptildeγ≍1, ptilde zero eigenvalues). These are chosen to model pretraining and instruction-tuning, but not verified on the actual models.
domain assumption Ridge regression captures early stopping (cited to Lin & Rosasco 2017 and Lu et al. 2022).
Identifies the no-regularizer estimator with overfitting and ridge with early stopping; the equivalence is borrowed from prior work.
domain assumption NTK/linearity of fine-tuning: f(x,ϑ)≈f(x,ϑ0)+∇ϑf(x,ϑ0)^T(ϑ-ϑ0) (Section 4).
Requires fine-tuning to stay in a locally linear regime; modern LLM fine-tuning may violate this.
standard math Benign overfitting for Task 1 from Bartlett et al. 2019; standard concentration lemmas.
Borrowed from cited literature and used throughout Appendix C.
standard math Subgaussian features and noise (Section 4, Assumptions 1-3).
Standard in high-dimensional statistics; constrains the data generating process.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods." pith.science (2026). https://pith.science/paper/M4CVSHAW

@misc{pith2026250601901,
  author       = {Pith},
  title        = {Pith review of: Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/M4CVSHAW}},
  note         = {Machine review of arXiv:2506.01901}
}

read the original abstract

Supervised fine-tuning (SFT) on domain-specific data is the dominant approach for adapting foundation models to specialized tasks. However, it has been observed that SFT models tend to forget knowledge acquired during pretraining. In vision models, ensembling a pretrained model with its fine-tuned counterpart has been shown to mitigate this issue. In this work, we demonstrate that the same holds for language models, and, more strikingly, we observe an overadaptation phenomenon: the ensemble model not only retains general knowledge from the foundation model but also outperforms the fine-tuned model even on the fine-tuning domain itself. Despite the empirical success of ensembling, a theoretical understanding of its benefits remains underexplored. We develop a formal theoretical analysis of the overadaptation phenomenon. Ensembling mitigates this by balancing two primary sources of error: bias, caused by insufficient fine-tuning, and variance, introduced by overfitting to fine-tuning data. While regularization techniques aim to address this trade-off, we show that ensembling provides a more effective solution. We analyze this phenomenon in over-parameterized linear settings and demonstrate that interpolating between pretrained and fine-tuned weights significantly improves performance. These findings offer theoretical justification for the observed advantages of model ensembling, supported by empirical experiments consistent with our analysis.

Figures

Figures reproduced from arXiv: 2506.01901 by the authors.

**Figure 1.** Early Stop Experiments: The performance on MTBench when the training epoch increases. We conduct vanilla fine-tuning and DiffNorm-Penalty fine-tuning with Llama-3-8B on Dolly dataset. MT-Bench (Zheng et al., 2023) with single-answer grading. This benchmark prompts conversational assistants with challenging multi-turn open-ended questions and utilizes “LLM-as-a-judge” for evaluation, which comprises 80 questions, … view at source ↗

**Figure 2.** Experiments on Commonsense-QA and MT-bench: The performance tradeoff on MT-Bench and Commonsense-QA for ensemble methods and fine-tune methods. The ensemble methods are based on the pre-trained model and the DiffNorm-Penalty model, where the results of different τ values and seeds are presented here. We use Llama-3-8B in our experiments. where for each model, we compare the vanilla fine-tuning approach to variants… view at source ↗

**Figure 3.** Experiments on MMLU and MT-bench: The performance tradeoff on MT-Bench and MMLU for ensemble methods and fine-tune methods. The ensemble methods are based on the pre-trained model and the DiffNorm-Penalty model, where results of different τ values and seeds are presented here. We use Llama3-8B in our experiments. Ensemble improves trade-off between pre-training and fine-tuning tasks. We also evaluate the trade-off… view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Performance of Ensemble with dimension p = 104 15 [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]

**Figure 5.** Figure 5: Performance of Ensemble with dimension p = 104 16 [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

52 extracted references · 24 canonical work pages

[1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...
[2]

L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

arXiv 2023
[3]

and Li, Y

Allen-Zhu, Z. and Li, Y. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. arXiv preprint arXiv:2012.09816, 2020

arXiv 2012
[4]

Introducing claude, 2023

Anthropic, A. Introducing claude, 2023

work page 2023
[5]

Ensemble of averages: Improving model selection and boosting performance in domain generalization

Arpit, D., Wang, H., Zhou, Y., and Xiong, C. Ensemble of averages: Improving model selection and boosting performance in domain generalization. Advances in Neural Information Processing Systems, 35: 0 8265--8277, 2022

work page 2022
[6]

L., Long, P

Bartlett, P. L., Long, P. M., Lugosi, G., and Tsigler, A. Benign overfitting in linear regression. arXiv preprint arXiv:1906.11300v3, 2019

arXiv 1906
[7]

L., Tino, P., and Bengio, Y

Brown, G., Wyatt, J. L., Tino, P., and Bengio, Y. Managing diversity in regression ensembles. Journal of machine learning research, 6 0 (9), 2005

work page 2005
[8]

Swad: Domain generalization by seeking flat minima

Cha, J., Chun, S., Lee, K., Cho, H.-C., Park, S., Lee, Y., and Park, S. Swad: Domain generalization by seeking flat minima. Advances in Neural Information Processing Systems, 34: 0 22405--22418, 2021

2021

Show all 52 references

[9]

Dna: Domain generalization with diversified neural averaging

Chu, X., Jin, Y., Zhu, W., Wang, Y., Wang, X., Zhang, S., and Mei, H. Dna: Domain generalization with diversified neural averaging. In International conference on machine learning, pp.\ 4010--4034. PMLR, 2022

2022
[10]

Free dolly: Introducing the world's first truly open instruction-tuned llm, 2023

Conover, M., Hayes, M., Mathur, A., Xie, J., Wan, J., Shah, S., Ghodsi, A., Wendell, P., Zaharia, M., and Xin, R. Free dolly: Introducing the world's first truly open instruction-tuned llm, 2023. URL https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viab...

2023
[11]

Dietterich, T. G. Ensemble methods in machine learning. In International workshop on multiple classifier systems, pp.\ 1--15. Springer, 2000

2000
[12]

Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravankumar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., Gregerson, A., Spataru, A., Rozie...

2024 arXiv
[13]

and Wang, Z

Fan, Z. and Wang, Z. Spectra of the conjugate kernel and neural tangent kernel for linear-width neural networks. Advances in neural information processing systems, 33: 0 7710--7721, 2020

2020
[14]

J., Mirza, M., Xiao, D., Courville, A., and Bengio, Y

Goodfellow, I. J., Mirza, M., Xiao, D., Courville, A., and Bengio, Y. An empirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint arXiv:1312.6211, 2013

2013 arXiv
[15]

Hansen, L. K. and Salamon, P. Neural network ensembles. IEEE transactions on pattern analysis and machine intelligence, 12 0 (10): 0 993--1001, 1990

1990
[16]

On the benefits of over-parameterization for out-of-distribution generalization

Hao, Y., Lin, Y., Zou, D., and Zhang, T. On the benefits of over-parameterization for out-of-distribution generalization. arXiv preprint arXiv:2403.17592, 2024

2024 arXiv
[17]

Measuring massive multitask language understanding, 2021

Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding, 2021. URL https://arxiv.org/abs/2009.03300

2021 arXiv
[18]

J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W

Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models, 2021. URL https://arxiv.org/abs/2106.09685

2021 arXiv
[19]

Continual learning for text classification with information disentanglement based regularization

Huang, Y., Zhang, Y., Chen, J., Wang, X., and Yang, D. Continual learning for text classification with information disentanglement based regularization. arXiv preprint arXiv:2104.05489, 2021

2021 arXiv
[20]

and Lounici, K

Koltchinskii, V. and Lounici, K. Concentration inequalities and moment bounds for sample covariance operators. Bernoulli, pp.\ 110--133, 2017

2017
[21]

and Vedelsby, J

Krogh, A. and Vedelsby, J. Neural network ensembles, cross validation, and active learning. Advances in neural information processing systems, 7, 1994

1994
[22]

Calibrated ensembles can mitigate accuracy tradeoffs under distribution shift

Kumar, A., Ma, T., Liang, P., and Raghunathan, A. Calibrated ensembles can mitigate accuracy tradeoffs under distribution shift. In Uncertainty in Artificial Intelligence, pp.\ 1041--1051. PMLR, 2022 a

2022
[23]

Fine-tuning can distort pretrained features and underperform out-of-distribution

Kumar, A., Raghunathan, A., Jones, R., Ma, T., and Liang, P. Fine-tuning can distort pretrained features and underperform out-of-distribution. arXiv preprint arXiv:2202.10054, 2022 b

2022 arXiv
[24]

On the eigenvalue decay rates of a class of neural-network related kernel functions defined on general domains

Li, Y., Yu, Z., Chen, G., and Lin, Q. On the eigenvalue decay rates of a class of neural-network related kernel functions defined on general domains. Journal of Machine Learning Research, 25 0 (82): 0 1--47, 2024

2024
[25]

and Rosasco, L

Lin, J. and Rosasco, L. Optimal rates for multi-pass stochastic gradient methods. Journal of Machine Learning Research, 18 0 (97): 0 1--47, 2017

2017
[26]

Spurious feature diversification improves out-of-distribution generalization

Lin, Y., Tan, L., Hao, Y., Wong, H., Dong, H., Zhang, W., Yang, Y., and Zhang, T. Spurious feature diversification improves out-of-distribution generalization. arXiv preprint arXiv:2309.17230, 2023

2023 arXiv
[27]

Mitigating the alignment tax of RLHF

Lin, Y., Lin, H., Xiong, W., Diao, S., Liu, J., Zhang, J., Pan, R., Wang, H., Hu, W., Zhang, H., Dong, H., Pi, R., Zhao, H., Jiang, N., Ji, H., Yao, Y., and Zhang, T. Mitigating the alignment tax of RLHF . In The 2024 Conference on Empirical Methods in Natural Language Process...

2024 arXiv
[28]

Sobolev acceleration and statistical optimality for learning elliptic equations via gradient descent

Lu, Y., Blanchet, J., and Ying, L. Sobolev acceleration and statistical optimality for learning elliptic equations via gradient descent. Advances in Neural Information Processing Systems, 35: 0 33233--33247, 2022

2022
[29]

B., Abedsoltan, A., Pandit, P., Belkin, M., and Nakkiran, P

Mallinar, N., Simon, J. B., Abedsoltan, A., Pandit, P., Belkin, M., and Nakkiran, P. Benign, tempered, or catastrophic: A taxonomy of overfitting. arXiv preprint arXiv:2207.06569, 2022

2022 arXiv
[30]

and Cohen, N

McCloskey, M. and Cohen, N. J. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp.\ 109--165. Elsevier, 1989

1989
[31]

and Maclin, R

Opitz, D. and Maclin, R. Popular ensemble methods: An empirical study. Journal of artificial intelligence research, 11: 0 169--198, 1999

1999
[32]

Perrone, M. P. and Cooper, L. N. When networks disagree: Ensemble methods for hybrid neural networks. In How We Learn; How We Remember: Toward An Understanding Of Brain And Neural Systems: Selected Papers of Leon N Cooper, pp.\ 342--358. World Scientific, 1995

1995
[33]

Ensemble based systems in decision making

Polikar, R. Ensemble based systems in decision making. IEEE Circuits and systems magazine, 6 0 (3): 0 21--45, 2006

2006
[34]

Diverse weight averaging for out-of-distribution generalization

Rame, A., Kirchmeyer, M., Rahier, T., Rakotomamonjy, A., Gallinari, P., and Cord, M. Diverse weight averaging for out-of-distribution generalization. arXiv preprint arXiv:2205.09739, 2022

2022 arXiv
[35]

Ensemble-based classifiers

Rokach, L. Ensemble-based classifiers. Artificial intelligence review, 33: 0 1--39, 2010

2010
[36]

Commonsenseqa: A question answering challenge targeting commonsense knowledge, 2019

Talmor, A., Herzig, J., Lourie, N., and Berant, J. Commonsenseqa: A question answering challenge targeting commonsense knowledge, 2019. URL https://arxiv.org/abs/1811.00937

2019 arXiv
[37]

M., Hauth, A., et al

Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

2023 arXiv
[38]

G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahriari, B., Ramé, A., Ferret, J., Liu, P., Tafti, P., Friesen, A., Casbon, M., Ramos, S., Kumar, R., Lan, C

Team, G., Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahriari, B., Ramé, A., Ferret, J., Liu, P., Tafti, P., Friesen, A., Casbon, M., Ramos, S., Kumar, R., Lan, C. L., Jerome, S., Tsitsulin, A., Vieillard, N., Stanczyk, P., ...

2024 arXiv
[39]

Trainable projected gradient method for robust fine-tuning

Tian, J., He, Z., Dai, X., Ma, C.-Y., Liu, Y.-C., and Kira, Z. Trainable projected gradient method for robust fine-tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 7836--7845, 2023

2023
[40]

High-dimensional probability: An introduction with applications in data science, volume 47

Vershynin, R. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018

2018
[41]

Y., Roelofs, R., Gontijo-Lopes, R., Morcos, A

Wortsman, M., Ilharco, G., Gadre, S. Y., Roelofs, R., Gontijo-Lopes, R., Morcos, A. S., Namkoong, H., Farhadi, A., Carmon, Y., Kornblith, S., et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In Internation...

2022
[42]

W., Li, M., Kornblith, S., Roelofs, R., Lopes, R

Wortsman, M., Ilharco, G., Kim, J. W., Li, M., Kornblith, S., Roelofs, R., Lopes, R. G., Hajishirzi, H., Farhadi, A., Namkoong, H., et al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 7959--7...

2022
[43]

Qwen2 technical report, 2024

Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., Dong, G., Wei, H., Lin, H., Tang, J., Wang, J., Yang, J., Tu, J., Zhang, J., Ma, J., Yang, J., Xu, J., Zhou, J., Bai, J., He, J., Lin, J., Dang, K., Lu, K., Chen, K., Yang, K., Li, M....

2024 arXiv
[44]

Yao, Z., Gholami, A., Keutzer, K., and Mahoney, M. W. Pyhessian: Neural networks through the lens of the hessian. In 2020 IEEE international conference on big data (Big data), pp.\ 581--590. IEEE, 2020

2020
[45]

Mathematical Analysis of Machine Learning Algorithms

Zhang, T. Mathematical Analysis of Machine Learning Algorithms. Cambridge University Press, 2023. doi:10.1017/9781009093057

2023 doi
[46]

Why transformers need adam: A hessian perspective

Zhang, Y., Chen, C., Ding, T., Li, Z., Sun, R., and Luo, Z. Why transformers need adam: A hessian perspective. Advances in Neural Information Processing Systems, 37: 0 131786--131823, 2024

2024
[47]

Pytorch fsdp: experiences on scaling fully sharded data parallel

Zhao, Y., Gu, A., Varma, R., Luo, L., Huang, C.-C., Xu, M., Wright, L., Shojanazeri, H., Ott, M., Shleifer, S., et al. Pytorch fsdp: experiences on scaling fully sharded data parallel. arXiv preprint arXiv:2304.11277, 2023

2023 arXiv
[48]

P., Zhang, H., Gonzalez, J

Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023. URL https://arxiv.org/abs/2306.05685

2023 arXiv
[49]

Ensembling neural networks: many could be better than all

Zhou, Z.-H., Wu, J., and Tang, W. Ensembling neural networks: many could be better than all. Artificial intelligence, 137 0 (1-2): 0 239--263, 2002

2002
[50]

@esa (Ref

\@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...
[51]

\@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...
[52]

@open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

[2] [2]

L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

arXiv 2023

[3] [3]

and Li, Y

Allen-Zhu, Z. and Li, Y. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. arXiv preprint arXiv:2012.09816, 2020

arXiv 2012

[4] [4]

Introducing claude, 2023

Anthropic, A. Introducing claude, 2023

work page 2023

[5] [5]

Ensemble of averages: Improving model selection and boosting performance in domain generalization

Arpit, D., Wang, H., Zhou, Y., and Xiong, C. Ensemble of averages: Improving model selection and boosting performance in domain generalization. Advances in Neural Information Processing Systems, 35: 0 8265--8277, 2022

work page 2022

[6] [6]

L., Long, P

Bartlett, P. L., Long, P. M., Lugosi, G., and Tsigler, A. Benign overfitting in linear regression. arXiv preprint arXiv:1906.11300v3, 2019

arXiv 1906

[7] [7]

L., Tino, P., and Bengio, Y

Brown, G., Wyatt, J. L., Tino, P., and Bengio, Y. Managing diversity in regression ensembles. Journal of machine learning research, 6 0 (9), 2005

work page 2005

[8] [8]

Swad: Domain generalization by seeking flat minima

Cha, J., Chun, S., Lee, K., Cho, H.-C., Park, S., Lee, Y., and Park, S. Swad: Domain generalization by seeking flat minima. Advances in Neural Information Processing Systems, 34: 0 22405--22418, 2021

2021