Pith. sign in

REVIEW 4 major objections 4 minor 2 cited by

Upweighting Easy Samples in Fine-Tuning Mitigates Forgetting

T0 review · 4 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A sample-reweighting scheme based only on the frozen pre-trained model's per-example losses is claimed to curb catastrophic forgetting during fine-tuning without any access to pre-training data.

desk verdict A simple, promising sample-weighting trick for forgetting, but the language results are confounded by a lower learning rate and the paper needs same-LR controls before I trust the mechanism. read the letter →

arxiv 2502.02797 v2 pith:DSO37KUN submitted 2025-02-05 cs.LG cs.AIstat.ML

Sunny Sanyal , Hayden Prairie , Rudrajit Das , Ali Kavis , Sujay Sanghavi This is my paper · ORCID
classification cs.LGcs.AIstat.ML
keywords catastrophicforgettingfine-tuningsamplereweightingdata-oblivioussettingpre-trainedlossFLOWlinearmodelanalysislanguage
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Fine-tuning a large pre-trained model on a new task typically erases part of what it already knew, and the standard remedies need the original training data, which is often unavailable. This paper claims that one parameter-free change to the loss fixes most of the damage: assign each fine-tuning example a weight based on the frozen pre-trained model's loss on it, upweighting the easy examples ($w_i = \exp(-f_i(\theta^*)/\tau)$) and downweighting the hard ones. The weights are computed once from the model itself, so the method needs no pre-training data, no replay buffer, and no knowledge of the training recipe, and the temperature $\tau$ is simply the median loss, making it parameter-free in practice. On Gemma 2 2B fine-tuned on MetaMathQA, the paper reports a drop of only about 0.8% on the GSM8K math benchmark relative to standard fine-tuning while keeping about 5.4 more points of average general-capability accuracy, and larger retention gains for ResNet-50 across six image classification datasets. The accompanying linear-model analysis explains the effect as gradient descent stalling along the direction that would overfit the new task, and shows the resulting trajectory is not reachable by any convex averaging of the pre-trained and fine-tuned models.

What carries the argument

The central object is the static sample-weight map $w_i = \exp(-f_i(\theta^*)/\tau)$, with $\tau$ set to the median pre-trained loss so the scheme is parameter-free; it is the minimizer of $g(\pi) = \sum_i \pi_i f_i(\theta^*) + \tau \sum_i \pi_i \log \pi_i$, the negative-entropy-regularized objective that is the exact inverse of DRO's hard-sample weighting. The weight map carries the argument by reshaping the fine-tuning data covariance into $\tilde{\Sigma}' = \mu(I_d - Q)$ (Theorem F.1), a matrix with a deliberately small eigenvalue along $\mathbf{e} - \beta\rho\|\mathbf{e}\|_2\mathbf{e}_\perp$; Theorem 7.2 gives the closed-form iterate $\hat{\theta}_K = \tilde{\theta}^* + (I_d - 2\hat{\eta}\tilde{\Sigma}')^K \mathbf{e}$, and Remark 7.4 shows the top eigenvector of $Q$ is precisely the stall direction along which FLOW stops learning, impeding overfitting to the fine-tuning task.

What would settle it

Run FLOW twice on the same fine-tuning dataset, once with the true per-example pre-trained losses and once with the same multiset of weights randomly permuted across examples; if ImageNet retention stays high in both runs, the per-example ordering is not the mechanism and FLOW is just implicit regularization from non-uniform weighting, whereas a collapse in the shuffled run would confirm the loss ordering is load-bearing. The paper's token-wise ablation (GSM8K 23.73 vs 62.55) is already a partial version of this test, since token-level losses are computed but their ordering turns out to be uninformative for the target task.

Watch

Extended reading notes

Core claim

The central claim is that in the data-oblivious setting, catastrophic forgetting can be controlled entirely in sample space: the fine-tuning objective becomes $\sum_i w_i f_i(\theta)$ with static weights $w_i \propto \exp(-f_i(\theta^*)/\tau)$, derived as the minimizer of an entropy-regularized weighted loss (Proposition 4.3) that inverts the distributionally robust optimization weighting that concentrates on hard samples. The paper proves that for linear models the weighted data covariance is $\tilde{\Sigma}' = \mu(I_d - Q)$, whose smallest eigenvalue lies along the direction $\mathbf{e} - \beta\rho\|\mathbf{e}\|_2 \mathbf{e}_\perp$, and that gradient descent on the weighted loss stalls there, so FLOW's iterates stop short of the fine-tuning optimum $\tilde{\theta}^*$ to which vanilla fine-tuning converges exponentially fast (Theorem 7.2 and Remark 7.4). This trajectory includes a component orthogonal to the pre-trained-to-fine-tuned parameter difference $\mathbf{e}$, so FLOW goes beyond model averaging, and its total error on both tasks is at least as good as the optimally tuned convex average (Remark 7.5 and Theorem F.4). Empirically, the paper reports that FLOW achieves the best average of pre-training and target accuracy against standard fine-tuning, linear probing, $\ell^2$-regularization, WiSE-FT, and LoRA on both vision and language benchmarks, and that adding FLOW's weights to each of those baselines improves the base method's average.

Load-bearing premise

The load-bearing premise is that one number per example — the frozen pre-trained model's loss on that example, measured once at the start — correctly identifies which examples, when learned, preserve pre-trained skills, a signal the paper's own token-wise ablation (GSM8K drops from 62.55 to 23.73) shows can fail at finer granularity.

Editorial extensions

If this is right

  • Retention without reference data: any pre-trained model can be fine-tuned with FLOW using one extra forward pass to compute the weights, since the weights depend only on the frozen model and the new task's data.
  • FLOW is a drop-in amplifier: combining its weights with WiSE-FT, $\ell^2$-regularization, or LoRA improves each base method's average accuracy, showing the sample-space mechanism is orthogonal to parameter- and gradient-space interventions.
  • The retention/accuracy trade-off is explicit and tunable: FLOW deliberately sacrifices accuracy on the hardest fine-tuning samples (those with the highest pre-trained loss), so the temperature percentile directly dials between pre-training retention and target-task performance.
  • The theory identifies where learning stops: if the linear analysis transfers to neural networks through the linear-dynamics link, fine-tuning with FLOW halts precisely along the subspace whose learning would erase pre-trained capabilities, leaving the transferable directions to be learned.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Granularity is the real dial: the paper's token-wise ablation (GSM8K drops from 62.55 to 23.73) shows the same formula flips from protective to destructive when the loss is read per token, so an adaptive-granularity version that weights information-bearing units rather than whole sequences or single tokens is a natural testable extension.
  • The DRO duality the paper exploits suggests a smooth family of recipes: interpolating the weight exponent between FLOW's $\exp(-f_i/\tau)$ and DRO's $\exp(f_i/\tau)$ would give practitioners an explicit curve of retention-versus-accuracy operating points that neither method alone provides.
  • Because weights are computed once from a frozen base model, they can be cached and reused across many fine-tuning runs of the same foundation model, making the method nearly free in workflows that repeatedly adapt one base model.
  • The theory's stall direction is derived for a symmetric two-direction covariance; on real data one could estimate the small-eigenvalue subspace of the weighted covariance directly and test whether restricting fine-tuning to its complement reproduces FLOW's retention, which would extend the mechanism beyond the paper's assumed covariance structure.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes FLOW, a data-oblivious fine-tuning method that assigns sample weights exp(-f_i(theta*)/tau) based on the pre-trained model's per-sample losses, thereby upweighting 'easy' samples. The weighting scheme is derived as the minimizer of an entropy-regularized objective (Proposition 4.3), with tau set to the median loss in practice. The authors present vision and language experiments (ResNets on six datasets; Gemma 2 2B and Llama 3.2 3B on MetaMathQA) comparing against standard fine-tuning, L2 regularization, WiSE-FT, LoRA, and linear probing, and they develop a linear-model theory (Theorem 7.2 and Remark 7.4) claiming that FLOW stalls training in a certain subspace and impedes overfitting. The paper also shows that FLOW can be combined with WiSE-FT, L2 regularization, and LoRA to improve those baselines.

Significance. If the central claim is established, FLOW would be a simple, practically appealing method for the data-oblivious setting, with public code and a reproducible experimental pipeline. The paper honestly reports the hard-sample trade-off (Table 5) and provides a first-principles derivation of the weighting scheme rather than a purely heuristic one. However, the empirical comparison is confounded by unequal learning rates across methods, and the theory is not connected to the practical hyperparameter choice; these issues currently leave the main claim unproven.

major comments (4)
  1. [§5.2, Tables 6 and 7; §7, Theorem 7.2] FLOW's reported gains over standard fine-tuning are confounded by learning rate. In Tables 6 and 7, FLOW uses exactly half the learning rate of standard FT for both LLMs (Gemma 2 2B: 5e-6 vs 1e-5; Llama 3.2 3B: 1e-5 vs 2e-5), and the vision setup shows similar gaps (Table 8, e.g., ResNet-50 CIFAR-10: 5e-4 vs 5e-3). A lower learning rate alone reduces parameter drift and thus forgetting; indeed, L2-Regularization, run at FLOW's learning rate, already achieves GSM8K 62.85 vs FLOW's 62.55 on Gemma with a comparable general-capability average (49.19 vs 49.98, Table 2). Since the method is supposed to work through sample weighting, the experiments must hold the learning rate (or the effective update magnitude) fixed across methods, e.g., by also running standard FT at FLOW's LR or FLOW at standard FT's LR. Without such a control, the central empirical claim is unsupported.
  2. [§7, Remark 7.4] The theoretical analysis does not establish the mechanism for the experimental configuration. Theorem 7.2 compares FLOW at b_eta = 1/(2*mu) with vanilla FT at eta = 1/2, i.e., the two methods already differ by a learning-rate-dependent factor. More importantly, Remark 7.4's 'stalling' conclusion requires beta approaching 0, which corresponds to tau approaching 0 in the weighting formula, whereas the experiments use tau = median(f_i(theta*)). The paper does not connect the beta regime to the mu induced by tau = median, nor report a trajectory analysis for that mu. Please state what beta (or mu) the median-tau choice corresponds to, or analyze the trajectory at the experimentally used tau.
  3. [§5.1 and Table 5] The unified average metric masks a severe degradation on hard samples of the fine-tuning dataset. Table 5 shows that FLOW's accuracy on the top-10% hardest samples (by pre-trained loss) collapses relative to standard FT (CIFAR-10: 86.60 to 30.70; CIFAR-100: 56.40 to 21.30; Stanford Cars: 71.30 to 13.18). Because the 'Average' column in Tables 1 and 2 is a plain average of one pre-training score and one target score, it does not expose this tail behavior. I recommend reporting performance by loss deciles or a tail-aware metric, and discussing the trade-off as a central property of the method rather than a side effect.
  4. [§5, Table 2; §G.2] All reported results are single-seed (seed 42 in Tables 6-8). Many of the headline comparisons are small (e.g., Gemma 2B average: FLOW 49.98 vs L2-Reg 49.19; GSM8K: 62.55 vs 62.85), and without variance estimates it is not possible to tell whether the differences are meaningful. Please provide multiple seeds (at least for the main comparisons) or error bars, or explicitly qualify the conclusions as single-run observations.
minor comments (4)
  1. [§1] In the first paragraph of the Introduction, 'apre-trained' is a typographical error and should read 'a pre-trained'.
  2. [§B] Remark B.1 cites 'Theorem 4.2' but the correct reference is Proposition 4.3.
  3. [§4] The claim that FLOW is 'essentially parameter-free' (Remark 4.2) is slightly misleading: tau is a free parameter in the derivation, and the choice tau = median is a heuristic; Table 11 shows that performance varies strongly with tau.
  4. [§H, Figure 2] Figure 2 compares FLOW with 'random selection' at the same percentage of data, but FLOW trains on all data with weights; the comparison would be clearer if the x-axis were the effective sample size or if both methods used the same total gradient budget.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: FLOW's weighting rule, theory, and evaluations form a self-contained chain.

full rationale

The paper's derivation chain is self-contained. The weighting rule in Algorithm 1 is derived as the minimizer of the entropy-regularized objective in Eq. (1) (Proposition 4.3), and the theoretical claims are direct calculations from that rule: Theorem 7.2 computes the weighted covariance E[exp(-<e,x>^2/tau) x x^T] for the chosen linear model, and the 'stalling' conclusion in Remark 7.4 follows by eigendecomposition of Q, not from any fitted value. The τ = median(f_i(θ*)) prescription is a heuristic set from pre-trained losses only; even the ViT-B/16 deviation to the 80th percentile is data-oblivious, so no evaluation quantity is used to fit the method. The sole overlapping-author citation (Das et al., 2024) is background on hard-sample weighting and is not load-bearing for FLOW's claims. Empirical comparisons are against external benchmarks (IN-1K, GSM8K, commonsense suites) under the same evaluation protocol for FLOW and baselines. The learning-rate mismatch between FLOW and standard FT in Tables 6-8 is a potential experimental-control confound, not a circularity; similarly, the token-wise weighting failure in Table 15 indicates a fragile granularity assumption but does not show that any prediction is equivalent to its input by construction. The paper also flags its own limitations (hard-sample sacrifice in Table 5, and the β→0 regime in Remark 7.4), which is consistent with an honest, non-circular presentation.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The method itself introduces no new entities. It rests on the heuristic that low pre-trained loss identifies samples whose updates preserve pre-training, and on a restricted covariance model in the theory. The only free parameter is the temperature tau, with a default prescription and one tuned exception.

free parameters (2)
  • temperature tau = median of per-sample pre-trained losses; 80th percentile in ViT-B/16 Food101
    Sets the softmax sharpness in w_i = exp(-l_i/tau). The paper calls the algorithm parameter-free, but the default is a heuristic and one experiment tunes it.
  • beta (theory reparameterization) = chosen in (0,1]; tau = 2*beta*(1-rho^2)*||e||^2 / (1-beta^2*rho^2)
    Used in Lemma F.3 to parametrize the trajectory and to argue that stalling can be controlled. It is a reparameterization of tau, not an independent method parameter.
assumptions (4)
  • standard math Pre-training data covariance Sigma satisfies Sigma >= I_d
    Used in Section 7 and Eq. (13) to lower-bound err_1(e theta*); a standard normalization in linear analyses.
  • ad hoc to paper Fine-tuning covariance is restricted to eSigma = I_d + rho(e e_perp^T + e_perp e^T)
    Remark 7.1 states this is the minimal analytically tractable case that separates FLOW from model averaging; the stalling result is not established for general covariances.
  • domain assumption Linear-model dynamics transfer to neural networks under gradient descent
    Footnote 3 invokes Lee et al. (2019a); this is an analogy for transformers, not a proven transfer.
  • domain assumption Pre-trained per-sample losses are a valid proxy for sample easiness w.r.t. preserving pre-training behavior
    Section 4 defines easiness informally via low f_i(theta*); Appendix I.2 shows the premise fails at token granularity (GSM8K collapses to 23.73 with token-wise FLOW).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Upweighting Easy Samples in Fine-Tuning Mitigates Forgetting." pith.science (2026). https://pith.science/paper/DSO37KUN

@misc{pith2026250202797,
  author       = {Pith},
  title        = {Pith review of: Upweighting Easy Samples in Fine-Tuning Mitigates Forgetting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DSO37KUN}},
  note         = {Machine review of arXiv:2502.02797}
}
abstract

Fine-tuning a pre-trained model on a downstream task often degrades its original capabilities, a phenomenon known as "catastrophic forgetting". This is especially an issue when one does not have access to the data and recipe used to develop the pre-trained model. Under this constraint, most existing methods for mitigating forgetting are inapplicable. To address this challenge, we propose a sample weighting scheme for the fine-tuning data solely based on the pre-trained model's losses. Specifically, we upweight the easy samples on which the pre-trained model's loss is low and vice versa to limit the drift from the pre-trained model. Our approach is orthogonal and yet complementary to existing methods; while such methods mostly operate on parameter or gradient space, we concentrate on the sample space. We theoretically analyze the impact of fine-tuning with our method in a linear setting, showing that it stalls learning in a certain subspace which inhibits overfitting to the target task. We empirically demonstrate the efficacy of our method on both language and vision tasks. As an example, when fine-tuning Gemma 2 2B on MetaMathQA, our method results in only a $0.8\%$ drop in accuracy on GSM8K (another math dataset) compared to standard fine-tuning, while preserving $5.4\%$ more accuracy on the pre-training datasets. Our code is publicly available at https://github.com/sanyalsunny111/FLOW_finetuning .

Figures

Figures reproduced from arXiv: 2502.02797 by the authors.

Figure 1
Figure 1. FLOW versus standard fine-tuning (FT) and relevant baselines for a ResNet-50 model pre-trained on ImageNet-1K (from [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Comparison of FLOW with different values of τ and some other baselines with different hyper-parameter values. This plot is for ResNet-50 on the Stanford cars dataset. FLOW’s plot (in red) is with τ = {10, 20, 30, 40, 50} percentile of the per-sample losses. As the name “random selection” implies, we just pick a random subset of the fine-tuning data and train on this subset to limit the drift from the pre-trained mod… view at source ↗
Figure 3
Figure 3. Histograms comparing the sample-wise distribution of weights in sequence-wise re-weighting schema for FLOW and token-wise distribution of weights token-wise re-weighting schema for FLOW. The sequence-wise weight distribution is given on the left, while the token-wise weight distribution is given on the right. While token-wise sample re-weighting performs comparably or slightly worse than sequence-wise sample re-weig… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: FLOW is complementary with model averaging (WiSE-FT) in language modeling. We compare WiSE-FT (Wortsman et al., 2021) with a standard model fine-tuning and with FLOW after fine-tuning Gemma 2 2B on MetaMathQA. We use varying α ∈ [0, 1] for WiSE-FT. The results indicate…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. UniGen-AR: Unifying Visual Generation with Auto-Regressive Modeling

    cs.CV 2026-07 conditional novelty 6.0 of 10

    An MLLM-conditioned next-scale VAR decoder handles 15+ unified visual generation tasks with competitive quality and substantially lower latency than diffusion baselines.

  2. Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learning

    cs.LG 2026-02 conditional novelty 6.0 of 10

    Reweighting SFT tokens by target-to-behavior likelihood ratios before RL improves post-RL accuracy, while raw SFT strength does not predict RL success.

Reference graph

Works this paper leans on

38 extracted references · 26 canonical work pages · cited by 2 Pith papers

  1. [1]

    HellaSwag presents a context followed by several plausible endings, and the model must choose the most appropriate continuation

    HellaSwag(Zellers et al., 2019): A benchmark designed to test commonsense reasoning. HellaSwag presents a context followed by several plausible endings, and the model must choose the most appropriate continuation

    work page 2019

  2. [2]

    exp − ⟨r,z⟩ 2 α ! ⟨r,z⟩ 2 # r+ d−1X j=1 E

    Then, we have: eΣ′ :=E ex∼ eP " exp − ⟨e, ex⟩ 2 α ! exex⊤ # =µ Id −(1−µ 2)ee⊤ −ρ 2(1−µ 2)e⊥e⊤ ⊥ +ρµ 2 ee⊤ ⊥ + e⊥e⊤ , whereµ= α α+2 1/2 = τ τ+2∥e∥ 2 2 1/2 . Proof. Recall that e and e⊥ are orthogonal to each other and both are unit-norm. Suppose {e⊥,3, e⊥,4, . . . ,e⊥,d} is an orthonormal basis for the (d−2) -dimensional subspace of Rd orthogonal to e and ...

    work page 2016

  3. [3]

    ARC Challenge(Clark et al., 2018): A benchmark part of the AI2 reasoning challenge designed to test advanced scientific reasoning and knowledge application. ARC Challenge presents 2,590 multiple-choice science questions drawn from grade 3-9 standardized tests, where each question typically includes a scientific scenario or phenomenon followed by four poss...

    work page 2018

  4. [4]

    URL https://proceedings.mlr.press/ v108/farajtabar20a.html. Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac’h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A. A framewo...

    arXiv 2024

  5. [5]

    SIQA(Sap et al., 2019): A benchmark designed to evaluate social commonsense intelligence and emotional reasoning. SIQA presents a social situation context followed by a question and three possible answers, requiring models to demonstrate an understanding of social interactions, emotional responses, and behavioral implications

    work page 2019

  6. [6]

    Open Book QA(Mihaylov et al., 2018): A benchmark designed to assess understanding of elementary science concepts in an open-book exam format. OBQA presents 5,957 multiple-choice questions paired with a small ”book” of 1,326 core science facts, requiring models to combine these facts with common knowledge to arrive at correct answers

    work page 2018

  7. [7]

    MMLU(Hendrycks et al., 2021a): A benchmark designed to evaluate massive multitask language understanding. MMLU presents approximately 16,000 multiple-choice questions spanning 57 subjects including mathematics, philoso- phy, law, and medicine, requiring models to demonstrate broad knowledge and reasoning capabilities

    work page

  8. [8]

    MBPP(Austin et al., 2021): A benchmark designed to evaluate basic Python programming capabilities. The entire MBPP dataset presents 974 Python programming problems, where each problem includes a natural language task description and three test cases written as assert statements, requiring models to generate functionally correct Python code solutions

    work page 2021

Show all 38 references
  1. [9]

    org/CorpusID:215786151

    URL https://api.semanticscholar. org/CorpusID:215786151. Lee, K., Lee, K., Shin, J., and Lee, H. Over- coming catastrophic forgetting with unlabeled data in the wild.2019 IEEE/CVF International Con- ference on Computer Vision (ICCV), pp. 312–321, 2019b. URL https://api.semanti...

    2019

  2. [10]

    ISBN 9781510860964

    Curran Associates Inc. ISBN 9781510860964. Loshchilov, I. and Hutter, F. Online batch selection for faster training of neural networks.ArXiv, abs/1511.06343,

    arXiv

  3. [11]

    org/CorpusID:5324823

    URL https://api.semanticscholar. org/CorpusID:5324823. Lubana, E. S., Trivedi, P., Koutra, D., and Dick, R. How do quadratic regularizers prevent catastrophic forgetting: The role of interpolation.COLLAS, 2021. Mallya, A. and Lazebnik, S. Packnet: Adding multiple tasks to a si...

    2021

  4. [13]

    Rajasegaran, J., Hayat, M., Khan, S

    URL https://proceedings.mlr.press/ v139/radford21a.html. Rajasegaran, J., Hayat, M., Khan, S. H., Khan, F. S., and Shao, L. Random path selection for continual learning. Advances in neural information processing systems, 32, 2019. 14 Upweighting Easy Samples in Fine-Tuning Mit...

    2019

  5. [14]

    org/CorpusID:245007201

    URL https://api.semanticscholar. org/CorpusID:245007201. Rebuffi, S.-A., Kolesnikov, A., Sperl, G., and Lampert, C. H. icarl: Incremental classifier and representation learning. In2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5533–5542, 2017. doi:...

    2017 arXiv

  6. [15]

    org/CorpusID:12253672

    URL https://api.semanticscholar. org/CorpusID:12253672. Wang, L., Zhang, X., Li, Q., Zhu, J., and Zhong, Y . Coscl: Cooperation of small continual learners is stronger than a big one. InComputer Vision – ECCV 2022: 17th Euro- pean Conference, Tel Aviv, Israel, October 23–27, 2...

    2022 doi

  7. [18]

    important

    URL https://openreview.net/forum? id=Sk7KsfW0-. Yu, L., Jiang, W., Shi, H., Yu, J., Liu, Z., Zhang, Y ., Kwok, J. T., Li, Z., Weller, A., and Liu, W. Metamath: Boot- strap your own mathematical questions for large language models.arXiv preprint arXiv:2309.12284, 2023. Zellers,...

    2016 arXiv

  8. [20]

    Following a similar setup as us, Biderman et al

    effectively mitigates forgetting. Following a similar setup as us, Biderman et al. (2024) fine-tuned language models on MetaMathQA (Yu et al., 2023) and then evaluated the fine-tuned model on several general capability tasks, viz., HellaSwag (Zellers et al., 2019), ARC-c (Clar...

    2024

  9. [22]

    ARC Easy(Clark et al., 2018): A benchmark part of the AI2 reasoning challenge designed to test basic scientific reasoning and knowledge. ARC Easy presents 5,197 multiple-choice science questions drawn from grade 3-9 standardized tests, where each question typically includes a ...

    2018

  10. [24]

    PIQA presents a goal and two possible solutions, requiring models to choose the most appropriate solution that demonstrates an understanding of everyday physical interactions

    PIQA(Bisk et al., 2020): A benchmark designed to evaluate physical commonsense understanding in natural language. PIQA presents a goal and two possible solutions, requiring models to choose the most appropriate solution that demonstrates an understanding of everyday physical i...

    2020

  11. [29]

    GSM8K(Cobbe et al., 2021): A benchmark designed to evaluate multi-step mathematical reasoning capabilities. The GSM8K test set contains 1,000 grade school math word problems, where each problem requires 2-8 steps to solve using basic arithmetic operations (addition, subtractio...

    2021

  12. [30]

    It is a widely used large-scale image classification dataset, consisting of over a million images spanning 1000 classes

    ImageNet-1K(Russakovsky et al., 2015) serves as the pre-training dataset for all our vision base models. It is a widely used large-scale image classification dataset, consisting of over a million images spanning 1000 classes

    2015

  13. [31]

    It consists of 60,000 32x32 color images divided into ten classes, with 6,000 images per class

    CIFAR-10(Krizhevsky, 2009) is a widely used dataset for image classification tasks. It consists of 60,000 32x32 color images divided into ten classes, with 6,000 images per class

    2009

  14. [32]

    This dataset is used for fine-grained image classification tasks

    CIFAR-100(Krizhevsky, 2009) extends CIFAR-10 by providing 100 classes containing 600 images each. This dataset is used for fine-grained image classification tasks

    2009

  15. [33]

    Caltech101(Li et al., 2022) comprises images of a diverse range of objects across 101 categories with diverse set of image classes

    2022

  16. [34]

    This dataset is commonly used for fine-grained image classification and flower recognition tasks

    Flowers102(Nilsback & Zisserman, 2008) comprises 102 categories of flowers, with each category containing between 40 to 258 images. This dataset is commonly used for fine-grained image classification and flower recognition tasks. 30 Upweighting Easy Samples in Fine-Tuning Miti...

    2008

  17. [35]

    It provides a rich resource for fine-grained car classification task

    Cars(Krause et al., 2013) refers to the Stanford Cars dataset, which includes 16,185 images of 196 classes of cars. It provides a rich resource for fine-grained car classification task

    2013

  18. [36]

    This dataset is widely used for fine-grained dog breed classification and recognition tasks

    Dogs(Parkhi et al., 2012) pertains to the Stanford Dogs dataset, containing 20,580 images of 120 breeds of dogs. This dataset is widely used for fine-grained dog breed classification and recognition tasks

    2012

  19. [37]

    learning without forgetting

    Food101(Bossard et al., 2014) is a large-scale dataset for food classification containing 101 categories with 1,000 images per class, commonly used to evaluate models on fine-grained object recognition tasks. 31 Upweighting Easy Samples in Fine-Tuning Mitigates Forgetting H. D...

    2016

  20. [38]

    We use varying α∈[0,1] for WiSE-FT

    with a standard model fine-tuning and with FLOW after fine-tuning Gemma 2 2B on MetaMathQA. We use varying α∈[0,1] for WiSE-FT. The results indicate that combining Wise-FT withFLOWoutperforms vanilla WiSE-FT with standard fine-tuning. 36

  21. [715]

    findings-emnlp.715/

    URL https://aclanthology.org/2023. findings-emnlp.715/. Wightman, R., Touvron, H., and J ´egou, H. Resnet strikes back: An improved training procedure in timm. NEURIPS, workshop. Wolf, T., Debut, L., Sanh, V ., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, ...

    2023 arXiv

  22. [2015]

    org/CorpusID:6546520

    URL https://api.semanticscholar. org/CorpusID:6546520. Aljundi, R., Chakravarty, P., and Tuytelaars, T. Expert gate: Lifelong learning with a network of experts. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7120–7129, 2017. doi: 10.1109/ CVPR....

    2017 arXiv

  23. [2016]

    org/CorpusID:4704285

    URL https://api.semanticscholar. org/CorpusID:4704285. Kleiman, A., Dziugaite, G. K., Frankle, J., Kakade, S., and Paul, M. Soup to go: mitigating forgetting during continual learning with model averaging, 2025. URL https://arxiv.org/abs/2501.05559. Krause, J., Stark, M., Deng...

    2025 arXiv

  24. [2017]

    org/CorpusID:19243534

    URL https://api.semanticscholar. org/CorpusID:19243534. Jung, S., Ahn, H., Cha, S., and Moon, T. Continual learn- ing with node-importance based adaptive group sparse regularization. InProceedings of the 34th International Conference on Neural Information Processing Systems, N...

    2020

  25. [2018]

    org/CorpusID:3652214

    URL https://api.semanticscholar. org/CorpusID:3652214. Xie, S. M., Pham, H., Dong, X., Du, N., Liu, H., Lu, Y ., Liang, P. S., Le, Q. V ., Ma, T., and Yu, A. W. Doremi: Optimizing data mixtures speeds up language model pre- training.Advances in Neural Information Processing Sy...

    2024

  26. [2019]

    org/CorpusID:102353035

    URL https://api.semanticscholar. org/CorpusID:102353035. Panda, A., Isik, B., Qi, X., Koyejo, S., Weissman, T., and Mittal, P. Lottery ticket adaptation: Mitigating destructive interference in llms, 2024.https://arxiv. org/abs/2406.16797, 2024. Parkhi, O. M., Vedaldi, A., and ...

    2024 arXiv

  27. [2020]

    cc/paper_files/paper/2020/file/ aa2a77371374094fe9e0bc1de3f94ed9-Paper

    URL https://proceedings.neurips. cc/paper_files/paper/2020/file/ aa2a77371374094fe9e0bc1de3f94ed9-Paper. pdf. Bossard, L., Guillaumin, M., and Van Gool, L. Food-101 – mining discriminative components with random forests. InEuropean Conference on Computer Vision, 2014. Caccia, ...

    2020 arXiv

  28. [2021]

    org/CorpusID:232427874

    URL https://api.semanticscholar. org/CorpusID:232427874. Ben-Tal, A., Den Hertog, D., De Waegenaere, A., Melen- berg, B., and Rennen, G. Robust solutions of optimization problems affected by uncertain probabilities.Manage- ment Science, 59(2):341–357, 2013. Biderman, D., Porte...

    2013

  29. [2022]

    Katharopoulos, A

    URL https://proceedings.mlr.press/ v162/kang22b.html. Katharopoulos, A. and Fleuret, F. Biased importance sam- pling for deep neural network training.arXiv preprint arXiv:1706.00043, 2017. Katharopoulos, A. and Fleuret, F. Not all samples are created equal: Deep learning with ...

    2017 arXiv

  30. [2023]

    URL https: //doi.org/10.48550/arXiv.2310.06825

    doi: 10.48550/ARXIV .2310.06825. URL https: //doi.org/10.48550/arXiv.2310.06825. Jung, H., Ju, J., Jung, M., and Kim, J. Less-forgetful learning for domain expansion in deep neural net- works. InAAAI Conference on Artificial Intelligence,

    doi arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.