Pith. sign in

REVIEW 5 major objections 6 minor 51 references

Backbone Augmented Training for Adaptations

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that adding carefully selected backbone pre-training data to a small adaptation dataset provably improves the adaptation's asymptotic convergence, and gives an algorithm, ALBAT, for choosing that data.

desk verdict The practical idea is sensible, but the central theory is broken: Prop 4.2 rests on a Hessian identity that contradicts the paper's own definitions. read the letter →

arxiv 2506.04288 v1 pith:EFY7DLN4 submitted 2025-06-04 cs.LG

Jae Wan Park , Junhyeok Kim , Youngjun Jun , Hyunah Ko , Seong Jae Hwang This is my paper
classification cs.LG MSC 62F1268T07
keywords backboneaugmentedtrainingadaptationdataselectionasymptoticerrorcoefficientALBATdiffusionmodelsLoRADreamBooth
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Adaptation fine-tunes a large frozen backbone on a small user-supplied dataset, so scarce data makes training unstable or overfit. This paper tries to show that mixing in a carefully chosen slice of the backbone's own pre-training data can make the adaptation converge to the same optimum, and to do so no slower, and typically faster, than standard adaptation. Two propositions are the load-bearing claims: Proposition 4.1 states that the BAT estimator converges to the adaptation optimum, and Proposition 4.2 gives an explicit inequality involving the selected backbone data, the Hessian of the combined risk, and the backbone augmentation ratio $\gamma$ under which BAT's asymptotic error coefficient is no worse than the unaugmented one. The paper then derives a biased selection score from that condition and packages it as ALBAT, reducing the selection cost to $O(nDL)$. The practical upshot is that if the propositions are correct, users with very few adaptation examples can choose backbone data with a computable criterion instead of guessing, and the reported DreamBooth, LoCon, LoRA, and DoRA experiments support that this improves personalization and language generation.

What carries the argument

The argument is carried by three linked objects. The first is the asymptotic error coefficient $\rho(S)=p\text{-}\lim_{k\to\infty} k\|\hat\theta-\theta^*\|_S^2$, a weighted measure of how quickly an estimator approaches the optimum; comparing $\rho_{\mathrm{bat}|A}$ and $\rho_A$ is how the paper decides whether BAT helps. The second is the BAT risk $R_{\mathrm{bat}|A}$ on the union of selected backbone data and adaptation data, with the requirement that the adapted model initialized from the backbone is the same model used in standard adaptation. The third is the biased selection score $Z(x;S)$, a curvature-adjusted influence-type score; thresholding it as $S(x)=\mathbf{1}\{Z(x;S)>\eta\}$ turns Proposition 4.2's condition into an algorithmic selection rule. Theorem 5.1 then brings the cost down to $O(nDL)$ by applying Bartlett's second identity, a damped Hessian, and the Sherman\textendash Morrison update, so the selection rule is feasible for modern diffusion and transformer adapters.

What would settle it

Train a small model with a synthetic backbone dataset and a disjoint adaptation dataset, compute the three curvature matrices exactly at the trained parameters, and check whether $H_{\mathrm{bat}|A}-H_{\mathrm{bat}}-H_A$ is zero; any nonzero entry in a direction used by Proposition 4.2 breaks the proof. A complementary experiment would select backbone data that violate the Proposition 4.2 inequality, run BAT with AdamW, and measure whether it still matches or beats standard adaptation on held-out loss; if it does, the asymptotic condition is not the operative mechanism for real optimizers.

Watch

Extended reading notes

Core claim

Formally, the paper's central claim is that Backbone Augmented Training is valid and conditionally better than ordinary adaptation. Let $R_{\mathrm{bat}|A}$ be the empirical risk on $D_{\mathrm{bat}|A}=D_{B'}\cup D_A$, let $\hat\theta_{\mathrm{bat}|A}$ be its minimizer, and let $\rho_{\mathrm{bat}|A}(S)=p\text{-}\lim_{k\to\infty} k\|\hat\theta_{\mathrm{bat}|A}-\theta_A^*\|_S^2$ be the asymptotic error coefficient. Proposition 4.1 asserts that $\rho_{\mathrm{bat}|A}$ exists under assumptions A1\textendash A5, so BAT converges to the adaptation optimum. Proposition 4.2 asserts that if the selected backbone data satisfy $\gamma\|(H_{\mathrm{bat}|A})^{-1}\sum_{D_{\mathrm{bat}}}\nabla_\theta L_{\mathrm{bat}|A}\| \le \|(H_{\mathrm{bat}|A}-H_{\mathrm{bat}})^{-1}\sum_{D_A}\nabla_\theta L_{\mathrm{bat}|A}\|+o_P(1)$, then $\rho_{\mathrm{bat}|A}\le \rho_A$, and strictly so unless $\gamma\to 1$. The condition says, roughly, that the curvature-adjusted signal from the added backbone data must not exceed the signal from the original adaptation data. ALBAT implements the resulting biased selection rule with a score function $Z(x;S)$ and a threshold $\eta$ chosen so that the expected selection rate is $\gamma$, reducing Hessian computation from cubic to $O(nDL)$ via a damped Hessian and the Sherman\textendash Morrison formula.

Load-bearing premise

The speedup guarantee stands on an unproved equality among three curvature matrices, that the BAT risk's curvature is exactly the sum of the backbone and adaptation curvatures, together with the simplifying model of the estimator as one Newton step; if either is wrong, the claimed inequality between convergence coefficients is not established.

Editorial extensions

If this is right

  • For any adaptation method whose risks satisfy the paper's regularity assumptions, adding backbone data that meets the Proposition 4.2 condition gives an asymptotic error coefficient no larger than that of standard adaptation, and strictly smaller unless the backbone fraction approaches the whole training set.
  • The $O(nDL)$ selection score makes the criterion computable for log-loss models, so the same ALBAT pipeline can be applied to other diffusion-based and transformer-based adapters, not only the four tested.
  • In the reported data-scarce settings, ALBAT improves DreamBooth and LoCon personalization metrics and raises most commonsense reasoning scores of LoRA and DoRA on LLaMA models, with the improvement generally growing as more backbone data are used for selection.
  • A partially trained surrogate model still produces gains over no augmentation, which means the selection overhead can be reduced by shortening surrogate training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural testable extension is to apply the same score $Z$ to rank the adaptation data itself, not just backbone data: the structure of the condition suggests it measures which points contribute usable curvature to the adaptation optimum, which could inform dataset pruning in scarce-data regimes.
  • The paper's theory uses the Hessian identity $H_{\mathrm{bat}|A}-H_{\mathrm{bat}}=H_A$; a direct finite-model check of that identity would show whether the sufficient condition is practically identifiable, since the identity is exactly what lets ALBAT avoid computing the adaptation Hessian.
  • The external-data experiment hints that distributionally similar substitute data can work when the true backbone data are inaccessible, so extending Proposition 4.2 to cover surrogate backbone distributions would materially widen the method's applicability.
  • Because the error-coefficient comparison is asymptotic, the ranking given by $Z$ should be tested on final held-out performance under AdamW or SGD; if the ranking holds under the actual optimizer, the condition is a practical selection rule rather than only an asymptotic guarantee.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes Backbone Augmented Training (BAT), which augments a small adaptation dataset with selected samples from the backbone's pre-training data, and claims two theoretical results: Proposition 4.1 (validity: the BAT estimator's asymptotic error coefficient exists) and Proposition 4.2 (a condition under which BAT is no worse than standard adaptation). It then introduces ALBAT, a practical selection algorithm based on a biased data-selection score and a DataInf-style complexity reduction to O(nDL), and reports experiments on DreamBooth/LoCon for personalized image generation and on LLAMA-2/3 with LoRA/DoRA for commonsense reasoning. The main theoretical contribution is the claimed proof that, under a stated condition on the Hessian and gradients, BAT's asymptotic error coefficient is no larger than that of ordinary adaptation.

Significance. If the theoretical claims were correct, the paper would provide a principled justification for using backbone data in adaptation, a topic of practical interest. The paper also makes a useful engineering contribution by adapting influence-function-style data selection to large models with a claimed O(nDL) complexity, and it ships a substantial experimental study. However, the central theoretical result is not established: the proof of Proposition 4.2 relies on an unproved and generically false Hessian identity, and the proof of Proposition 4.1 uses an undefined auxiliary function. The experimental section does not directly test the proposition's condition and tunes the augmentation ratio per benchmark. Thus, the paper's main claimed guarantee is unsupported.

major comments (5)
  1. [Supp. A.7, Eqs. (25)–(26); Def. 3.1 and Def. 3.2] The proof of Proposition 4.2 asserts the identity Hbat|A − Hbat = HA without derivation. Under the paper's own definitions, this is generically false. From Definition 3.1, Rbat|A is the average over DA ∪ DB′, so for twice-differentiable losses Hbat|A = (|DA|/k)HA + (|DB′|/k)HB′, while Hbat is the Hessian on DB′ only, so Hbat = HB′. Therefore Hbat|A − Hbat = (|DA|/k)(HA − HB′). By Definition 3.2, |DA|/k → 1−γ (if γ is the backbone proportion) or more generally lies in (0,1), so the difference equals HA only in degenerate cases such as HB′ = 0 and |DA|/k = 1. The claimed reduction in Eq. (26) and the subsequent inequality ρbat|A ≤ ρA therefore do not follow. This is a load-bearing step in the central theorem.
  2. [Supp. A.7, Eqs. (27)–(28)] The proof models the estimators through a single Newton step: θ̂bat|A_k − θA* = (1/k)(Hbat|A)^{-1} Σ ∇Lbat|A and θ̂A_n − θA* = (1/NA)(HA)^{-1} Σ ∇LA. No argument connects these Newton updates to the AdamW/SGD dynamics actually used in the experiments (Sections 6.2 and Appendix B). Without such a connection, the asymptotic error coefficients derived here do not describe the optimizers that produce the reported results, so the theoretical guarantee is not relevant to the empirical evaluation.
  3. [Supp. A.6, Eq. (24)] The proof of Proposition 4.1 introduces a function fB\A satisfying fB(x; θB*) = fA(x, θB*) − fB\A(x, θB*\θA), but fB\A is never defined, its existence is not established, and the notation θB*\θA is not explained. The subsequent lower-semicontinuity argument for fA(xB; θA) depends critically on this entity. As a result, the proof of Proposition 4.1 is incomplete, and the convergence guarantee it claims is not rigorously supported.
  4. [Sec. 5.1, Eqs. (3)–(5)] The selection scheme is imported from [22] and claimed to satisfy the paper's propositions, but no derivation shows that the thresholded score Z(x; S) yields data satisfying the condition in Proposition 4.2. The statement "According to [22], this setting yields an asymptotic error coefficient ρ" and the appeal to Proposition 4.1 do not bridge the gap: [22] analyzes a different objective, and Proposition 4.1 only asserts existence of ρbat|A, not that the selected DB′ satisfies the inequality of Proposition 4.2. The link between the selection algorithm and the theoretical condition is therefore not established.
  5. [Sec. 6.2 and Sec. 6.4] The experimental validation does not test the condition of Proposition 4.2 directly; instead, the augmentation ratio γ is tuned per benchmark (e.g., γ=0.9 for LoRA/DoRA, γ=0.992/0.9993 for HellaSwag, γ=0.95 for DreamBooth), and the ablation in Sec. 6.4 is interpreted through [22] rather than through the paper's own proposition. Consequently, the reported improvements are partly fitted to the benchmarks and do not serve as a parameter-free confirmation of the theory. Reporting error bars or significance tests would also be needed to support the empirical claims.
minor comments (6)
  1. [Def. 3.2] The definition 'n/k → γ' does not specify what n denotes; earlier the paper uses N_A and N_B for dataset sizes, and Definition 3.1 uses k = |Dbat|A|. Please clarify whether n = |DA|, n = |DB′|, or something else, and state the relation between γ and the proportions of DA and DB′ in Dbat|A.
  2. [Prop. 4.2] The condition uses 'Σ_Dbat ∇θLbat|A' but Dbat is not defined; presumably Dbat = Dbat|A. Also, 'for any θ∈(θA∩θB)' is unclear since θA and θB are parameter vectors, not sets. Please rewrite the quantifier precisely.
  3. [Table 1 caption] There is a typo: 'acoross' should be 'across'.
  4. [Prop. 4.1 title] The title 'V alidity of BAT' contains an extra space; should be 'Validity of BAT'.
  5. [Sec. 6.1] The text says 'if γ = 0.95, the sample ratio is 0.5, and the adaptation dataset contains 57 images, we first sample 50% of the images from the backbone dataset and then select 3 images for Dbat.' This conflates the sample ratio with γ and with the selection threshold; please clarify the precise relationship among these quantities.
  6. [References] The Stanford Alpaca dataset is cited twice, as [45] and [48], with identical URLs; please consolidate or differentiate the citations.

Circularity Check

3 steps flagged · score 6.0 of 10

Prop. 4.2's guarantee rests on the unproved identity Hbat|A − Hbat = HA, which contradicts Def. 3.1; benchmark gains are partly fitted via per-benchmark γ.

  1. self definitional [Supp. A.7 (Proof of Proposition 4.2); Def. 3.1]
    "using the fact that Lbat|A → LA∗ by Proposition4.1 and the nature of adaptation regarding (θA\θB), one can derive that Hbat|A − Hbat = HA."

    Under Def. 3.1, Rbat|A is defined as the average over Dbat|A = DB′ ∪ DA, so for a twice-differentiable loss Hbat|A = (|DA|/k)HA + (|DB′|/k)HB′, while Prop. 4.2 defines Hbat as the Hessian on DB′. Therefore Hbat|A − Hbat = (|DA|/k)(HA − HB′) by definition; it equals HA only in degenerate cases (HB′ = 0 and |DA|/k = 1), both incompatible with γ ∈ (0,1). Prop. 4.1 only proves existence of ρbat|A, not Hessian equality. The proof thus replaces the RHS of the condition with the standard adaptation term by assuming the identity that would collapse BAT's risk into the adaptation risk; the claimed comparison ρbat|A ≤ ρA is not derived from the stated condition.

  2. fitted input called prediction [Appendix B (Experimental Details) and Sec. 6.4 (Fig. 3)]
    "The backbone augmentation ratio is fixed at γ = 0.95 for DreamBooth and LoCon, meaning that 5% of the original adaptation data is replaced with backbone data. For LoRA and DoRA, we set γ = 0.9. Exceptionally, for the HellaSwag benchmark, we set γ = 0.992 for DoRA and γ = 0.9993 for LoRA, respectively."

    The main empirical claims are reported for γ values selected on the same benchmarks: Sec. 6.4 first identifies γ = 0.95 as the DreamBooth peak ('at a certain augmentation ratio (γ = 0.95), BAT achieved peak performance across all metrics'), and Appendix B fixes that value for DreamBooth/LoCon, with per-benchmark adjustments for HellaSwag (γ = 0.992/0.9993). The Prop. 4.2 condition is not used to set γ, and the selection threshold η is determined only by E[S(x)] = γ. Consequently the benchmark improvements are not parameter-free predictions of BAT's theory; they are partly fitted to the evaluation sets, so their confirmatory value is reduced.

1 more flagged steps
  1. other [Sec. 6.4 (Backbone Augmentation Ratio Ablation)]
    "According to theoretical and empirical studies on biased selection schemes with toy models, [22] demonstrated that a toy model’s performance is maximized when the proportion of selected and original data reaches an optimal point. Until now, biased schemes were not extended to current large models. Through this experiment, we confirm that our propositions and the ALBAT algorithm, which incorporates a biased selection scheme in BAT, align with the data selection theory."

    ALBAT's biased selection rule is taken from [22] ('Drawing inspiration from data selection studies under weak supervision [22], we propose a biased selection scheme'), and Sec. 6.4 then treats a peak at γ = 0.95 as confirmation that 'our propositions and the ALBAT algorithm ... align with the data selection theory.' This is a self-referential check: the external theory is both the source of the algorithm and the criterion used to interpret the experiment. The experiment does not independently test Props. 4.1/4.2; it reproduces a phenomenon already predicted by the imported theory.

full rationale

The paper's claimed first-principles result is not self-contained: the proof of Prop. 4.2 inserts a Hessian identity that is false under the paper's own definitions, and the algorithm's empirical confirmation uses imported [22] theory as both constructor and validator. The flaw in the Hessian identity is the strongest issue; it means the central inequality ρbat|A ≤ ρA is not established by the given argument. The per-benchmark γ selection means the headline improvements are partly fitted rather than predicted. However, there is no self-citation chain: [22] and [23] are external works, and the ALBAT implementation makes an independent computational contribution. The circularity is partial: one core proof step reduces to a definitional confusion, and one empirical validation loop is self-referential. Score 6.

Assumptions & free parameters 5 free parameters · 6 assumptions · 2 invented entities

The central claims rest on five standard regularity assumptions (A1-A4) plus a novel assumption A5, on the unverified assumption that backbone and adaptation share the loss, and on an ad hoc Newton-dynamics model for training that does not match the empirical SGD/AdamW procedures. The main free parameter γ is tuned per benchmark, and the selection threshold and damping factors are unspecified.

free parameters (5)
  • backbone augmentation ratio γ = 0.95 (DreamBooth/LoCon), 0.9 (LoRA/DoRA), 0.992/0.9993 (HellaSwag DoRA/LoRA)
    The proportion of backbone data in the BAT set; tuned per benchmark and per model. The theory predicts strict improvement only when γ does not approach 1, yet the chosen values are very close to 1.
  • selection threshold η = chosen so that E[S(x)] = γ
    Controls which backbone points are selected; its value is not reported and depends on γ and the score distribution.
  • damping factor λ in Hessian approximation = not specified ('very small number')
    Needed for invertibility in the DataInf approximation; exact value not given, affecting reproducibility.
  • noise sampling count δ = 3
    Used for normalized loss sampling with diffusion models; chosen without analysis.
  • surrogate training steps = 800 (strong), 400/200 (weak)
    Controls surrogate quality; experiments show weaker surrogates still help, but the choice affects the selection scores.
assumptions (6)
  • standard math A1-A4: uniqueness, lower semi-continuity, differentiability, and convexity of backbone and adaptation risks.
    Standard regularity conditions from asymptotic estimation theory, cited from [22] and [21]; not independently verified for the actual deep learning models used.
  • ad hoc to paper A5: For any neighborhood U_n of θA*, RA(θ) − Rbat|A(θ) ≠ RA(θA*) − Rbat|A(θA*; Dbat|A) for any θ ≠ θA*.
    Introduced solely to guarantee a unique minimizer of the composite BAT risk; no independent motivation is given beyond making the proof work.
  • domain assumption Backbone and adaptation share the same loss function L.
    Stated in Sec. 3.1. In practice, DreamBooth uses a diffusion denoising loss and LLaMA cross-entropy; these are different losses, and the theory does not cover the mismatch.
  • domain assumption The adapter function fA is continuous when evaluated on backbone data (x,y) in DB'.
    Assumed implicitly in the proof of Prop 4.1 (Supp. A.6) through an undefined decomposition fB\A; the continuity of the adaptation model on out-of-domain backbone inputs is nontrivial.
  • ad hoc to paper The estimator follows a Newton update: theta_hat_bat|A_k - theta_A* = (1/k)(Hbat|A)^{-1} sum grad Lbat|A.
    Used in the proof of Prop 4.2 (Supp. A.7), but the experiments train with AdamW/SGD, and no equivalence is established.
  • ad hoc to paper There exists a function fB\A satisfying fB(x; theta_B*) = fA(x, theta_B*) - fB\A(x, theta_B*\theta_A).
    Introduced in Eq. 24 of Supp. A.6 without definition or existence proof; load-bearing for the lower semi-continuity argument.
invented entities (2)
  • fB\A
    purpose: Decomposes the backbone model output into an adapter component and a remainder in the proof of Prop 4.1.
    Defined in Supp. A.6 Eq. 24 on the fly; no construction, existence, or properties are given.
  • Backbone augmentation coefficient rho_bat|A
    purpose: Asymptotic convergence rate of the BAT estimator, used to state Props 4.1 and 4.2.
    A new coefficient whose existence is the content of Prop 4.1; it has no external measurable counterpart beyond the paper's own definitions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Backbone Augmented Training for Adaptations." pith.science (2026). https://pith.science/paper/EFY7DLN4

@misc{pith2026250604288,
  author       = {Pith},
  title        = {Pith review of: Backbone Augmented Training for Adaptations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EFY7DLN4}},
  note         = {Machine review of arXiv:2506.04288}
}
read the original abstract

Adaptations facilitate efficient training of large backbone models, including diffusion models for image generation and transformer-based language models. While various adaptation techniques enhance performance with minimal computational resources, limited adaptation data often leads to challenges in training. To address this, we focus on the enormous amount of backbone data used to pre-train the backbone models. We propose Backbone Augmented Training (BAT), a method that leverages backbone data to augment the adaptation dataset. First, we formulate and prove two mathematical key propositions: one establishes the validity of BAT, while the other identifies a condition under which BAT benefits adaptation. Furthermore, we introduce an advanced data selection scheme that satisfies these propositions and present ALBAT algorithm to implement this approach. ALBAT efficiently enhances adaptation training in both personalization and language generation tasks with scarce data.

Figures

Figures reproduced from arXiv: 2506.04288 by the authors.

Figure 1
Figure 1. Framework of our Backbone Augmented Train￾ing. (A) The surrogate model is first trained using the adap￾tation dataset, while the backbone model is pre-trained sepa￾rately. (B) With the help of the surrogate model, backbone data are selected using a selection scheme grounded in our solid mathematical foundation. This approach differentiates itself from traditional adaptation methods that do not use backbone data and … view at source ↗
Figure 2
Figure 2. Application Results of BAT in Personalized Image Generation Task. The comparison is conducted between two approaches: augmenting a text-to-image model personalized via DreamBooth [9] with arbitrary selected backbone data (Random) and utilizing our BAT data selection scheme (BAT). The sample ratio represents the proportion of backbone dataset used for data selection while equal amount of backbone data is selected for… view at source ↗
Figure 3
Figure 3. Results of Backbone Augmentation Ratio Test. This figure displays the benchmark scores on different backbone augmentation ratio γ with identical adaptation dataset. This simply means more adaptation data is replaced with backbone data with BAT as γ increases. Results [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: ALBAT Results for personalization task. Given adapation data, these are the backbone data (LAION dataset) examples retrieved from ALBAT. Since we have added regularization data from DreamBooth as adaptation data as well, ALBAT has chosen data that are similar to the re…
Figure 5
Figure 5. Figure 5: ALBAT Results of commonsense reasoning task. These are the backbone data examples retrieved from ALBAT. Best score data add conducive knowledge to adaptation tasks but worst score data do not show such behavior. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_5.png]
Figure 6
Figure 6. Figure 6: ALBAT with COCO Dataset. This is the same experiment as in [PITH_FULL_IMAGE:figures/full_fig_p025_6.png]
Figure 7
Figure 7. Figure 7: ALBAT Selection on Spine MRI Synthesis. The setting of this experiment is the same as that in [PITH_FULL_IMAGE:figures/full_fig_p025_7.png]
Figure 8
Figure 8. Figure 8: ALBAT Selection on Synthetic Data Augmentation. This is an ablation result with augmentation data generated by a diffusion model. Pre-trained Stable Diffusion is used to generate 100 data points with the prompt a photo of dog and a photo of military pilot. 26 [PITH_FU…
Figure 9
Figure 9. Figure 9: DreamBooth Qualitative Outcomes. These outcomes are gathered from original DreamBooth and BAT. Some of the images do not show acceptable personalization outcome as our experiments assume limited adaptation data setting. This experiment has limited the training steps to…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

51 extracted references · 26 canonical work pages

  1. [22]

    Towards a statistical theory of data selection under weak supervision

    Germain Kolossov, Andrea Montanari, and Pulkit Tandon. Towards a statistical theory of data selection under weak supervision. In The Twelfth International Conference on Learning Representations, 2024

    work page 2024

  2. [1]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwi...

    2020

  3. [2]

    High- resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684–10695, June 2022

    2022

  4. [3]

    The llama 3 herd of models, 2024

    Meta. The llama 3 herd of models, 2024

    2024

  5. [4]

    Scalable diffusion models with transformers, 2023

    William Peebles and Saining Xie. Scalable diffusion models with transformers, 2023

    2023

  6. [5]

    Fast high-resolution image synthesis with latent adversarial diffusion distillation, 2024

    Axel Sauer, Frederic Boesel, Tim Dockhorn, Andreas Blattmann, Patrick Esser, and Robin Rombach. Fast high-resolution image synthesis with latent adversarial diffusion distillation, 2024

    work page 2024

  7. [6]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    arXiv 2021

  8. [7]

    Qlora: Efficient finetuning of quantized llms, 2023

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms, 2023

    2023

Show all 51 references
  1. [8]

    Dora: Weight-decomposed low-rank adaptation

    Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.09353, 2024

    2024 arXiv

  2. [9]

    Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation, 2023

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation, 2023

    2023

  3. [10]

    Bermano, Gal Chechik, and Daniel Cohen-Or

    Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion, 2022

    2022

  4. [11]

    Hyperdreambooth: Hypernetworks for fast personalization of text-to-image models, 2023

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Wei Wei, Tingbo Hou, Yael Pritch, Neal Wad- hwa, Michael Rubinstein, and Kfir Aberman. Hyperdreambooth: Hypernetworks for fast personalization of text-to-image models, 2023

    2023

  5. [12]

    Enhancing dreambooth with lora for generating unlimited characters with stable diffusion, 06 2024

    Rubén Pascual, Adrián Maiza, Mikel Sesma-Sara, Daniel Paternain, and Mikel Galar. Enhancing dreambooth with lora for generating unlimited characters with stable diffusion, 06 2024

    2024

  6. [13]

    Overfitting mechanism and avoidance in deep neural networks, 2019

    Shaeke Salman and Xiuwen Liu. Overfitting mechanism and avoidance in deep neural networks, 2019

    2019

  7. [14]

    White, and Sujay Kumar Jauhar

    Dong-Ho Lee, Jay Pujara, Mohit Sewak, Ryen W. White, and Sujay Kumar Jauhar. Making large language models better data creators, 2023

    2023

  8. [15]

    Nlp evaluation in trouble: On the need to measure llm data contamination for each benchmark, 2023

    Oscar Sainz, Jon Ander Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. Nlp evaluation in trouble: On the need to measure llm data contamination for each benchmark, 2023

    2023

  9. [16]

    Does synthetic data make large language models more efficient?, 2023

    Sia Gholami and Marwan Omar. Does synthetic data make large language models more efficient?, 2023

    2023

  10. [17]

    Aligning instruction tuning with pre-training

    Yiming Liang, Tianyu Zheng, Xinrun Du, Ge Zhang, Xingwei Qu, Xiang Yue, Chujie Zheng, Jiaheng Liu, Lei Ma, Wenhu Chen, et al. Aligning instruction tuning with pre-training. arXiv preprint arXiv:2501.09368, 2025. 10

    2025 arXiv

  11. [18]

    Suchin Gururangan, Ana Marasovi´c, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. Don’t stop pretraining: Adapt language models to domains and tasks, 2020

    2020

  12. [19]

    Improved fine-tuning by better leveraging pre-training data, 2022

    Ziquan Liu, Yi Xu, Yuanhong Xu, Qi Qian, Hao Li, Xiangyang Ji, Antoni Chan, and Rong Jin. Improved fine-tuning by better leveraging pre-training data, 2022

    2022

  13. [20]

    Sdxl photorealistic lora tips: Reflections on training and releasing 10 different models, 2024

    jiwenji. Sdxl photorealistic lora tips: Reflections on training and releasing 10 different models, 2024

    2024

  14. [21]

    A. W. van der Vaart. Asymptotic Statistics, volume 1. Cambridge University Press, 2020

    2020

  15. [23]

    Datainf: Efficiently estimating data influence in lora-tuned llms and diffusion models, 2024

    Yongchan Kwon, Eric Wu, Kevin Wu, and James Zou. Datainf: Efficiently estimating data influence in lora-tuned llms and diffusion models, 2024

    2024

  16. [24]

    Parameter-efficient transfer learning for nlp

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pages 2790–2799. PMLR, 2019

    2019

  17. [25]

    Prefix-tuning: Optimizing continuous prompts for generation

    Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190, 2021

    2021 arXiv

  18. [26]

    Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models

    Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. arXiv preprint arXiv:2106.10199, 2021

    2021

  19. [27]

    Parameter-efficient transfer learning with diff pruning

    Demi Guo, Alexander M Rush, and Yoon Kim. Parameter-efficient transfer learning with diff pruning. arXiv preprint arXiv:2012.07463, 2020

    2012 arXiv

  20. [28]

    Intrinsic dimensionality explains the effectiveness of language model fine-tuning

    Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. arXiv preprint arXiv:2012.13255, 2020

    2012 arXiv

  21. [29]

    Compacter: Efficient low-rank hypercomplex adapter layers

    Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. Advances in Neural Information Processing Systems, 34:1022–1035, 2021

    2021

  22. [30]

    Denoising diffusion implicit models, 2022

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models, 2022

    2022

  23. [31]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020

    2020

  24. [32]

    Navigating text-to-image customization: From lycoris fine-tuning to model evaluation

    Shih-Ying Yeh, Yu-Guan Hsieh, Zhidong Gao, Bernard BW Yang, Giyeong Oh, and Yanmin Gong. Navigating text-to-image customization: From lycoris fine-tuning to model evaluation. In The Twelfth International Conference on Learning Representations, 2023

    2023

  25. [33]

    Dora Zhao, Jerone T. A. Andrews, Orestis Papakyriakopoulos, and Alice Xiang. Position: Measure dataset diversity, don’t just claim it, 2024

    2024

  26. [34]

    Infobatch: Lossless training speed up by unbiased dynamic data pruning

    Ziheng Qin, Kai Wang, Zangwei Zheng, Jianyang Gu, Xiangyu Peng, xu Zhao Pan, Daquan Zhou, Lei Shang, Baigui Sun, Xuansong Xie, and Yang You. Infobatch: Lossless training speed up by unbiased dynamic data pruning. In The Twelfth International Conference on Learning Representati...

    2024

  27. [35]

    Wang, Tianji Yang, James Zou, Yongchan Kwon, and Ruoxi Jia

    Jiachen T. Wang, Tianji Yang, James Zou, Yongchan Kwon, and Ruoxi Jia. Rethinking data shapley for data selection tasks: Misleads and merits. In Forty-first International Conference on Machine Learning, 2024

    2024

  28. [36]

    Optimal sub-sampling with influence functions, 2017

    Daniel Ting and Eric Brochu. Optimal sub-sampling with influence functions, 2017

    2017

  29. [37]

    Less is better: Unweighted data subsampling via influence function, 2021

    Zifeng Wang, Hong Zhu, Zhenhua Dong, Xiuqiang He, and Shao-Lun Huang. Less is better: Unweighted data subsampling via influence function, 2021. 11

    2021

  30. [38]

    A statistical perspective on algorithmic leveraging

    Ping Ma, Michael Mahoney, and Bin Yu. A statistical perspective on algorithmic leveraging. In Eric P. Xing and Tony Jebara, editors,Proceedings of the 31st International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, pages 91–99, 2014

    2014

  31. [39]

    New insights and perspectives on the natural gradient method

    James Martens. New insights and perspectives on the natural gradient method. Journal of Machine Learning Research, 21(146):1–76, 2020

    2020

  32. [40]

    Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, Evan Hubinger, Kamil ˙e Lukoši ¯ut˙e, Karina Nguyen, Nicholas Joseph, Sam McCandlish, Jared Kaplan, and Samuel R. Bowman. Studying large lan...

    2023

  33. [41]

    Optimal subsampling algorithms for big data regressions

    Mingyao Ai, Jun Yu, Huiming Zhang, and HaiYing Wang. Optimal subsampling algorithms for big data regressions. Statistica Sinica, 2021

    2021

  34. [42]

    Laion-5b: An open large-scale dataset for training next generation image-text models, 2022

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. Laion...

    2022

  35. [43]

    The star wars dataverse, 2024

    Me. The star wars dataverse, 2024

    2024

  36. [44]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  37. [46]

    Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

    1905 arXiv

  38. [47]

    An introduction to measure theory, volume 126

    Terence Tao. An introduction to measure theory, volume 126. American Mathematical Soc., 2011

    2011

  39. [48]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023

    2023

  40. [49]

    A question-entailment approach to question answering

    Asma Ben Abacha and Dina Demner-Fushman. A question-entailment approach to question answering. BMC bioinformatics, 20:1–23, 2019

    2019

  41. [50]

    Winogrande: An adversarial winograd schema challenge at scale

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106, 2021

    2021

  42. [51]

    Cohen, and Mirella Lapata

    Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. ArXiv, abs/1808.08745, 2018

    2018 arXiv

  43. [52]

    Star Wars

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabb...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.