Pith. sign in

REVIEW 4 major objections 6 minor 68 references

Noise Consistency Regularization for Improved Subject-Driven Image Synthesis

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fine-tuning Stable Diffusion with two added consistency losses preserves subject identity and increases background diversity, outperforming DreamBooth on CLIP and DINO scores.

desk verdict A clean idea for prior-consistency in DreamBooth fine-tuning, but the diversity claim rests on a single-subject histogram and needs direct measurement before the headline numbers can be trusted. read the letter →

arxiv 2506.06483 v1 pith:ATWF7R3N submitted 2025-06-06 cs.GR cs.AIcs.CVcs.LGeess.IV

Yao Ni , Song Wen , Piotr Koniusz , Anoop Cherian This is my paper · ORCID
classification cs.GRcs.AIcs.CVcs.LGeess.IV
keywords subject-drivensynthesisconsistencyregularizationStableDiffusionDreamBoothLoRAfine-tuningmultiplicativenoisemodelsbackgrounddiversity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that fine-tuning a diffusion model on a few photos of a subject suffers from two failure modes: underfitting, where the model does not capture the subject's identity, and overfitting, where it memorizes the training photos and loses background diversity. To fix this, it adds two auxiliary losses to the standard fine-tuning objective. The first, prior consistency, forces the fine-tuned model to predict the same diffusion noise as the frozen pretrained model on class images, so the LoRA adapter does not waste capacity re-learning classes it already knows. The second, subject consistency, perturbs the latent codes of subject images with multiplicative noise and requires the model to make consistent predictions on clean and perturbed latents, which the paper claims expands the feasible latent region without changing subject semantics. On the 30-subject benchmark, the combined objective improves CLIP-I and DINO scores over DreamBooth and DCO while keeping CLIP-T competitive.

What carries the argument

The central mechanism is a pair of consistency regularizers stacked on the LoRA fine-tuning objective. The first, $L_{cp}$ (Eq. 4), computes the $\ell_2$ distance between noise predictions of the fine-tuned and frozen pretrained models on identically diffused prior images; since the pretrained model already encodes class semantics, this keeps the adapter from re-fitting class priors and preserves the pretrained data manifold. The second, $L_{cs}$ (Eq. 6), generates perturbed latents via elementwise multiplication $z' = z \odot \varepsilon_m$ with $\varepsilon_m \sim \mathcal{N}(1, \sigma^2 I)$ and enforces consistency between the model's noise predictions on the clean and perturbed latents at the same diffusion timestep and shared diffusion noise. The multiplicative form is chosen because additive noise would interfere with the additive diffusion schedule; the paper supports that choice with an additive-versus-multiplicative qualitative comparison and a KL-divergence histogram on one subject.

What would settle it

Sweep $\sigma$ from 0 to 0.5 in Eq. (5) and measure per-subject CLIP-I and DINO scores; if identity retention systematically drops as $\sigma$ grows, or qualitative samples show clear identity drift, the premise that multiplicative noise preserves semantics while widening the latent distribution is false. A cheaper check is to rerun the KL-divergence analysis of Figure 6 over all 30 subjects rather than the single dog subject.

Watch

Extended reading notes

Core claim

The central claim is that the objective $L = L_s + 0.5 L_{cp} + 0.5 L_{cs}$ (Eq. 7), where $L_{cp}$ enforces predictive consistency between the fine-tuned and pretrained models on prior class images and $L_{cs}$ enforces predictive consistency between clean and multiplicatively noised subject latents, achieves both higher subject fidelity and higher background diversity than DreamBooth-style fine-tuning. The paper reports average CLIP-I 0.792 vs 0.778, DINO 0.634 vs 0.602, with CLIP-T 0.324 vs 0.329, and shows qualitative gains in identity preservation across dogs, cats, candles, and toys. It claims the prior-consistency term eliminates the noise-prediction mismatch that corrupts the learned manifold, and the multiplicative-noise term diversifies latent patterns without conflicting with the additive noise used by the diffusion process itself.

Load-bearing premise

The load-bearing premise is that multiplying a subject image's latent code by Gaussian noise around 1 changes only peripheral variation and leaves the subject's identity intact; if that perturbation actually distorts the subject, the consistency loss would either reduce fidelity or add no real diversity.

Editorial extensions

If this is right

  • LoRA fine-tuning on a handful of subject photos can gain identity fidelity and background diversity at roughly double the training cost, which the paper states explicitly.
  • The prior-consistency loss removes the need to fit ground-truth noise on class images, so prior-preservation capacity in DreamBooth can be redirected toward the subject.
  • The method composes with existing fine-tuning techniques: plugging the losses into DreamBooth and into DCO both improve the reported metrics.
  • The choice of multiplicative rather than additive noise matters: additive noise conflicts with the diffusion process and degrades identity, while multiplicative noise maintains it.
  • The objective yields higher CLIP-I and DINO scores than DreamBooth and DCO while keeping CLIP-T competitive.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same multiplicative-noise consistency trick could plausibly transfer to other parameter-efficient diffusion adapters, such as adapter layers, textual inversion, or SVD-based fine-tuning, wherever a small training set risks overfitting; the paper only demonstrates LoRA.
  • Because $L_{cs}$ broadens the latent distribution, it might reduce the number of prior images required for prior preservation, or even eliminate class priors for well-represented categories; the paper does not test this.
  • The doubling of training cost invites a cheaper variant: applying the consistency losses only on a subset of timesteps or intermediate layers could retain most of the benefit, an option the paper names as future work.
  • A sharper mechanistic test would compare $L_{cs}$ against simply training with multiplicatively noised latents but without the consistency term, to check whether the diversity gain comes from the consistency constraint itself or merely from seeing perturbed examples.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The manuscript proposes two auxiliary consistency losses for subject-driven image synthesis with Stable Diffusion fine-tuned via LoRA. The first, L_cp (Eq. 4), enforces agreement between the fine-tuned and pretrained models on class-prior images, replacing DreamBooth's prior-preservation MSE. The second, L_cs (Eq. 6), enforces agreement between predictions on clean latents and latents perturbed by multiplicative Gaussian noise (Eq. 5). The final objective (Eq. 7) is L_s + 0.5 L_cp + 0.5 L_cs. Experiments on the 30-subject DreamBooth dataset report CLIP-I, CLIP-T, and DINO scores (Table 1) plus qualitative ablations; the authors claim improved fidelity and background diversity relative to DreamBooth and DCO.

Significance. The contribution is a clean, training-time-only regularization scheme that is agnostic to the base fine-tuning method and is demonstrated on both DreamBooth and DCO. The losses are simple and the paper is honest about the roughly doubled training cost. If the diversity claim were substantiated, the method would be a practical improvement. As it stands, however, the quantitative evidence does not establish the main advertised advantage in diversity, and the reported CLIP-I/DINO gains are small and lack variance estimates. The paper's strengths are the formulation and the integration with existing methods; the weakness is the mismatch between claims and evidence.

major comments (4)
  1. [§5.4, Table 1; Abstract] The abstract claims the method outperforms DreamBooth 'in terms of CLIP scores', but Table 1 reports CLIP-T 0.324 for Ours versus 0.329 for DreamBooth, which is lower, and CLIP-T 0.310 for Ours+DCO versus 0.309 for DreamBooth+DCO, essentially tied. The body text (§5.4) correctly says 'competitive CLIP-T', so the abstract should be reworded or the metric-specific claim removed.
  2. [§5.4, Table 1; §5.6, Figure 6] No diversity metric is reported. CLIP-I, CLIP-T, and DINO all measure similarity (to reference images or to the prompt), not the spread or variation of generated backgrounds. The only quantitative diversity evidence is a 1D latent-code histogram for a single dog subject (Figure 6); the KL difference between DreamBooth (0.00793) and the full method (0.00766) is 0.00027, and a marginal histogram of latent values cannot separate background diversity from global brightness or color shifts. A direct diversity measure (e.g., pairwise image/CLIP/LPIPS distances within each prompt, or background-region statistics) and per-subject reporting are needed to support the central 'enhances image diversity' claim.
  3. [§5.4, Table 1] No error bars, multiple seeds, or statistical significance tests are reported. The headline improvements are small (CLIP-I +0.014, DINO +0.032, CLIP-T -0.005) and could be within run-to-run variation for diffusion fine-tuning; without repeated runs or per-subject standard errors, the superiority claim is not quantitatively established.
  4. [§5.6, Eq. (5), Figure 7] The load-bearing premise that multiplicative noise z' = z ⊙ ε_m preserves subject semantics while expanding the latent distribution is supported only by a qualitative comparison on one dog (Figure 7) and the same single-subject latent histogram. Since L_cs (Eq. 6) is the mechanism claimed to add diversity, the paper should provide a quantitative test, for example identity-similarity scores for clean versus noise-modulated latents or ablations across several subjects, before this premise can be accepted.
minor comments (6)
  1. [§5.1 vs. Introduction] Section 5.1 says the dataset is from [47], while the Introduction says the benchmark dataset is from [41]; please make the dataset provenance consistent.
  2. [Figure 3 caption] Figure 3 caption contains 'A photo of a [V] cat candle on a cobblestone street', which appears to be a typo for 'cat on a cobblestone street'.
  3. [Various] There are several typos: 'regulization' (§5), 'DreamBototh' (Figure 6 caption), 'V AEs' (§1), and missing spaces in prompts such as 'a[V] dog'.
  4. [Eq. (3)] Equation (3) is missing a closing parenthesis in the norm: ∥ε−fΔθ(zt,t,τ(pp)∥2 should be ∥ε−fΔθ(zt,t,τ(pp))∥2.
  5. [Figure 6] Figure 6's KL values are computed on 1D histograms, but the bin width and the latent dimension aggregated are not specified; please state them.
  6. [§5.4] The claim in Section 5.4 that results are 'state-of-the-art' is stronger than the evidence, as only DreamBooth and DCO are compared; consider tempering to 'outperforms the evaluated baselines'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the consistency losses are defined independently of the evaluation metrics, and borrowed multiplicative noise is cited as prior evidence rather than as a forced derivation.

full rationale

The paper's chain is: define L_s (Eq. 2), L_cp (Eq. 4) as consistency with the frozen pretrained model on prior images, and L_cs (Eq. 6) as self-consistency between clean and multiplicatively modulated latents, then optimize L = L_s + 0.5 L_cp + 0.5 L_cs (Eq. 7). Evaluation uses CLIP-I, CLIP-T, DINO, and a latent-code KL analysis (Section 5, Figure 6). None of these metrics appears in the training objective, so the reported gains cannot be equalities-by-construction; no fitted parameter is renamed as a prediction. The only self-citational element is borrowing multiplicative noise from the authors' NICE and PACE papers ([32], [35]) to justify the semantics-preserving perturbation in Eq. (5). That citation is used as prior evidence for a technique in other settings, and the diffusion-specific claim is additionally tested qualitatively in Figure 7; even if the premise is under-supported, that is a correctness/robustness weakness rather than a circular reduction. The weak diversity evidence in Figure 6 (single-subject, 1D marginal KL) is likewise a methodological concern, not a circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No invented physical or architectural entities; the method only adds loss terms and hyperparameters.

free parameters (3)
  • lambda_cp = 0.5
    Weight on the prior consistency loss; selected by visual inspection of one dog subject in Section 5.6, Figure 4.
  • lambda_cs = 0.5
    Weight on the subject consistency loss; selected by visual inspection of one dog subject in Section 5.6, Figure 5.
  • sigma = 0.2
    Standard deviation of the multiplicative noise in Eq. (5); fixed by hand and not swept across subjects.
assumptions (4)
  • domain assumption The pretrained Stable Diffusion model's noise prediction on prior class images is the correct target for preserving fidelity (Eq. 4).
    L_cp anchors the fine-tuned model to f_ref's predictions on prior images; if f_ref is imperfect on these images, the anchor inherits that imperfection.
  • ad hoc to paper Multiplicative noise on latent codes preserves subject semantics while expanding the latent distribution (Eq. 5).
    This is the core mechanism behind L_cs; the paper offers no proof and only a qualitative additive-versus-multiplicative comparison in Figure 7 plus one-subject KL histograms.
  • domain assumption LoRA fine-tuning does not need to re-learn prior class images because the pretrained model already captures class concepts.
    This motivation in the Introduction and Section 4.1 justifies replacing DreamBooth's L_prior with L_cp, but the redundancy of LoRA fitting on prior images is not formally shown.
  • domain assumption KL divergence between generated-image latents and prior-image latents is a valid proxy for background diversity.
    Section 5.6 uses Figure 6 as evidence that L_cs improves diversity; no human study or direct diversity metric validates this proxy.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Noise Consistency Regularization for Improved Subject-Driven Image Synthesis." pith.science (2026). https://pith.science/paper/ATWF7R3N

@misc{pith2026250606483,
  author       = {Pith},
  title        = {Pith review of: Noise Consistency Regularization for Improved Subject-Driven Image Synthesis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ATWF7R3N}},
  note         = {Machine review of arXiv:2506.06483}
}
read the original abstract

Fine-tuning Stable Diffusion enables subject-driven image synthesis by adapting the model to generate images containing specific subjects. However, existing fine-tuning methods suffer from two key issues: underfitting, where the model fails to reliably capture subject identity, and overfitting, where it memorizes the subject image and reduces background diversity. To address these challenges, we propose two auxiliary consistency losses for diffusion fine-tuning. First, a prior consistency regularization loss ensures that the predicted diffusion noise for prior (non-subject) images remains consistent with that of the pretrained model, improving fidelity. Second, a subject consistency regularization loss enhances the fine-tuned model's robustness to multiplicative noise modulated latent code, helping to preserve subject identity while improving diversity. Our experimental results demonstrate that incorporating these losses into fine-tuning not only preserves subject identity but also enhances image diversity, outperforming DreamBooth in terms of CLIP scores, background variation, and overall visual quality.

Figures

Figures reproduced from arXiv: 2506.06483 by the authors.

Figure 1
Figure 1. Generated images using different methods with the prompt [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Our pipeline applies two forms of consistency regularization: one over subject images and one over prior images. For the subject [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Generated images and their corresponding prompts for both DreamBooth and our method are presented. As shown, our method [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Ablation study for λcp. Images generated by different λcp with prompt “A photo of a [V] dog on the playground". (a) λcs = 0.0 (b) λcs = 0.2 (c) λcs = 0.5 (d) λcs = 0.8 [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Ablation study for λcs. Images generated by different λcs with prompt “A photo of a [V] dog on the playground" [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Histogram of the latent codes for generated and prior images under different consistency losses. (a) shows the distribution of prior [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Consistency regularization applied to (a) additive noise [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

68 extracted references · 38 canonical work pages

  1. [1]

    Break-a-scene: Extracting multi- ple concepts from a single image

    Omri Avrahami, Kfir Aberman, Ohad Fried, Daniel Cohen- Or, and Dani Lischinski. Break-a-scene: Extracting multi- ple concepts from a single image. InSIGGRAPH Asia 2023 Conference Papers, pages 1–12, 2023. 2

    2023

  2. [2]

    Improving image generation with better captions.Computer Science

    James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions.Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf, 2:3, 2023. 1

    work page 2023

  3. [3]

    Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127, 2023

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127, 2023. 1

    arXiv 2023

  4. [4]

    Align your latents: High-resolution video synthesis with la- tent diffusion models

    Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dock- horn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with la- tent diffusion models. InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 22563–22575, 2023. 1

    2023

  5. [5]

    Emerg- ing properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- ing properties in self-supervised vision transformers. InPro- ceedings of the IEEE/CVF international conference on com- puter vision, pages 9650–9660, 2021. 2, 6

    work page 2021

  6. [6]

    Muse: Text-to-image generation via masked generative transform- ers.arXiv preprint arXiv:2301.00704, 2023

    Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Mur- phy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image generation via masked generative transform- ers.arXiv preprint arXiv:2301.00704, 2023. 1

    arXiv 2023

  7. [7]

    Disenbooth: Identity- preserving disentangled tuning for subject-driven text-to- image generation

    Hong Chen, Yipeng Zhang, Simin Wu, Xin Wang, Xuguang Duan, Yuwei Zhou, and Wenwu Zhu. Disenbooth: Identity- preserving disentangled tuning for subject-driven text-to- image generation. InThe Twelfth International Conference on Learning Representations, 2024. 1

    work page 2024

  8. [8]

    Adaptformer: Adapting vision transformers for scalable visual recogni- tion.Advances in Neural Information Processing Systems, 35:16664–16678, 2022

    Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recogni- tion.Advances in Neural Information Processing Systems, 35:16664–16678, 2022. 2

    2022

Show all 68 references
  1. [9]

    Sem-gan: Semantically- consistent image-to-image translation

    Anoop Cherian and Alan Sullivan. Sem-gan: Semantically- consistent image-to-image translation. In2019 ieee winter conference on applications of computer vision (wacv), pages 1797–1806. IEEE, 2019. 3

    2019

  2. [10]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. 1

    2009

  3. [11]

    Qlora: Efficient finetuning of quantized llms

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36,

  4. [12]

    Krona: Parameter efficient tuning with kronecker adapter.arXiv preprint arXiv:2212.10650, 2022

    Ali Edalati, Marzieh Tahaei, Ivan Kobyzev, Vahid Partovi Nia, James J Clark, and Mehdi Rezagholizadeh. Krona: Parameter efficient tuning with kronecker adapter.arXiv preprint arXiv:2212.10650, 2022. 2

    2022 arXiv

  5. [13]

    Gradient- free textual inversion

    Zhengcong Fei, Mingyuan Fan, and Junshi Huang. Gradient- free textual inversion. InProceedings of the 31st ACM In- ternational Conference on Multimedia, pages 1364–1373,

  6. [14]

    An image is worth one word: Personalizing text-to-image gener- ation using textual inversion

    Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-or. An image is worth one word: Personalizing text-to-image gener- ation using textual inversion. InThe Eleventh International Conference on Learning Representations, 2023. 2, 8

    2023

  7. [15]

    Designing an encoder for fast personalization of text-to-image models.arXiv preprint arXiv:2302.12228, 2023

    Rinon Gal, Moab Arar, Yuval Atzmon, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. Designing an encoder for fast personalization of text-to-image models.arXiv preprint arXiv:2302.12228, 2023. 2

    2023 arXiv

  8. [16]

    Encoder-based domain tuning for fast personalization of text-to-image models.ACM Transactions on Graphics (TOG), 42(4):1–13, 2023

    Rinon Gal, Moab Arar, Yuval Atzmon, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. Encoder-based domain tuning for fast personalization of text-to-image models.ACM Transactions on Graphics (TOG), 42(4):1–13, 2023. 2

    2023

  9. [17]

    Generative adversarial nets.Advances in neural information processing systems, 27, 2014

    Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets.Advances in neural information processing systems, 27, 2014. 1

    2014

  10. [18]

    Svdiff: Compact param- eter space for diffusion fine-tuning

    Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang. Svdiff: Compact param- eter space for diffusion fine-tuning. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 7323–7334, 2023. 2

    2023

  11. [19]

    Lora+: Effi- cient low rank adaptation of large models.arXiv preprint arXiv:2402.12354, 2024

    Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lora+: Effi- cient low rank adaptation of large models.arXiv preprint arXiv:2402.12354, 2024. 2

    2024 arXiv

  12. [20]

    Lora: Low- rank adaptation of large language models

    Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low- rank adaptation of large language models. InInternational Conference on Learning Representations, 2021. 1, 2

    2021

  13. [21]

    Videocontrolnet: A motion-guided video-to-video translation framework by using diffusion model with controlnet.arXiv preprint arXiv:2307.14073,

    Zhihao Hu and Dong Xu. Videocontrolnet: A motion-guided video-to-video translation framework by using diffusion model with controlnet.arXiv preprint arXiv:2307.14073,

    arXiv

  14. [22]

    Llm-adapters: An adapter family for parameter- efficient fine-tuning of large language models

    Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Ka- Wei Lee. Llm-adapters: An adapter family for parameter- efficient fine-tuning of large language models. InThe 2023 Conference on Empirical Methods in Natural Language Pr...

    2023

  15. [23]

    Auto-encoding vari- ational bayes, 2013

    Diederik P Kingma, Max Welling, et al. Auto-encoding vari- ational bayes, 2013. 1

    2013

  16. [24]

    Vera: Vector-based random matrix adaptation.arXiv preprint arXiv:2310.11454, 2023

    Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki Markus Asano. Vera: Vector-based random matrix adaptation.arXiv preprint arXiv:2310.11454, 2023. 2

    2023 arXiv

  17. [25]

    Multi-concept customization of text-to-image diffusion

    Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi-concept customization of text-to-image diffusion. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1931–1941, 2023. 1, 2

    1931

  18. [26]

    Direct consistency optimization for compositional text- to-image personalization.arXiv preprint arXiv:2402.12004,

    Kyungmin Lee, Sangkyung Kwak, Kihyuk Sohn, and Jinwoo Shin. Direct consistency optimization for compositional text- to-image personalization.arXiv preprint arXiv:2402.12004,

    arXiv

  19. [27]

    Parameter-efficient orthogonal finetun- ing via butterfly factorization

    Weiyang Liu, Zeju Qiu, Yao Feng, Yuliang Xiu, Yuxuan Xue, Longhui Yu, Haiwen Feng, Zhen Liu, Juyeon Heo, Songyou Peng, et al. Parameter-efficient orthogonal finetun- ing via butterfly factorization. InThe Twelfth International Conference on Learning Representations, 2023. 2

    2023

  20. [28]

    Subject- diffusion: Open domain personalized text-to-image genera- tion without test-time fine-tuning

    Jian Ma, Junhao Liang, Chen Chen, and Haonan Lu. Subject- diffusion: Open domain personalized text-to-image genera- tion without test-time fine-tuning. InACM SIGGRAPH 2024 Conference Papers, pages 1–12, 2024. 2

    2024

  21. [29]

    Diffusekrona: A pa- rameter efficient fine-tuning method for personalized diffu- sion model.arXiv preprint arXiv:2402.17412, 2024

    Shyam Marjit, Harshit Singh, Nityanand Mathur, Sayak Paul, Chia-Mu Yu, and Pin-Yu Chen. Diffusekrona: A pa- rameter efficient fine-tuning method for personalized diffu- sion model.arXiv preprint arXiv:2402.17412, 2024. 2

    2024 arXiv

  22. [30]

    Steered diffusion: A generalized framework for plug- and-play conditional image synthesis

    Nithin Gopalakrishnan Nair, Anoop Cherian, Suhas Lohit, Ye Wang, Toshiaki Koike-Akino, Vishal M Patel, and Tim K Marks. Steered diffusion: A generalized framework for plug- and-play conditional image synthesis. InProceedings of the IEEE/CVF International Conference on Computer...

    2023

  23. [31]

    Ti2v-zero: Zero-shot image condition- ing for text-to-video diffusion models

    Haomiao Ni, Bernhard Egger, Suhas Lohit, Anoop Cherian, Ye Wang, Toshiaki Koike-Akino, Sharon X Huang, and Tim K Marks. Ti2v-zero: Zero-shot image condition- ing for text-to-video diffusion models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni...

    2024

  24. [32]

    Nice: Noise-modulated consis- tency regularization for data-efficient gans.Advances in Neu- ral Information Processing Systems, 36:13773–13801, 2023

    Yao Ni and Piotr Koniusz. Nice: Noise-modulated consis- tency regularization for data-efficient gans.Advances in Neu- ral Information Processing Systems, 36:13773–13801, 2023. 2, 3, 4

    2023

  25. [33]

    Chain: Enhancing generaliza- tion in data-efficient gans via lipschitz continuity constrained normalization

    Yao Ni and Piotr Koniusz. Chain: Enhancing generaliza- tion in data-efficient gans via lipschitz continuity constrained normalization. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6763–6774, 2024. 1

    2024

  26. [34]

    Cagan: Consistent adversarial training enhanced gans

    Yao Ni, Dandan Song, Xi Zhang, Hao Wu, and Lejian Liao. Cagan: Consistent adversarial training enhanced gans. In IJCAI, pages 2588–2594, 2018. 3

    2018

  27. [35]

    Pace: Marrying generalization in parameter-efficient fine-tuning with consis- tency regularization.Advances in Neural Information Pro- cessing Systems, 37:61238–61266, 2024

    Yao Ni, Shan Zhang, and Piotr Koniusz. Pace: Marrying generalization in parameter-efficient fine-tuning with consis- tency regularization.Advances in Neural Information Pro- cessing Systems, 37:61238–61266, 2024. 2, 3, 4

    2024

  28. [36]

    Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis.arXiv preprint arXiv:2307.01952, 2023

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis.arXiv preprint arXiv:2307.01952, 2023. 1

    2023 arXiv

  29. [37]

    Dreamfusion: Text-to-3d using 2d diffusion.arXiv preprint arXiv:2209.14988, 2022

    Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Milden- hall. Dreamfusion: Text-to-3d using 2d diffusion.arXiv preprint arXiv:2209.14988, 2022. 1

    2022 arXiv

  30. [38]

    Controlling text-to-image diffusion by orthogo- nal finetuning.Advances in Neural Information Processing Systems, 36:79320–79362, 2023

    Zeju Qiu, Weiyang Liu, Haiwen Feng, Yuxuan Xue, Yao Feng, Zhen Liu, Dan Zhang, Adrian Weller, and Bernhard Schölkopf. Controlling text-to-image diffusion by orthogo- nal finetuning.Advances in Neural Information Processing Systems, 36:79320–79362, 2023. 2

    2023

  31. [39]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. InInternational conference on machine learning, p...

    2021

  32. [40]

    Direct preference optimization: Your language model is secretly a reward model.Advances in Neural Information Processing Systems, 36:53728–53741, 2023

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christo- pher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model.Advances in Neural Information Processing Systems, 36:53728–53741, 2023. 3

    2023

  33. [41]

    Dream- booth3d: Subject-driven text-to-3d generation

    Amit Raj, Srinivas Kaza, Ben Poole, Michael Niemeyer, Nataniel Ruiz, Ben Mildenhall, Shiran Zada, Kfir Aber- man, Michael Rubinstein, Jonathan Barron, et al. Dream- booth3d: Subject-driven text-to-3d generation. InProceed- ings of the IEEE/CVF international conference on compu...

    2023

  34. [42]

    Zero-shot text-to-image generation

    Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea V oss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. InInternational Confer- ence on Machine Learning, pages 8821–8831. PMLR, 2021. 1

    2021

  35. [43]

    Hierarchical text-conditional image gener- ation with clip latents.arXiv preprint arXiv:2204.06125, 1 (2):3, 2022

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image gener- ation with clip latents.arXiv preprint arXiv:2204.06125, 1 (2):3, 2022. 1

    2022 arXiv

  36. [44]

    Learning multiple visual domains with residual adapters.Ad- vances in neural information processing systems, 30, 2017

    Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters.Ad- vances in neural information processing systems, 30, 2017. 2

    2017

  37. [45]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 1, 5

    2022

  38. [46]

    Consistency-guided prompt learning for vision-language models

    Shuvendu Roy and Ali Etemad. Consistency-guided prompt learning for vision-language models. InThe Twelfth Interna- tional Conference on Learning Representations, 2024. 3

    2024

  39. [47]

    Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 22500–...

    2023

  40. [48]

    Hyperdreambooth: Hypernetworks for fast personalization of text-to-image models

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Wei Wei, Tingbo Hou, Yael Pritch, Neal Wadhwa, Michael Rubinstein, and Kfir Aberman. Hyperdreambooth: Hypernetworks for fast personalization of text-to-image models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pat-...

    2024

  41. [49]

    Photorealistic text-to-image diffusion models with deep language understanding.Advances in Neural Information Processing Systems, 35:36479–36494, 2022

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding.Advances in Neural Information ...

    2022

  42. [50]

    Open- match: Open-set semi-supervised learning with open-set consistency regularization.Advances in Neural Information Processing Systems, 34:25956–25967, 2021

    Kuniaki Saito, Donghyun Kim, and Kate Saenko. Open- match: Open-set semi-supervised learning with open-set consistency regularization.Advances in Neural Information Processing Systems, 34:25956–25967, 2021. 3

    2021

  43. [51]

    Laion-5b: An open large-scale dataset for training next generation image-text models.Advances in Neural In- formation Processing Systems, 35:25278–25294, 2022

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, et al. Laion-5b: An open large-scale dataset for training next generation image-text models.Advances in Neural In- fo...

    2022

  44. [52]

    Instant- booth: Personalized text-to-image generation without test- time finetuning

    Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung. Instant- booth: Personalized text-to-image generation without test- time finetuning. InProceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pages 8543–8552, 2024. 1

    2024

  45. [53]

    Fixmatch: Simplifying semi-supervised learning with consistency and confidence

    Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in neural information processing systems, 33:5...

    2020

  46. [54]

    Styledrop: Text-to-image generation in any style.arXiv preprint arXiv:2306.00983,

    Kihyuk Sohn, Nataniel Ruiz, Kimin Lee, Daniel Castro Chin, Irina Blok, Huiwen Chang, Jarred Barber, Lu Jiang, Glenn Entis, Yuanzhen Li, et al. Styledrop: Text-to-image generation in any style.arXiv preprint arXiv:2306.00983,

    arXiv

  47. [55]

    Key-locked rank one editing for text-to-image personaliza- tion

    Yoad Tewel, Rinon Gal, Gal Chechik, and Yuval Atzmon. Key-locked rank one editing for text-to-image personaliza- tion. InACM SIGGRAPH 2023 Conference Proceedings, pages 1–11, 2023. 2

    2023

  48. [56]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. InAdvances in Neu- ral Information Processing Systems. Curran Associates, Inc.,

  49. [57]

    Andrey V oynov, Qinghao Chu, Daniel Cohen-Or, and Kfir Aberman.p+: Extended textual conditioning in text-to- image generation.arXiv preprint arXiv:2303.09522, 2023. 2

    2023 arXiv

  50. [58]

    Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distilla- tion.Advances in Neural Information Processing Systems, 36, 2024

    Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distilla- tion.Advances in Neural Information Processing Systems, 36, 2024. 1

    2024

  51. [59]

    Elite: Encoding visual con- cepts into textual embeddings for customized text-to-image generation

    Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. Elite: Encoding visual con- cepts into textual embeddings for customized text-to-image generation. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 15943–15953, 2023. 2

    2023

  52. [60]

    R-drop: Regularized dropout for neural networks.Advances in Neural Informa- tion Processing Systems, 34:10890–10905, 2021

    Lijun Wu, Juntao Li, Yue Wang, Qi Meng, Tao Qin, Wei Chen, Min Zhang, Tie-Yan Liu, et al. R-drop: Regularized dropout for neural networks.Advances in Neural Informa- tion Processing Systems, 34:10890–10905, 2021. 3

    2021

  53. [61]

    Infinite-id: Identity-preserved personalization via id- semantics decoupling paradigm

    Yi Wu, Ziqiang Li, Heliang Zheng, Chaoyue Wang, and Bin Li. Infinite-id: Identity-preserved personalization via id- semantics decoupling paradigm. InEuropean Conference on Computer Vision, pages 279–296. Springer, 2024. 1

    2024

  54. [62]

    Ip- adapter: Text compatible image prompt adapter for text-to- image diffusion models.arXiv preprint arXiv:2308.06721,

    Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip- adapter: Text compatible image prompt adapter for text-to- image diffusion models.arXiv preprint arXiv:2308.06721,

    arXiv

  55. [63]

    Consistency regularization for generative adversarial networks.arXiv preprint arXiv:1910.12027, 2019

    Han Zhang, Zizhao Zhang, Augustus Odena, and Honglak Lee. Consistency regularization for generative adversarial networks.arXiv preprint arXiv:1910.12027, 2019. 3

    1910 arXiv

  56. [64]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847, 2023. 1, 8

    2023

  57. [65]

    Adaptive budget allocation for parameter-efficient fine- tuning

    Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine- tuning. InThe Eleventh International Conference on Learn- ing Representations, 2023. 2

    2023

  58. [66]

    Spectrum-aware parameter ef- ficient fine-tuning for diffusion models.arXiv preprint arXiv:2405.21050, 2024

    Xinxi Zhang, Song Wen, Ligong Han, Felix Juefei-Xu, Akash Srivastava, Junzhou Huang, Hao Wang, Molei Tao, and Dimitris N Metaxas. Spectrum-aware parameter ef- ficient fine-tuning for diffusion models.arXiv preprint arXiv:2405.21050, 2024. 2

    2024 arXiv

  59. [67]

    Uni-controlnet: All-in-one control to text-to-image diffusion models.Advances in Neural Information Processing Sys- tems, 36, 2024

    Shihao Zhao, Dongdong Chen, Yen-Chun Chen, Jianmin Bao, Shaozhe Hao, Lu Yuan, and Kwan-Yee K Wong. Uni-controlnet: All-in-one control to text-to-image diffusion models.Advances in Neural Information Processing Sys- tems, 36, 2024. 1

    2024

  60. [68]

    Asymmetry in low-rank adapters of foundation models.arXiv preprint arXiv:2402.16842, 2024

    Jiacheng Zhu, Kristjan Greenewald, Kimia Nadjahi, Haitz Sáez de Ocáriz Borde, Rickard Brüel Gabrielsson, Leshem Choshen, Marzyeh Ghassemi, Mikhail Yurochkin, and Justin Solomon. Asymmetry in low-rank adapters of foundation models.arXiv preprint arXiv:2402.16842, 2024. 2

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.