REVIEW 8 cited by
Interpreting the Weight Space of Customized Diffusion Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We investigate the space of weights spanned by a large collection of customized diffusion models. We populate this space by creating a dataset of over 60,000 models, each of which is a base model fine-tuned to insert a different person's visual identity. We model the underlying manifold of these weights as a subspace, which we term weights2weights. We demonstrate three immediate applications of this space that result in new diffusion models -- sampling, editing, and inversion. First, sampling a set of weights from this space results in a new model encoding a novel identity. Next, we find linear directions in this space corresponding to semantic edits of the identity (e.g., adding a beard), resulting in a new model with the original identity edited. Finally, we show that inverting a single image into this space encodes a realistic identity into a model, even if the input image is out of distribution (e.g., a painting). We further find that these linear properties of the diffusion model weight space extend to other visual concepts. Our results indicate that the weight space of fine-tuned diffusion models can behave as an interpretable meta-latent space producing new models.
Forward citations
Cited by 8 Pith papers
-
Can this Model Also Recognize Dogs? Zero-Shot Model Search from Weights
ProbeLog represents each classifier output by its responses to fixed probe images and uses CLIP to answer text queries, achieving 43.8% top-1 accuracy when searching 1,500 ImageNet-trained models for a concept.
-
HyperNet Fields: Efficiently Training Hypernetworks without Ground Truth by Learning Weight Trajectories
A hypernetwork is trained to predict the whole weight trajectory of a task network by matching the task loss gradient at each step, removing the need for per-sample ground truth weights.
-
SliderSpace: Decomposing the Visual Capabilities of Diffusion Models
SliderSpace uses PCA on CLIP embeddings of a diffusion model's own samples, then trains low-rank adapters for each principal component, turning them into composable image control sliders.
-
A LoRA is Worth a Thousand Pictures
LoRA weight vectors, projected with PCA and a per-PC calibration, cluster and retrieve artistic styles more accurately than CLIP, DINO, and style-specialized image features.
-
FluxSpace: Disentangled Semantic Editing in Rectified Flow Transformers
FluxSpace performs training-free, disentangled semantic editing in rectified flow transformers by combining attention outputs with prompt-derived linear directions.
-
Safety Alignment Backfires: Preventing the Re-emergence of Suppressed Concepts in Fine-tuned Text-to-Image Diffusion Models
Fine-tuning text-to-image diffusion models on benign data can reactivate suppressed unsafe concepts, and training the task adapter separately from a frozen safety LoRA prevents this.
-
MyTimeMachine: Personalized Facial Age Transformation
A personalized facial age transformation method that uses an adapter network on top of the SAM global aging model, trained with 10 to 50 photos of one person, to produce re-aged images that resemble that person's actu...
-
LoRA Diffusion: Zero-Shot LoRA Synthesis for Diffusion Model Personalization
A VAE plus diffusion hypernetwork synthesizes Stable Diffusion LoRAs for faces from ArcFace embeddings, aiming for zero-shot personalization without per-user fine-tuning.
Discussion (0). Continue with ORCID to comment.