REVIEW 31 cited by
In-Context LoRA for Diffusion Transformers
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Recent research arXiv:2410.15027 has explored the use of diffusion transformers (DiTs) for task-agnostic image generation by simply concatenating attention tokens across images. However, despite substantial computational resources, the fidelity of the generated images remains suboptimal. In this study, we reevaluate and streamline this framework by hypothesizing that text-to-image DiTs inherently possess in-context generation capabilities, requiring only minimal tuning to activate them. Through diverse task experiments, we qualitatively demonstrate that existing text-to-image DiTs can effectively perform in-context generation without any tuning. Building on this insight, we propose a remarkably simple pipeline to leverage the in-context abilities of DiTs: (1) concatenate images instead of tokens, (2) perform joint captioning of multiple images, and (3) apply task-specific LoRA tuning using small datasets (e.g., 20~100 samples) instead of full-parameter tuning with large datasets. We name our models In-Context LoRA (IC-LoRA). This approach requires no modifications to the original DiT models, only changes to the training data. Remarkably, our pipeline generates high-fidelity image sets that better adhere to prompts. While task-specific in terms of tuning data, our framework remains task-agnostic in architecture and pipeline, offering a powerful tool for the community and providing valuable insights for further research on product-level task-agnostic generation systems. We release our code, data, and models at https://github.com/ali-vilab/In-Context-LoRA
Forward citations
Cited by 31 Pith papers
-
CtrlVTON: Controllable Virtual Try-On via Visual-Instance-Prompt Segmentation
CtrlVTON recasts virtual try-on as mask-conditioned editing and introduces VIP-SAM for instance-level garment segmentation, beating proprietary editors on layout fidelity while matching garment quality.
-
MAVIN: Multi-Shot Audio-Visual Generation with Customized Narrative Control
MAVIN proposes boundary-aware attention, ID-aware propagation, a multi-agent scripting pipeline, and the MAVINSet dataset as the first framework for multi-shot audio-visual generation with narrative control, claiming ...
-
WeEdit: A Dataset, Benchmark and Glyph-Guided Framework for Text-centric Image Editing
WeEdit trains a glyph-guided, RL-optimized image editor on a 330K-pair synthetic multilingual dataset and reports open-source SOTA on its own bilingual and multilingual text-editing benchmarks.
-
Story2Board: A Training-Free Approach for Expressive Storyboard Generation
Story2Board uses reciprocal attention value mixing and latent panel anchoring to generate consistent yet visually diverse storyboards from text without any training.
-
Imagine for Me: Creative Conceptual Blending of Real Images and Text via Blended Attention
IT-Blender blends a real image and a text prompt by injecting clean reference features into a trained attention module, improving disentangled concept blending in SD and FLUX.
-
DynEval: Holistic Evaluations of T2I Generative Models in the Wild
DynEval distills a 235B teacher VLM into 2B/4B evaluators via 250K synthetic instruction triplets, yielding higher human correlation than existing T2I metrics while enabling open-set dynamic QA and scene-graph quality checks.
-
Global Pose Control for Generative View Synthesis in Normalized Object Coordinate Space
A diffusion image-editing model conditioned on Plücker ray-map tokens and text-defined NOCS fronts generates high-fidelity novel views with absolute global pose control from unposed inputs.
-
VIPER: Visual In-Context Physics Reasoning for Physically Plausible Video Generation
An MLLM extracts transferable physical cues from a reference video and conditions a pretrained I2V generator so new scenes follow that physics without exhaustive prompts.
-
LuxRemix: Lighting Decomposition and Remixing for Indoor Scenes
A three-stage pipeline decomposes indoor scene lighting into individually controllable OLAT sources, harmonizes the decomposition across views, and encodes it in 3D Gaussian splatting for real-time per-light editing.
-
TinyHistory: Lightweight Video History Embeddings via Two-Stage Context Learning
TinyHistory compresses long video history into a ~5k-token context via a two-stage learning scheme, achieving consistency on par with heavier baselines at lower memory cost.
-
UMO: Scaling Multi-Identity Consistency for Image Customization via Matching Reward
A reinforcement-learning reward based on bipartite face matching improves multi-identity consistency and reduces identity confusion in image customization models.
-
FocusDPO: Dynamic Preference Optimization for Multi-Subject Personalized Image Generation via Adaptive Focus
FocusDPO adds dynamic spatial weighting to preference-based fine-tuning, improving subject fidelity and reducing attribute leakage in multi-subject personalized image generation.
-
USO: Unified Style and Subject-Driven Generation via Disentangled and Reward Learning
USO trains one DiT model for subject-driven, style-driven, and joint generation by disentangling content and style from triplet data and adding a style-reward objective, claiming SOTA on USO-Bench.
-
ToonComposer: Streamlining Cartoon Production with Generative Post-Keyframing
ToonComposer generates cartoon videos from a colored reference frame and sparse keyframe sketches, merging inbetweening and colorization in one diffusion model.
-
AnimeColor: Reference-based Animation Colorization with Diffusion Transformers
AnimeColor colorizes animation sketch sequences from a reference image using a diffusion transformer with high-level and low-level color guidance.
-
FreeCus: Free Lunch Subject-driven Customization in Diffusion Transformers
FreeCus is a training-free method that combines pivotal attention sharing, reversed noise shifting, and MLLM captions to personalize Flux.1 text-to-image generation from a single reference image.
-
ViewPoint: Panoramic Video Generation with Pretrained Diffusion Models
A panorama representation and attention scheme that lets a pretrained perspective video diffusion model generate spatially consistent 360-degree videos from an input perspective clip.
-
RecipeGen: A Step-Aligned Multimodal Benchmark for Real-World Recipe Generation
RecipeGen is a new benchmark with 26,453 recipes, 196,724 step-aligned images, and 4,491 cooking videos, plus three domain-specific evaluation metrics for recipe generation.
-
IA-T2I: Internet-Augmented Text-to-Image Generation
IA-T2I uses active retrieval, hierarchical image selection, and self-reflection to supply internet reference images to T2I models, improving generation accuracy on uncertain-knowledge prompts.
-
ShoulderShot: Generating Over-the-Shoulder Dialogue Videos
ShoulderShot generates over-the-shoulder dialogue videos by pairing two linked camera shots and looping them, so characters stay consistent through long multi-turn conversations.
-
ICM-Fusion: In-Context Meta-Optimized LoRA Fusion for Multi-Task Adaptation
ICM-Fusion uses a conditional VAE plus task-vector guidance to fuse multiple LoRA adapters into one model, reporting marginal average gains on vision and language benchmarks and larger gains in a few-shot long-tail setup.
-
Steering Guidance for Personalized Text-to-Image Diffusion Models
Weight-interpolated null-text weak model in classifier-free guidance improves subject fidelity with minimal text-fidelity loss in personalized text-to-image diffusion.
-
Hybrid Scandium Aluminum Nitride/Silicon Nitride Integrated Photonic Circuits
The abstract reports a low-loss ScAlN/Si3N4 hybrid waveguide, but the full text is an unrelated diffusion-model paper, leaving the photonics claim without supporting evidence.
-
Captain Cinema: Towards Short Movie Generation
A two-stage text-to-movie system that plans keyframes for the story and then synthesizes video between them, using a compressed memory bank to keep long narratives consistent.
-
From Wardrobe to Canvas: Wardrobe Polyptych LoRA for Part-level Controllable Human Image Generation
Wardrobe Polyptych LoRA lets a single diffusion model compose a person's face and clothing from multiple reference photos into new full-body images, generalizing to unseen identities without inference-time fine-tuning.
-
PairEdit: Learning Semantic Variations for Exemplar-based Image Editing
PairEdit trains two LoRA adapters on a pretrained diffusion model to capture the semantic direction between paired source-target images, enabling text-free, controllable image editing from as few as one pair.
-
FullDiT2: Efficient In-Context Conditioning for Video Diffusion Transformers
FullDiT2 accelerates FullDiT-style in-context conditioning for video by dynamic token selection and selective context caching, cutting per-step time by 2-3x with minimal quality loss.
-
Dimension-Reduction Attack! Video Generative Models are Experts on Controllable Image Synthesis
A video diffusion model, HunyuanVideo-I2V, is adapted with mixup transitions, frame-skip position embeddings, and attention masking to outperform image-only models on several controllable image generation benchmarks.
-
In-Context Brush: Zero-shot Customized Subject Insertion with Context-Aware Latent Space Manipulation
In-Context Brush performs zero-shot customized subject insertion by amplifying prompt and reference attention and reweighting attention heads in a pre-trained Flux-Fill diffusion transformer.
-
Hunyuan-Game: Industrial-grade Intelligent Game Creation Model
Tencent's Hunyuan-Game applies diffusion transformers to game asset creation across nine image and video generation tasks, with self-reported gains that are partly contradicted by its own evaluation table.
-
Fuel Consumption in Platoons: A Literature Review
A literature review compiling factors that affect fuel consumption in vehicle platoons, including drag reduction, coordination, and instability.
Discussion (0). Sign in to comment.