REVIEW 7 cited by
AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformers
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Numerous works have recently integrated 3D camera control into foundational text-to-video models, but the resulting camera control is often imprecise, and video generation quality suffers. In this work, we analyze camera motion from a first principles perspective, uncovering insights that enable precise 3D camera manipulation without compromising synthesis quality. First, we determine that motion induced by camera movements in videos is low-frequency in nature. This motivates us to adjust train and test pose conditioning schedules, accelerating training convergence while improving visual and motion quality. Then, by probing the representations of an unconditional video diffusion transformer, we observe that they implicitly perform camera pose estimation under the hood, and only a sub-portion of their layers contain the camera information. This suggested us to limit the injection of camera conditioning to a subset of the architecture to prevent interference with other video features, leading to a 4x reduction of training parameters, improved training speed, and 10% higher visual quality. Finally, we complement the typical dataset for camera control learning with a curated dataset of 20K diverse, dynamic videos with stationary cameras. This helps the model distinguish between camera and scene motion and improves the dynamics of generated pose-conditioned videos. We compound these findings to design the Advanced 3D Camera Control (AC3D) architecture, the new state-of-the-art model for generative video modeling with camera control.
Forward citations
Cited by 7 Pith papers
-
MotionCanvas: Cinematic Shot Design with Controllable Image-to-Video Generation
A controllable image-to-video method that jointly drives camera and object motion by translating scene-space user designs into DCT-coded point trajectories and color-coded bounding boxes for a DiT-based diffusion model.
-
EPiC: Efficient Video Camera Control Learning with Precise Anchor-Video Guidance
EPiC trains a 30M-parameter visibility-aware ControlNet on mask-based anchor videos from 5,000 in-the-wild videos and 500 steps, reaching SOTA camera accuracy on RealEstate10K and MiraData.
-
AniCrafter: Customizing Realistic Human-Centric Animation via Avatar-Background Conditioning in Video Diffusion Models
A diffusion model animates a character into arbitrary dynamic backgrounds by conditioning on a rendered 3D-avatar video, reframing open-domain animation as a restoration problem.
-
LiON-LoRA: Rethinking LoRA Fusion to Unify Controllable Spatial and Temporal Generation for Video Diffusion
LiON-LoRA adds a learned scaling token to video-diffusion LoRA adapters, enabling linear and independent control of camera trajectory and object motion strength.
-
Follow-Your-Creation: Empowering 4D Creation through Video Inpainting
Follow-Your-Creation fine-tunes the Wan2.1 video inpainting model on composite point-cloud and editing masks so a single monocular video can be converted into editable 4D video with new camera motion.
-
RealCam-I2V: Real-World Image-to-Video Generation with Interactive Complex Camera Control
Metric-scale depth alignment plus scene-constrained noise shaping improves camera controllability and video quality for image-to-video generation on RealEstate10K.
-
Hunyuan-Game: Industrial-grade Intelligent Game Creation Model
Tencent's Hunyuan-Game applies diffusion transformers to game asset creation across nine image and video generation tasks, with self-reported gains that are partly contradicted by its own evaluation table.
Discussion (0). Continue with ORCID to comment.