REVIEW 7 cited by
Motion Prompting: Controlling Video Generation with Motion Trajectories
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Motion control is crucial for generating expressive and compelling video content; however, most existing video generation models rely mainly on text prompts for control, which struggle to capture the nuances of dynamic actions and temporal compositions. To this end, we train a video generation model conditioned on spatio-temporally sparse or dense motion trajectories. In contrast to prior motion conditioning work, this flexible representation can encode any number of trajectories, object-specific or global scene motion, and temporally sparse motion; due to its flexibility we refer to this conditioning as motion prompts. While users may directly specify sparse trajectories, we also show how to translate high-level user requests into detailed, semi-dense motion prompts, a process we term motion prompt expansion. We demonstrate the versatility of our approach through various applications, including camera and object motion control, "interacting" with an image, motion transfer, and image editing. Our results showcase emergent behaviors, such as realistic physics, suggesting the potential of motion prompts for probing video models and interacting with future generative world models. Finally, we evaluate quantitatively, conduct a human study, and demonstrate strong performance. Video results are available on our webpage: https://motion-prompting.github.io/
Forward citations
Cited by 7 Pith papers
-
CineMaster: A 3D-Aware and Controllable Framework for Cinematic Text-to-Video Generation
Users control a text-to-video model by arranging 3D bounding boxes and camera motion, and the system renders depth maps that steer the diffusion model to generate matching object and camera motion.
-
FlexComposer: Unified Video Compositing from Images to Dynamic Footage with Flexible Trajectory Control
A single video-diffusion framework composites both static images and dynamic footage along user-defined trajectories by transporting canonical foreground latents directly into the background latent sequence.
-
Precise Action-to-Video Generation Through Visual Action Prompts
Skeleton-based visual action prompts give precise, cross-domain action control for video generation of human and robot interactions.
-
AniCrafter: Customizing Realistic Human-Centric Animation via Avatar-Background Conditioning in Video Diffusion Models
A diffusion model animates a character into arbitrary dynamic backgrounds by conditioning on a rendered 3D-avatar video, reframing open-domain animation as a restoration problem.
-
VidCRAFT3: Camera, Object, and Lighting Control for Image-to-Video Generation
VidCRAFT3 is a single image-to-video diffusion system that accepts camera, object, and lighting direction controls separately or jointly, trained in three stages with a new synthetic lighting dataset.
-
ATI: Any Trajectory Instruction for Controllable Video Generation
ATI injects user-drawn point trajectories as soft Gaussian feature masks into a pretrained image-to-video diffusion model, enabling unified camera, object, and local motion control.
-
EndoControlMag: Robust Endoscopic Vascular Motion Magnification with Periodic Reference Resetting and Hierarchical Tissue-aware Dual-Mask Control
A training-free Lagrangian motion magnification framework with periodic reference resetting and tissue-aware dual-mask control improves vascular pulsation visibility in endoscopic surgery videos.
Discussion (0). Continue with ORCID to comment.