Pith. sign in

REVIEW 6 major objections 10 minor 26 references

Populate-A-Scene: Affordance-Aware Human Video Generation

T0 review · 6 major / 10 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that a text-to-video model can be repurposed to insert a person into a single scene image and animate them according to a text prompt, with no location or pose guidance, by drawing on latent affordance knowledge in the…

desk verdict A genuinely new task and a solid engineering result, but the 'inherent affordance' claim is softer than the paper's framing. read the letter →

arxiv 2507.00334 v1 pith:MEZFAPF7 submitted 2025-07-01 cs.CV

Mengyi Shan , Zecheng He , Haoyu Ma , Felix Juefei-Xu , Peizhao Zhang , Tingbo Hou , Ching-Yao Chuang This is my paper
classification cs.CV
keywords text-to-videogenerationaffordanceperceptionhuman-sceneinteractionhumanvideocross-attentionanalysissceneconditioning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Populate-A-Scene claims that a pre-trained text-to-video generation model can be repurposed as a human-world interaction simulator: given a single photo of an empty scene and a sentence describing an action, it can generate a video of a person naturally performing that action in the scene. The person's placement, body pose, and motion are not specified anywhere; the model is supposed to infer them from the scene's affordances. The authors argue that this affordance knowledge is already latent in the pre-trained video model, and that a relatively small fine-tuning on scene-conditioned human-interaction videos surfaces it. If the claim is right, it makes human video creation much cheaper and offers a window into how generative models understand interaction possibilities in the physical world.

What carries the argument

The mechanism is a dual-stream conditioning branch added to the pre-trained text-to-video transformer. One stream concatenates the scene image's latent code with the noisy video latent along the channel dimension before feeding the transformer, using a temporal Gaussian-noise decay (strength $\gamma = 0.8$) so the scene is enforced at the start but can change with camera motion and interaction later. The other stream is a fused text-image feature enhancer: the image feature map and the three text embeddings are refined through deformable self-attention and mutual image-text cross-attention, then inserted into every transformer block via cross-attention. This fusion lets the model locate the action region in the image, and the cross-attention heatmap over action-related text tokens is used as the diagnostic that reveals the model's affordance perception.

What would settle it

Generate videos for a set of scenes where the human was removed by inpainting, and measure where the generated person appears relative to the known removed-person location; if the person consistently appears at the position of a residual ghost or inpainting artifact rather than at the scene's genuine interaction region, the affordance claim collapses.

Watch

Extended reading notes

Core claim

The paper's central claim is that text-to-video models have an inherent, learnable sense of affordance: what actions a scene invites and where a person should be to perform them. To establish this, the authors remove people from real video frames to create pairs of empty scenes and human-interaction videos, then fine-tune a 4-billion-parameter text-to-video model with an added scene-conditioning branch. The resulting model takes a scene image and action/appearance text and generates a video without any location or pose condition. The paper's evidence that this is genuine affordance perception comes from cross-attention heatmaps: for prompts such as 'riding a horse,' the attention associated with the action verb lands on the saddle region of an input image, and on the PAD affordance benchmark the predicted heatmaps align with human-labeled interaction regions. The authors interpret this as uncovering latent affordance knowledge rather than teaching it from labeled affordance data.

Load-bearing premise

The load-bearing premise is that the automatic human-removal pipeline that builds the training data yields clean empty-scene images with no residual human traces or inpainting artifacts, so the model learns genuine scene affordances rather than shortcut cues.

Editorial extensions

If this is right

  • Users can produce a human-in-scene video from a single photo and a sentence, removing the need for pose capture, masks, or bounding boxes in amateur video creation.
  • The same scene image can support many different action prompts, with the model placing the human at the appropriate location for each action, such as riding versus standing beside a horse.
  • The approach extends to scenes that already contain a person, treating the existing person as part of the environment and generating a second interacting subject.
  • Scenes synthesized by text-to-image models can be populated directly, enabling an AI movie-studio pipeline without 3D capture.
  • Cross-attention heatmaps could serve as an affordance predictor for real images, with spatial accuracy that is strongest in early diffusion steps.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: if the latent-affordance interpretation is correct, training on more diverse human-scene interaction videos should improve interaction reasoning even without explicit affordance labels, and the heatmap analysis provides a way to monitor that improvement.
  • Beyond the paper: the same attention-based probe could be turned into a training signal to align generated human placement with external affordance maps, though the paper does not test this.
  • Beyond the paper: a direct stress test would be action verbs that are under-represented in stock video data, such as rare or passive interactions; poor generalization there would indicate reliance on dataset priors rather than a general affordance sense.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 10 minor

Summary. The paper proposes Populate-A-Scene, a method to fine-tune a pre-trained text-to-video model (Movie Gen, 4B parameters) so that, given a scene image and an action/appearance prompt, it generates a video of a human naturally interacting with the scene, without any bounding-box, mask, or pose condition. Training data are created from stock videos by detecting and inpainting humans out of the first/last frames, then rewriting captions with LLaMA to separate appearance from action prompts. The method uses channel-wise latent concatenation with a temporal noise decay, a fused text-image cross-attention module, and controlled guidance scales. The authors support their central claim through cross-attention heatmap visualizations, an experiment on the PAD affordance dataset, a synthetic multi-action benchmark, and quantitative and human evaluations against image/video editing and generation baselines.

Significance. If the claims hold, the work is significant: it suggests that text-to-video models can be repurposed as interactive world simulators that infer human placement and motion from scene semantics alone, without explicit spatial conditions. The paper contributes a large-scale data curation pipeline, a minimal conditioning design, and an attention-map analysis that could serve as a diagnostic tool for latent affordance understanding. The claim that pre-trained video models already possess some affordance knowledge is interesting and testable. The authors also provide a comparison against several baselines and ablations, and they explicitly acknowledge the use of a 4B-parameter model as a proof of concept. However, the current evidence has important gaps: the key affordance experiment lacks reported quantitative numbers, the training-data pipeline is not validated against shortcut learning from inpainting artifacts, and the quantitative comparisons are reported without error bars or a clear evaluation protocol.

major comments (6)
  1. [Sec. 5.2, Fig. 5] The PAD affordance experiment is the primary quantitative evidence for the claim that the model perceives affordance in real images, but the paper reports no numeric IoU values, no binarization threshold, no specification of how attention maps are aggregated across layers/tokens/timesteps, and no error bars. The "Prediction Accuracy" panel in Fig. 5 appears to be a curve without labeled axes or values. Please report the mean and standard deviation of the pixel-wise IoU over the 235 filtered PAD images, define the exact thresholding and aggregation procedure, and include a baseline comparison (e.g., a center-prior or random attention baseline) to show that the alignment is not an artifact of the chosen procedure.
  2. [Sec. 4.2] The human-removal pipeline (GroundingDINO + SAM + mask dilation by 50 pixels + inpainting with negative prompt "human") is load-bearing for the entire training set, because the model could learn to place and pose humans by exploiting residual inpainting artifacts, shadows, or incomplete human traces instead of scene affordance. The paper provides no validation of the empty-scene images, such as human-detection recall on the inpainted frames, a manual artifact-rate study on a random sample, or an analysis of whether cross-attention peaks correlate with inpainting boundaries. Please add such an analysis, or an ablation that re-trains on data produced by a different removal strategy, to rule out shortcut learning.
  3. [Sec. 6.4, Table 1] The quantitative comparison (FVD, CLIP similarity, Action Score) is reported without error bars, the number of evaluation samples is not stated, and the FVD reference distribution is not defined. Since the "consistently outperforms" claim rests on these numbers, please provide the evaluation protocol (number of scenes/prompts, random seeds, generation steps, number of generated videos per input), report standard deviations or confidence intervals, and state exactly which video set constitutes the real distribution for FVD.
  4. [Sec. 6.1] The synthetic benchmark of 300 images with two prompts per scene is described as an "evaluation set," but no quantitative results are reported on it in the main text; it appears to be used only for qualitative demonstration. Please clarify whether this benchmark is used in any quantitative evaluation, and if so, report the results; if it is only for qualitative purposes, stating this explicitly would remove ambiguity.
  5. [Sec. 4.3] The conditioning mechanism adds Gaussian noise to the conditional image latent with a "temporal scaling factor of gamma = 0.8," but the exact frame-wise schedule is not specified. The sentence "weakening control as the video progresses until the last frame is fully masked" suggests a specific decay, but the formula is missing. Please provide the precise schedule (e.g., a linear or exponential function of frame index) so the method is reproducible.
  6. [Sec. 7 / Abstract] The paper claims to "uncover the inherent affordance perception of a pre-trained video model without labeled affordance datasets." The evidence in Fig. 4 shows that the pre-trained model's attention aligns with action regions when the scene is itself generated by the model, which is a weaker form of evidence than attention alignment on real input scenes. The fine-tuned model's success on real images is consistent with the claim, but whether this is an inherent property of the pre-trained model or a consequence of the fine-tuning task is not fully disentangled. Please discuss this distinction explicitly and, if possible, provide an experiment that tests the pre-trained model's attention on real images (e.g., by using the pre-trained model without fine-tuning to condition on a real scene).
minor comments (10)
  1. [Sec. 4.2 / Sec. 6.4] The two-person data creation generates two training samples per video by removing one person at a time, but the paper does not specify how the text prompts for these samples are constructed (i.e., whether the prompt describes only the inserted person or both persons, and how the remaining person is treated in the appearance/action split). This detail is needed to interpret the human-human interaction results in Fig. 8 and the quantitative action scores.
  2. [Sec. 6.4, Table 1] For image-based baselines, the paper attaches CogVideoX to animate the edited still images, but the same animation model is not used to animate the outputs of the video-generation baselines. This asymmetry could bias the comparison in favor of the proposed method, which is trained end-to-end for video. Please justify this protocol or provide results with a more symmetric evaluation setup.
  3. [Sec. 5.2] The PAD experiment only reports visualizations for a few categories (ride, kick, sit) and does not state the number of images per action verb category after filtering. Given that some categories may have very few images, the stability of the reported IoU is unclear. Please provide per-category counts and, if possible, per-category IoU numbers, so readers can assess the robustness of the affordance prediction claim.
  4. [Sec. 2] The related-work section does not clearly position the method against recent human-scene interaction generation works that also use large pre-trained diffusion or video models, such as those that predict motion in 3D scenes or use language-conditioned generation. A more explicit comparison of the input-output settings (image+prompt vs. text+scene geometry, 2D vs. 3D) would help readers understand the novelty.
  5. [Sec. 6.3, Fig. 9] The qualitative baseline comparison shows only one example (treadmill scene). Since the human evaluation also appears to be based on a small set of videos (the exact number is not reported), a single qualitative example is insufficient to support the claim that the method "outperforms baselines in terms of human placing, motion simulation and appearance rendering." Please report the number of video clips used in the human evaluation and provide additional qualitative examples or a larger user study.
  6. [Sec. 6.5, Table 2] Human evaluation preference percentages are presented without the number of raters, the number of videos rated, or the interface/protocol (e.g., whether raters were shown pairs side-by-side and whether order was randomized). Without this information, the preference values cannot be interpreted. Please include these details.
  7. [Sec. 4.2] The caption-based filtering step ("pure background filtering") is described briefly, and the threshold of 200 for color variance is mentioned, but the keyword exclusion list is not given and the success rate of this filtering stage (how many videos are retained) is not reported. Please provide the retention statistics for each stage of the data pipeline, since the final dataset size depends on these choices.
  8. [Sec. 4.2 / Sec. 4.3] The paper states that the model is trained with "dummy condition images" to preserve text-to-video capability, but the fraction of dummy-condition training samples and the schedule for inserting them are not specified. This is a training-detail that could affect the reproducibility of the method and the strength of the conditioning.
  9. [Sec. 5.1, Eq. (1)] The attention formula is introduced as S = softmax(QK^T / sqrt(d)), but the dimensions of Q and K are not defined in relation to the video latent features (n spatial locations and m text tokens). A brief definition of n and d would improve the precision of the affordance attention analysis.
  10. [Sec. 5.2] The paper says "Our model even outperforms the ground-truth by predicting object parts relevant to each action rather than entire objects" (e.g., seat vs. legs of a bench). While this is an interesting qualitative observation, the IoU metric as defined would penalize the model for this behavior if the ground-truth mask covers the whole object. Please clarify how this observation is compatible with the reported IoU-based accuracy.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: empirical fine-tuning and external benchmarks; Movie Gen reuse is not load-bearing.

full rationale

The paper's central derivation is an empirical fine-tuning recipe, not a formal derivation. The model is fine-tuned on (empty scene, action prompt, human-video) tuples, then tested on held-out scenes, on the external PAD affordance dataset, against image/video editing baselines, and through human preference A/B tests. None of these evaluation quantities is defined in terms of a parameter fitted to the same data and then reported as a prediction: the PAD IoU measures cross-attention maps against independently annotated affordance masks, and the video metrics (FVD, CLIP, Action Score) compare generation quality to external references. The only self-citation is the use of Movie Gen as the pre-trained base model; because the paper's contribution is the conditioning and fine-tuning method, and because the affordance claim is tested on data outside the fine-tuning distribution (PAD images contain no persons), this reuse does not make the conclusion equivalent to its input. No equation-level reduction or fitted-input-renamed-as-prediction is present.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim depends on the assumption that the training scenes, created by automated human removal, are faithful empty versions of the original videos, and that the evaluation benchmarks (PAD, VLM-generated prompts) measure true affordance understanding. The fine-tuned model itself is an engineered artifact with hand-chosen hyperparameters and guidance scales.

free parameters (4)
  • Noise decay factor gamma = 0.8
    Controls how quickly the scene conditioning is weakened over video frames; chosen by hand (Sec. 4.3).
  • Guidance scales for image and text strength = not specified in text
    InstructPix2Pix-style controlled multi-scale guidance; values are not reported (Sec. 4.3).
  • Background color variance threshold = 200
    Used to filter out low-variance backgrounds in dataset curation (Sec. 4.2).
  • Mask dilation size for human removal = 50 pixels
    Dilated human mask before inpainting (Sec. 4.2).
assumptions (5)
  • domain assumption Pre-trained text-to-video models contain latent affordance knowledge that can be elicited by scene conditioning and fine-tuning.
    The whole approach depends on the base Movie Gen model having learned interaction semantics from internet videos (Sec. 1, Sec. 5).
  • domain assumption Cross-attention maps for action-related tokens correspond to spatial affordance localization.
    The affordance analysis interprets attention heatmaps as predictors of where a human would interact (Sec. 5.1).
  • domain assumption The PAD dataset, filtered to 24 action verbs and 235 images, is a reliable external benchmark for affordance prediction.
    Used as the external validation; the filtering discards passive verbs and may change dataset statistics (Sec. 5.2).
  • domain assumption LLaMA-3 and the VLM used for prompt rewriting and scene-action generation produce prompts that preserve the original action semantics.
    Dataset labels and evaluation prompts are generated by language models, which can introduce errors (Sec. 4.2, Sec. 6.1).
  • ad hoc to paper The human removal inpainting does not leave artifacts that the model can exploit.
    The empty-scene images are synthetic products of inpainting; if artifacts remain, the model could learn to locate humans from artifacts rather than affordance (Sec. 4.2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Populate-A-Scene: Affordance-Aware Human Video Generation." pith.science (2026). https://pith.science/paper/MEZFAPF7

@misc{pith2026250700334,
  author       = {Pith},
  title        = {Pith review of: Populate-A-Scene: Affordance-Aware Human Video Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MEZFAPF7}},
  note         = {Machine review of arXiv:2507.00334}
}
read the original abstract

Can a video generation model be repurposed as an interactive world simulator? We explore the affordance perception potential of text-to-video models by teaching them to predict human-environment interaction. Given a scene image and a prompt describing human actions, we fine-tune the model to insert a person into the scene, while ensuring coherent behavior, appearance, harmonization, and scene affordance. Unlike prior work, we infer human affordance for video generation (i.e., where to insert a person and how they should behave) from a single scene image, without explicit conditions like bounding boxes or body poses. An in-depth study of cross-attention heatmaps demonstrates that we can uncover the inherent affordance perception of a pre-trained video model without labeled affordance datasets.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

26 extracted references · 6 canonical work pages

  1. [5]

    Fouhey, Ivan Laptev, Josef Sivic, Abhinav Gupta, and Alexei A

    Vincent Delaitre, David F. Fouhey, Ivan Laptev, Josef Sivic, Abhinav Gupta, and Alexei A. Efros. Scene semantics from long-term observation of people. In Andrew Fitzgibbon, Svetlana Lazebnik, Pietro Perona, Yoichi Sato, and Cordelia Schmid, editors,Computer Vision – ECCV 2012, pages 284–298, Berlin, Heidelberg,

    work page 2012

  2. [8]

    https://arxiv.org/abs/2305.10474. 13 J. J. Gibson.The Senses Considered as Perceptual Systems. George Allen and Unwin LTD,

    arXiv

  3. [11]

    Animate anyone 2: High-fidelity character image animation with environment affordance.arXiv preprint arXiv:2502.06145,

    Li Hu, Guangyuan Wang, Zhen Shen, Xin Gao, Dechao Meng, Lian Zhuo, Peng Zhang, Bang Zhang, and Liefeng Bo. Animate anyone 2: High-fidelity character image animation with environment affordance.arXiv preprint arXiv:2502.06145,

    arXiv

  4. [12]

    David: Modeling dynamic affordance of 3d objects using pre-trained video diffusion models, 2025.https://arxiv.org/abs/2501.08333

    Hyeonwoo Kim, Sangwon Beak, and Hanbyul Joo. David: Modeling dynamic affordance of 3d objects using pre-trained video diffusion models, 2025.https://arxiv.org/abs/2501.08333. Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross B. Girsh...

    arXiv 2025

  5. [14]

    14 Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al

    https://arxiv.org/abs/2210.02747. 14 Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection.arXiv preprint arXiv:2303.05499,

    arXiv

  6. [16]

    Synthesizing Environment-Specific People in Photographs

    https://arxiv.org/abs/2312.14579. William Peebles and Saining Xie. Scalable diffusion models with transformers.arXiv preprint arXiv:2212.09748,

    work page Pith review arXiv

  7. [17]

    https://arxiv.org/abs/2410.13720. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. InInternational Conference on Machine Learning,

    arXiv

  8. [18]

    Weiming Ren, Harry Yang, Ge Zhang, Cong Wei, Xinrun Du, Stephen Huang, and Wenhu Chen

    https: //api.semanticscholar.org/CorpusID:231591445. Weiming Ren, Harry Yang, Ge Zhang, Cong Wei, Xinrun Du, Stephen Huang, and Wenhu Chen. Consisti2v: Enhancing visual consistency for image-to-video generation.arXiv preprint arXiv:2402.04324,

    arXiv

Show all 26 references
  1. [19]

    Nirat Saini, Navaneeth Bodla, Ashish Shrivastava, Avinash Ravichandran, Xiao Zhang, Abhinav Shrivastava, and Bharat Singh

    Accessed: 2025-03-07. Nirat Saini, Navaneeth Bodla, Ashish Shrivastava, Avinash Ravichandran, Xiao Zhang, Abhinav Shrivastava, and Bharat Singh. Invi: Object insertion in videos using off-the-shelf diffusion models, 2024.https://arxiv.org/abs/2407.10958. Mengyi Shan, Brian Cur...

    2025 arXiv

  2. [20]

    Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman

    doi: https://doi.org/10.1145/3610548.3618230. Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. Make-a-video: Text-to-video generation without text-video data, 2...

  3. [21]

    Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Siamak Shakeri, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Denny Zhou, Neil Houlsby, and Donald Metzler

    Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Siamak Shakeri, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Denny Zhou, Neil Houlsby, and Donald Metzler. Ul2: Unifying language learning paradigms, 2023.https://arxiv.org/abs/2...

    2023 arXiv

  4. [23]

    Dat++: Spatially dynamic vision transformer with deformable attention

    Zhuofan Xia, Xuran Pan, Shiji Song, Li Erran Li, and Gao Huang. Dat++: Spatially dynamic vision transformer with deformable attention. arXiv preprint arXiv:2309.01430,

    arXiv

  5. [24]

    doi: 10.1162/tacl_a_00461.https://doi.org/ 10.1162/tacl_a_00461

    ISSN 2307-387X. doi: 10.1162/tacl_a_00461.https://doi.org/ 10.1162/tacl_a_00461. Zhangsihao Yang, Mengyi Shan, Mohammad Farazi, Wenhui Zhu, Yanxi Chen, Xuanzhao Dong, and Yalin Wang. Amg: Avatar motion guided video generation, 2024a.https://arxiv.org/abs/2409.01502. Zhuoyi Yan...

    arXiv

  6. [25]

    Make pixels dance: High-dynamic video generation.arXiv:2311.10982,

    Yan Zeng, Guoqiang Wei, Jiani Zheng, Jiaxin Zou, Yang Wei, Yuchen Zhang, and Hang Li. Make pixels dance: High-dynamic video generation.arXiv:2311.10982,

    arXiv

  7. [26]

    Shenhao Zhu, Junming Leo Chen, Zuozhuo Dai, Yinghui Xu, Xun Cao, Yao Yao, Hao Zhu, and Siyu Zhu

    https://llava-vl.github.io/blog/ 2024-04-30-llava-next-video/. Shenhao Zhu, Junming Leo Chen, Zuozhuo Dai, Yinghui Xu, Xun Cao, Yao Yao, Hao Zhu, and Siyu Zhu. Champ: Controllable and consistent human image animation with 3d parametric guidance,

    2024

  8. [1999]

    Max Ku, Cong Wei, Weiming Ren, Harry Yang, and Wenhu Chen

    ISBN 9780415209625.https: //books.google.com/books?id=cLnqI3dvi4kC. Max Ku, Cong Wei, Weiming Ren, Harry Yang, and Wenhu Chen. Anyv2v: A tuning-free framework for any video-to-video editing tasks.arXiv preprint arXiv:2403.14468,

    arXiv

  9. [2011]

    Photorealistic video generation with diffusion models, 2023.https://arxiv.org/abs/2312.06662

    Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Li Fei-Fei, Irfan Essa, Lu Jiang, and José Lezama. Photorealistic video generation with diffusion models, 2023.https://arxiv.org/abs/2312.06662. Mohamed Hassan, Partha Ghosh, Joachim Tesch, Dimitrios Tzionas, and Michae...

    2023 arXiv

  10. [2012]

    ISBN 978-3-642-33783-3

    Springer Berlin Heidelberg. ISBN 978-3-642-33783-3. Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929,

    2010 arXiv

  11. [2017]

    Modelscope text-to-video technical report, 2023a.https://arxiv.org/abs/2308.06571

    Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Modelscope text-to-video technical report, 2023a.https://arxiv.org/abs/2308.06571. Tan Wang, Linjie Li, Kevin Lin, Chung-Ching Lin, Zhengyuan Yang, Hanwang Zhang, Zicheng Liu, and Lijuan Wang. D...

    2017 arXiv

  12. [2018]

    Emu: Enhancing image generation models using photogenic needles in a haystack

    Xiaoliang Dai, Ji Hou, Chih-Yao Ma, Sam Tsai, Jialiang Wang, Rui Wang, Peizhao Zhang, Simon Vandenhende, Xiaofang Wang, Abhimanyu Dubey, et al. Emu: Enhancing image generation models using photogenic needles in a haystack. arXiv preprint arXiv:2309.15807,

    arXiv

  13. [2020]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zha...

    2024 arXiv

  14. [2021]

    Latte: Latent diffusion transformer for video generation.arXiv preprint arXiv:2401.03048,

    Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation.arXiv preprint arXiv:2401.03048,

    arXiv

  15. [2022]

    Kingma, Ben Poole, Mohammad Norouzi, David J

    Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P. Kingma, Ben Poole, Mohammad Norouzi, David J. Fleet, and Tim Salimans. Imagen video: High definition video generation with diffusion models, 2022.https://arxiv.org/abs/2210.02303. L...

    2022 arXiv

  16. [2023]

    Boffi, Michael S

    Nicholas M. Boffi, Michael S. Albergo, and Eric Vanden-Eijnden. Flow map matching, 2024.https://arxiv.org/abs/ 2406.07507. 11 Tim Brooks, Aleksander Holynski, and Alexei A. Efros. Instructpix2pix: Learning to follow image editing instructions. In CVPR,

    2024 arXiv

  17. [2024]

    Accessed: 2025-03-07. Omer Bar-Tal, Hila Chefer, Omer Tov, Charles Herrmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Guanghui Liu, Amit Raj, Yuanzhen Li, Michael Rubinstein, Tomer Michaeli, Oliver Wang, Deqing Sun, Tali Dekel, and Inbar Mosseri. Lumiere: A space-tim...

    2025

  18. [2025]

    doi: 10.1145/3715140.https://doi.org/10.1145/3715140

    ISSN 1551-6857. doi: 10.1145/3715140.https://doi.org/10.1145/3715140. Just Accepted. Z. Cao, G. Hidalgo Martinez, T. Simon, S. Wei, and Y. A. Sheikh. Openpose: Realtime multi-person 2d pose estimation using part affinity fields.IEEE Transactions on Pattern Analysis and Machine...

    doi

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.