Pith. sign in

REVIEW 4 major objections 5 minor 53 references

VIRES: Video Instance Repainting via Sketch and Text Guided Generation

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A sketch sequence and a text caption can repaint, replace, generate, or remove a single object in a video.

desk verdict Solid engineering, but Table 1 compares masked local editing against unmasked full-frame baselines, so the headline quantitative claim is not supported. read the letter →

arxiv 2411.16199 v6 pith:RT7TW4WR submitted 2024-11-25 cs.CV

Shuchen Weng , Haojie Zheng , Peixuan Zhang , Yuchen Hong , Han Jiang , Si Li , Boxin Shi This is my paper · ORCID
classification cs.CV
keywords videoinstancerepaintingsketch-guidededitingtext-to-videodiffusiontemporalconsistencysequentialcontrolnetworkflowmatchingdataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a single video instance—a person, an animal, an object—can be repainted, replaced, generated from scratch, or removed under the joint control of a sketch sequence and a text caption, without disturbing the rest of the frame. The proposed system, VIRES, builds on a pre-trained text-to-video diffusion transformer and adds four components: a Sequential ControlNet that reads the sketch as a spatial-temporal structure signal, a standardized self-scaling that sharpens high-contrast sketch lines, a sketch attention that lets video tokens consult sketch tokens inside each transformer block, and a sketch-aware encoder that aligns output textures to the sketch during decoding. The authors also construct a training and evaluation dataset of 85K video clips with instance masks, sketch sequences, and text captions, and report that VIRES beats existing video editing methods on visual quality, temporal consistency, condition alignment, and human preference. If these findings hold, fine-grained instance-level video editing becomes practical for non-experts: the user draws or modifies a rough sketch and types a description, and the model propagates the change consistently through time.

What carries the argument

The load-bearing object is the sketch-conditioning stack. Sequential ControlNet is a 3D-causal-convolution branch that converts the sketch sequence into feature maps matching the DiT backbone. Standardized self-scaling is the operation $\hat{f}^s = ((f^s - \mu(f^s))/\sigma(f^s)) \odot f^s$, followed by a mean shift toward the video features; it emphasises the sharp black-white transitions of sketch lines. Sketch attention is a parallel branch in the spatial self-attention modules that adds a predefined binary correspondence matrix $\mathcal{A}$ inside the softmax, $\bar{f}^z = \mathrm{Softmax}((QK^\top + \mathcal{A})/\sqrt{C})V$, so video tokens can borrow structure from sketch tokens. The sketch-aware encoder reuses the VAE encoder architecture and injects its multi-level features into the decoder as $\bar{x} = D(z', E_s(s))$. These modules are trained with a flow-matching objective and a combined SSIM, L1, perceptual, and KL loss for the encoder.

What would settle it

Take a set of videos with sustained occlusions, fast motion, and drastic viewpoint changes, provide user-drawn sketches that intentionally deviate from automatic edge detection, and measure warp error and frame consistency against the edited output; if VIRES falls to baseline levels whenever the sketch or mask is imperfect, the claim that its modules give robust temporal consistency and condition alignment is refuted.

Watch

Extended reading notes

Core claim

The central claim is that reliable, temporally consistent video instance repainting can be achieved by conditioning a pre-trained DiT-based text-to-video model on sketch and text, rather than by composing independently edited frames. VIRES encodes the original clip, selectively adds noise only inside the instance mask, and learns a flow-matching vector field that reconstructs the masked region while leaving the unmasked background untouched. The sketch signal enters at three levels: coarse structure via the Sequential ControlNet, token-level semantics via sketch attention in the spatial self-attention modules, and multi-level texture via the sketch-aware encoder during decoding. On both the proposed dataset and an external video benchmark, the paper reports the best PSNR, SSIM, warp error, frame consistency, and text consistency among compared methods, plus higher human preference in two user studies.

Load-bearing premise

The whole pipeline is trained on a self-constructed dataset built from automatically segmented real videos, and if that distribution differs from what users actually want to edit—say, heavily occluded, fast-moving, or rarely appearing objects—the reported gains may not transfer.

Editorial extensions

If this is right

  • Users can repaint a single instance, such as changing a shirt's color, or replace it entirely while the rest of the video stays unchanged.
  • The same model supports removing an instance, generating a new instance in an empty region, and generating a full video from a sketch sequence with text guidance.
  • Sparse sketch guidance means a single sketch frame can repaint an entire clip, lowering the annotation burden for practical editing.
  • Longer videos are handled by editing overlapping 51-frame chunks, using the last frames of one chunk as hints for the next.
  • Because the conditioning modules sit on a DiT backbone, the design can be ported to other DiT-based text-to-video generators.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to apply the same conditioning stack to multi-instance editing: if the modules are truly mask-agnostic, the repainting should remain consistent when several interacting instances are masked simultaneously.
  • The high-contrast self-scaling idea should transfer to other binary or high-contrast condition maps such as user scribbles, silhouette masks, or cartoon frames; the paper's own edge-map experiments support this, but depth maps show little benefit, so the boundary of the mechanism is itself a research question.
  • Because the reported text-consistency gains are small on both datasets, the main practical advantage is likely structure alignment and temporal stability rather than semantic fidelity; an editing tool built on VIRES should be evaluated with that weighting in mind.
  • The chunk-based long-video strategy is likely to accumulate drift, so an external evaluation over one-minute clips with no ground-truth reference would test whether temporal consistency actually holds beyond 51 frames.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces VIRES, a video instance repainting method that uses sketch and text guidance on top of a pre-trained text-to-video diffusion transformer (Open-Sora). The proposed system combines a Sequential ControlNet with standardized self-scaling, a sketch-attention branch inside the DiT backbone, and a sketch-aware encoder that injects multi-level features during decoding. The authors also contribute VireSet, a large dataset of 85K training and 1K evaluation clips with instance masks, HED sketches, and LLM-generated captions. The central claim is that VIRES outperforms state-of-the-art video editing methods in visual quality, temporal consistency, condition alignment, and human preference, based on quantitative comparisons on VireSet and DAVIS and on two user studies.

Significance. If the claims hold, VIRES is a practical advance for controllable instance-level video editing, and VireSet is a potentially useful training and evaluation resource. Strengths of the manuscript include a clearly specified diffusion formulation, a modular architecture with individual ablations for each proposed component, and an external benchmark in DAVIS that provides some generalization evidence beyond the authors' own dataset. However, the principal quantitative evidence is weakened by a comparison protocol that is not like-for-like: VIRES is mask-conditioned and preserves the background by construction, whereas the baselines are full-frame editing methods run without masks and are evaluated with global metrics. In addition, the quantitative comparisons lack error bars and significance tests. The stress-test concern about the evaluation asymmetry is confirmed by the masking equation, and it directly affects the main 'outperforms state-of-the-art' claim.

major comments (4)
  1. [§5.2, Table 1; Eq. (2)] The comparison in Table 1 is not like-for-like. By construction in Eq. (2), VIRES leaves the latent code outside the instance mask unchanged, so the background is preserved; the five baselines are full-frame editing methods run with default configurations, and no instance mask is mentioned for them. Computing global PSNR, SSIM, and Warp Error against the input video therefore rewards VIRES for not modifying background pixels rather than for superior editing. For example, the VireSet PSNR margin (23.87 vs 18.99) and the DAVIS Warp Error (6.49 vs 8.74) plausibly reflect this protocol. Please report metrics computed only inside the masked instance region, add a background-preservation metric, and either provide the same mask condition to the baselines or explain clearly why each baseline cannot accept it.
  2. [§5.2, Table 1] No error bars, confidence intervals, or significance tests are reported for any quantitative or user-study result. Several headline differences are small, e.g., FC 92.23 vs 92.17 and TC 16.19 vs 16.13 on VireSet, and the VQE/TAE percentages in Table 1 are aggregates over 10 samples per dataset and 25 participants without any reported variance. Without statistical support, the claim that VIRES consistently outperforms all baselines on every metric is not established, even if the underlying editing outputs are visually reasonable.
  3. [§3 and §5.2] The DAVIS evaluation is external in video source, but the sketch, mask, and caption conditions for DAVIS are extracted with the same pipeline used to create the VireSet training data. The evaluation therefore tests distribution shift in the source videos but not robustness to different condition-annotation distributions. This is a secondary concern, but the paper should either acknowledge this limitation explicitly or add a small subset with human-annotated conditions to support the generalization claim.
  4. [§5.1 and Table 1] The claimed superiority in 'condition alignment' is not quantified for the sketch condition. The only automatic alignment metric, Text Consistency, measures text-video alignment; there is no automatic metric that measures whether the repainted instance follows the provided sketch sequence. Please add a sketch-alignment metric (e.g., HED edge distance within the masked region) or restrict the condition-alignment claim to the user study and qualitative examples.
minor comments (5)
  1. [Eq. (8)] The construction of the predefined binary matrix A is not specified. Please state whether it is a fixed spatial correspondence matrix, how it is derived for the 1×2×2 tokenization, and whether it is identical across all videos and resolutions.
  2. [Eq. (10)] The hyperparameters λ1, λ2, and λ3 are stated to be insensitive to variations, but no sensitivity analysis is shown. Please report the range over which they were varied and the resulting metric changes.
  3. [Table 1 and Fig. 4] The ablation names 'W/o SSS' and 'W/o SS' are easy to confuse in the captions. Please spell out 'standardized self-scaling' and 'self-scaling' in the table or use more distinct abbreviations.
  4. [Throughout] The dataset name is rendered inconsistently as both 'VIRE SET' and 'VireSet'; please use a single consistent spelling throughout the manuscript.
  5. [Eq. (9)] The notation D(z′, E_s(s))_i should be defined more explicitly; as written, the subscript i can be read as a frame index rather than a decoder feature level.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: VIRES's modules are evaluated on external benchmarks, and the mask-preserving comparison issue is an evaluation asymmetry, not a derivation loop.

full rationale

VIRES's central claims are empirical: the proposed modules (Sequential ControlNet, standardized self-scaling, sketch attention, sketch-aware encoder) are trained with a flow-matching objective (Eq. 4) on a held-out split of VireSet and evaluated on the external DAVIS benchmark; no equation in the method section defines a target result in terms of a fitted input, and no 'prediction' is statistically forced by a parameter fit. The forward-process masking in Eq. (2) does mean that VIRES preserves the unmasked latent, which plausibly inflates full-frame PSNR/SSIM relative to full-frame baselines that do not receive a mask; however, this is a comparison-protocol asymmetry, not circularity, because the metric is not the model's training objective and the baselines' outputs are not derived from the mask input. The only self-citations ([8], [41]) appear in related-work listings and are not load-bearing; the pretrained backbones (Open-Sora, SAM-2, PLLaVa, HED) are external. Hence there is no circular step, and the paper is self-contained against external benchmarks for its core comparison.

Assumptions & free parameters 2 free parameters · 6 assumptions · 4 invented entities

The central claim rests on a stack of domain assumptions about the transferability of Open-Sora priors, the reliability of automatically generated masks and sketches, and the quality of LLM-generated captions. The newly introduced architectural modules are only validated through internal ablations and self-constructed evaluation splits, which increases the burden on the authors to provide external validation and release artifacts.

free parameters (2)
  • Lambda_1, Lambda_2, Lambda_3 in sketch-aware encoder loss (Eq. 10) = 1, 0.1, 1e-6
    Weights for SSIM, L1, perceptual, and KL terms in the VAE loss. Chosen by hand and claimed to be insensitive to variation.
  • Lambda in sketch attention (Eq. 8) = learned
    Learnable scaling parameter that weights the sketch-attention branch output during training.
assumptions (6)
  • standard math Flow matching formulation of Lipman et al. is used for training the denoising network.
    Section 4.1, Eq. (1)-(4); accepted background in diffusion and flow matching.
  • domain assumption Open-Sora v1.2 provides pre-trained spatial-temporal VAE and DiT backbone that transfer to video editing.
    Section 4.1 and Section 5; the whole method relies on these pretrained priors.
  • domain assumption SAM-2 annotations at 24 FPS accurately propagate instance masks across frames.
    Section 3; used to build VireSet and to define repainting regions.
  • domain assumption HED edge detection produces a sketch sequence that encodes sufficient structure for repainting.
    Section 3; the sketch is the main control signal for structure.
  • domain assumption PLLaVa-generated text descriptions are sufficiently accurate for training text conditioning.
    Section 3; only 1% of samples were human-reviewed, with a 91% acceptance rate.
  • ad hoc to paper The predefined binary matrix A in sketch attention reflects the correct correspondence between latent tokens and sketch tokens.
    Section 4.3, Eq. (8); the construction of A is not fully specified in the main text.
invented entities (4)
  • Sequential ControlNet
    purpose: Extracts spatial-temporal structure layouts from sketch sequences and injects them into the DiT backbone.
    Validated only by internal ablations in Section 5.3; no external test.
  • Standardized self-scaling
    purpose: Adaptively scales sketch features by their standardized values to emphasize high-contrast lines.
    Internal ablation; additional edge and depth experiments in Appendix 7.2.
  • Sketch attention
    purpose: Parallel attention branch in spatial self-attention to inject fine-grained sketch semantics into latent tokens.
    Internal ablation; the matrix design is not independently verified.
  • Sketch-aware encoder
    purpose: Provides multi-level texture features to the VAE decoder so reconstructed frames align with the sketch.
    Internal ablation; no external benchmark.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VIRES: Video Instance Repainting via Sketch and Text Guided Generation." pith.science (2026). https://pith.science/paper/RT7TW4WR

@misc{pith2026241116199,
  author       = {Pith},
  title        = {Pith review of: VIRES: Video Instance Repainting via Sketch and Text Guided Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RT7TW4WR}},
  note         = {Machine review of arXiv:2411.16199}
}
read the original abstract

We introduce VIRES, a video instance repainting method with sketch and text guidance, enabling video instance repainting, replacement, generation, and removal. Existing approaches struggle with temporal consistency and accurate alignment with the provided sketch sequence. VIRES leverages the generative priors of text-to-video models to maintain temporal consistency and produce visually pleasing results. We propose the Sequential ControlNet with the standardized self-scaling, which effectively extracts structure layouts and adaptively captures high-contrast sketch details. We further augment the diffusion transformer backbone with the sketch attention to interpret and inject fine-grained sketch semantics. A sketch-aware encoder ensures that repainted results are aligned with the provided sketch sequence. Additionally, we contribute the VireSet, a dataset with detailed annotations tailored for training and evaluating video instance editing methods. Experimental results demonstrate the effectiveness of VIRES, which outperforms state-of-the-art methods in visual quality, temporal consistency, condition alignment, and human ratings. Project page: https://hjzheng.net/projects/VIRES/

Figures

Figures reproduced from arXiv: 2411.16199 by the authors.

Figure 1
Figure 1. Our VIRES model demonstrates powerful video editing capabilities with sketch and text guidance, as shown in four typical [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The pipeline of VIRES model. (a) The input video clip and corresponding text descriptions are encoded using independent [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Visual quality comparisons of video editing methods with text and sketch guidance. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (7 more)
Figure 5
Figure 5. Figure 5: Examples of extended applications: (a) Generating an [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Examples of repainting with additional conditions. [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Typical application scenarios. Left: Video instance repainting. Right: Video instance replacement. 11 [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Typical application scenarios. Left: Custom instance generation. Right: Specified instance removal. 12 [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 9
Figure 9. Figure 9: Sketch-to-video generation and inpainting. [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Sparse sketch guidance. Repainting the video using different first sketch frames. [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Long-duration video repainting. Restoring the damaged video to clearly depict a realistic female character. [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

53 extracted references · 27 canonical work pages

  1. [1]

    Frozen in time: A joint video and image encoder for end-to-end retrieval

    Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisser- man. Frozen in time: A joint video and image encoder for end-to-end retrieval. In ICCV, 2021. 3

    2021

  2. [2]

    Is space-time attention all you need for video understanding? In ICML, 2021

    Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In ICML, 2021. 2

    work page 2021

  3. [3]

    Sigmoid- weighted linear units for neural network function approxi- mation in reinforcement learning

    Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid- weighted linear units for neural network function approxi- mation in reinforcement learning. Neural networks, 2018. 4

    2018

  4. [4]

    Digital image process- ing

    Rafael C Gonzales and Paul Wintz. Digital image process- ing. Addison-Wesley Longman Publishing Co., Inc., 1987. 9

    work page 1987

  5. [5]

    Latent video diffusion models for high-fidelity long video generation

    Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity long video generation. arXiv preprint arXiv:2211.13221 ,

    arXiv

  6. [6]

    Denoising diffu- sion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffu- sion probabilistic models. In NeurIPS, 2020. 2

    2020

  7. [7]

    Video dif- fusion models

    Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video dif- fusion models. In NeurIPS, 2022. 2

    2022

  8. [8]

    L-DiffER: Single image reflection removal with language-based diffusion model

    Yuchen Hong, Haofeng Zhong, Shuchen Weng, Jinxiu Liang, and Boxin Shi. L-DiffER: Single image reflection removal with language-based diffusion model. ECCV, 2024. 2

    work page 2024

Show all 53 references
  1. [9]

    Arbitrary style transfer in real-time with adaptive instance normalization

    Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In ICCV,

  2. [10]

    VBench: Comprehensive bench- mark suite for video generative models

    Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. VBench: Comprehensive bench- mark suite for video generative models. In CVPR, 2024. 3

    2024

  3. [11]

    Scope of va- lidity of psnr in image/video quality assessment

    Quan Huynh-Thu and Mohammed Ghanbari. Scope of va- lidity of psnr in image/video quality assessment. Electronics letters, 2008. 6

    2008

  4. [12]

    Perceptual losses for real-time style transfer and super-resolution

    Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV, 2016. 5

    2016

  5. [13]

    RA VE: Randomized noise shuf- fling for fast and consistent video editing with diffusion mod- els

    Ozgur Kara, Bariscan Kurtkaya, Hidir Yesiltepe, James M Rehg, and Pinar Yanardag. RA VE: Randomized noise shuf- fling for fast and consistent video editing with diffusion mod- els. In CVPR, 2024. 6, 7, 10

    2024

  6. [14]

    Text2Video-Zero: Text- to-image diffusion models are zero-shot video generators

    Levon Khachatryan, Andranik Movsisyan, Vahram Tade- vosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2Video-Zero: Text- to-image diffusion models are zero-shot video generators. In ICCV, 2023. 2, 6, 7, 9, 10

    2023

  7. [15]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 ,

    arXiv

  8. [16]

    On information and sufficiency

    Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathematical statistics, 1951. 5

    1951

  9. [17]

    Multi-concept customization of text-to-image diffusion

    Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi-concept customization of text-to-image diffusion. In CVPR, 2023. 2

    2023

  10. [18]

    VidToMe: Video token merging for zero-shot video editing

    Xirui Li, Chao Ma, Xiaokang Yang, and Ming-Hsuan Yang. VidToMe: Video token merging for zero-shot video editing. In CVPR, 2024. 2, 6, 7, 10

    2024

  11. [19]

    Flow matching for generative modeling

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximil- ian Nickel, and Matthew Le. Flow matching for generative modeling. In ICLR, 2023. 3

    2023

  12. [20]

    Sora: A review on background, technology, limitations, and opportunities of large vision models

    Yixin Liu, Kai Zhang, Yuan Li, Zhiling Yan, Chujie Gao, Ruoxi Chen, Zhengqing Yuan, Yue Huang, Hanchi Sun, Jian- feng Gao, et al. Sora: A review on background, technology, limitations, and opportunities of large vision models. arXiv preprint arXiv:2402.17177, 2024. 2

    2024 arXiv

  13. [21]

    SDEdit: Guided image synthesis and editing with stochastic differential equa- tions

    Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jia- jun Wu, Jun-Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equa- tions. In ICLR, 2021. 2

    2021

  14. [22]

    T2I-Adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models

    Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2I-Adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In AAAI, 2024. 2

    2024

  15. [23]

    Semantic image synthesis with spatially-adaptive nor- malization

    Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. Semantic image synthesis with spatially-adaptive nor- malization. In CVPR, 2019. 4

    2019

  16. [24]

    Zero-shot image-to-image translation

    Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun-Yan Zhu. Zero-shot image-to-image translation. In SIGGRAPH, 2023. 2

    2023

  17. [25]

    A benchmark dataset and evaluation methodology for video object segmentation

    Federico Perazzi, Jordi Pont-Tuset, Brian McWilliams, Luc Van Gool, Markus Gross, and Alexander Sorkine-Hornung. A benchmark dataset and evaluation methodology for video object segmentation. In CVPR, 2016. 3, 6

    2016

  18. [26]

    FiLM: Visual reasoning with a general conditioning layer

    Ethan Perez, Florian Strub, Harm De Vries, Vincent Du- moulin, and Aaron Courville. FiLM: Visual reasoning with a general conditioning layer. In AAAI, 2018. 4

    2018

  19. [27]

    FateZero: Fus- ing attentions for zero-shot text-based video editing

    Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen. FateZero: Fus- ing attentions for zero-shot text-based video editing. In ICCV, 2023. 2

    2023

  20. [28]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In ICML, 2021. 6

    2021

  21. [29]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 2020. 3

    2020

  22. [30]

    SAM 2: Segment anything in images and videos

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. SAM 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024. 2, 3

    2024 arXiv

  23. [31]

    High-resolution image syn- thesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image syn- thesis with latent diffusion models. In CVPR, 2022. 2 15

    2022

  24. [32]

    DreamBooth: Fine tuning text-to-image diffusion models for subject-driven generation

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. DreamBooth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR, 2023. 2

    2023

  25. [33]

    Photorealistic text-to-image diffusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. In NeurIPS, 2022. 2

    2022

  26. [34]

    Denois- ing diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denois- ing diffusion implicit models. In ICLR, 2021. 2

    2021

  27. [35]

    Interpretable 3D human action analysis with temporal convolutional networks

    Tae Soo Kim and Austin Reiter. Interpretable 3D human action analysis with temporal convolutional networks. In CVPR workshops, 2017. 2

    2017

  28. [36]

    Gen-L-Video: Multi-text to long video generation via temporal co-denoising

    Fu-Yun Wang, Wenshuo Chen, Guanglu Song, Han-Jia Ye, Yu Liu, and Hongsheng Li. Gen-L-Video: Multi-text to long video generation via temporal co-denoising. arXiv preprint arXiv:2305.18264, 2023. 2

    2023 arXiv

  29. [37]

    VideoCLIP-XL: Advancing long descrip- tion understanding for video clip models

    Jiapeng Wang, Chengyu Wang, Kunzhe Huang, Jun Huang, and Lianwen Jin. VideoCLIP-XL: Advancing long descrip- tion understanding for video clip models. arXiv preprint arXiv:2410.00741, 2024. 6

    2024 arXiv

  30. [38]

    VideoComposer: Compositional video synthesis with motion controllability

    Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Ji- uniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jin- gren Zhou. VideoComposer: Compositional video synthesis with motion controllability. In NeurIPS, 2024. 2, 3, 6, 7, 9, 10

    2024

  31. [39]

    SinSR: diffusion-based image super- resolution in a single step

    Yufei Wang, Wenhan Yang, Xinyuan Chen, Yaohui Wang, Lanqing Guo, Lap-Pui Chau, Ziwei Liu, Yu Qiao, Alex C Kot, and Bihan Wen. SinSR: diffusion-based image super- resolution in a single step. In CVPR, 2024. 2

    2024

  32. [40]

    Bovik, Hamid R

    Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. Image quality assessment: from error visibility to structural similarity. TIP, 2004. 5, 6

    2004

  33. [41]

    L-CAD: Language-based colorization with any-level descriptions using diffusion priors

    Shuchen Weng, Peixuan Zhang, Yu Li, Si Li, Boxin Shi, et al. L-CAD: Language-based colorization with any-level descriptions using diffusion priors. In NeurIPS, 2024. 2

    2024

  34. [42]

    Tune-A-Video: One-shot tuning of image diffusion models for text-to-video generation

    Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-A-Video: One-shot tuning of image diffusion models for text-to-video generation. In ICCV, 2023. 2

    2023

  35. [43]

    Group normalization

    Yuxin Wu and Kaiming He. Group normalization. In ECCV,

  36. [44]

    Holistically-nested edge de- tection

    Saining Xie and Zhuowen Tu. Holistically-nested edge de- tection. In ICCV, 2015. 2, 3

    2015

  37. [45]

    PLLaV A: Parameter-free llava extension from images to videos for video dense captioning

    Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. PLLaV A: Parameter-free llava extension from images to videos for video dense captioning. arXiv preprint arXiv:2404.16994, 2024. 2, 3

    2024 arXiv

  38. [46]

    Depth any- thing v2

    Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiao- gang Xu, Jiashi Feng, and Hengshuang Zhao. Depth any- thing v2. arXiv preprint arXiv:2406.09414, 2024. 9

    2024 arXiv

  39. [47]

    Rerender a video: Zero-shot text-guided video-to-video translation

    Shuai Yang, Yifan Zhou, Ziwei Liu, and Chen Change Loy. Rerender a video: Zero-shot text-guided video-to-video translation. In SIGGRAPH Asia, 2023. 2, 6, 7, 10

    2023

  40. [48]

    CogVideoX: Text-to-video diffusion models with an expert transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiao- han Zhang, Guanyu Feng, et al. CogVideoX: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072, 2024. 2, 9

    2024 arXiv

  41. [49]

    Language model beats diffusion-tokenizer is key to visual generation

    Lijun Yu, Jose Lezama, Nitesh Bharadwaj Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, et al. Language model beats diffusion-tokenizer is key to visual generation. In ICLR. 4

  42. [50]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV, 2023. 2, 4

    2023

  43. [51]

    ControlVideo: Training-free controllable text-to-video generation

    Yabo Zhang, Yuxiang Wei, Dongsheng Jiang, XIAOPENG ZHANG, Wangmeng Zuo, and Qi Tian. ControlVideo: Training-free controllable text-to-video generation. In ICLR. 2

  44. [52]

    Open-Sora: Democratizing efficient video production for all, 2024

    Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-Sora: Democratizing efficient video production for all, 2024. 2, 3, 4, 5, 9

    2024

  45. [53]

    Al- legro: Open the black box of commercial-level video gen- eration model

    Yuan Zhou, Qiuyue Wang, Yuxuan Cai, and Huan Yang. Al- legro: Open the black box of commercial-level video gen- eration model. arXiv preprint arXiv:2410.15458, 2024. 2, 10 16

    2024 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.