Pith. sign in

REVIEW 4 major objections 4 minor 65 references

VanGogh: A Unified Multimodal Diffusion-based Framework for Video Colorization

T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read VanGogh is a unified multimodal diffusion framework that colorizes grayscale videos using text, exemplar, and hint conditions, alone or combined, with temporally stable results.

desk verdict A genuinely useful integration of multimodal conditioning for video colorization, but the quantitative SOTA claim rests on an unvalidated metric and the paper overreaches in its own tables. read the letter →

arxiv 2501.09499 v1 pith:ZRBPFVOV submitted 2025-01-16 cs.CV

Zixun Fang , Zhiheng Liu , Kai Zhu , Yu Liu , Ka Leong Cheng , Wei Zhai , Yang Cao , Zheng-Jun Zha This is my paper · ORCID
classification cs.CV
keywords videocolorizationmultimodaldiffusiontemporalconsistencyDualQformercolorbleedingopticalflowlosslumachannelreplacementhints-based
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

VanGogh is a diffusion-based system for turning grayscale videos into color, built so a user can steer the result with a text description, a reference image, an interactive color hint, or any combination of these. The paper argues that previous video colorization methods rely on a single control signal and therefore suffer from color bleeding into neighboring regions and from flicker across frames, while VanGogh fuses all modalities through a Dual Qformer and adds depth, optical flow, and luma-channel safeguards to keep color inside object boundaries and stable over time. Why care: if the claim holds, the same model replaces separate text-, exemplar-, and hint-based colorization pipelines, giving global and local control in one tool. The paper supports the claim with qualitative comparisons, a set of quantitative metrics including a newly proposed Colorfulness/FVMD ratio, and a user study.

What carries the argument

The machinery has four load-bearing parts. A Dual Qformer encodes the text prompt and the exemplar's color features into learnable queries, fuses them with scaling factors $\lambda_1$ and $\lambda_2$, and feeds the fused queries into the cross-attention of Stable Video Diffusion, allowing global text/exemplar control and user-adjustable balance. A Color Projector extracts color distribution from an exemplar by multi-scale patching before CLIP encoding, weakening structural coupling so arbitrary reference images can transfer color. Hints are injected by concatenating a grayscale canvas with superpixel-average color patches and a hint mask to the UNet input, giving local control without a separate ControlNet branch. A Depth Guider and an optical flow loss between consecutive predicted frames constrain the diffusion output to reduce color overflow; at inference the luma channel of the predicted Lab output is replaced by the luma of the input grayscale video to suppress video-VAE flicker.

What would settle it

Run the same four-method comparison (VCGAN, L-CAD+DVP, ColorMNet, SVCNet, VanGogh) on a held-out set of grayscale videos and ask at least 100 naive raters to judge temporal flicker, color bleeding, and condition alignment separately from colorfulness. If raters prefer another method on flicker or bleeding while VanGogh wins only on the Colorfulness/FVMD ratio, the central claim fails; if the ratio ranks methods opposite to human preference, the metric itself is falsified.

Watch

Extended reading notes

Core claim

The central claim is that a single model can do automatic, text-guided, exemplar-guided, and hint-guided video colorization with temporal consistency while outperforming dedicated single-modality methods. On the paper's terms, VanGogh achieves superior temporal consistency and color fidelity by aligning text and image features in a shared space, injecting hints directly as canvas and mask channels, constraining generation with depth and an optical-flow loss, and replacing the output luma channel with the known grayscale structure at inference. The paper concludes that it is the most versatile method and reaches state-of-the-art results across several metrics, including the proposed Colorfulness/FVMD ratio.

Load-bearing premise

The claim of state-of-the-art quality rests on treating the proposed Colorfulness/FVMD ratio as a valid measure of video colorization quality and on a 20-response user study being representative; if that ratio is not perceptually calibrated, the quantitative superiority conclusion is unsupported.

Editorial extensions

If this is right

  • A user can colorize the same grayscale video from text, a reference frame, or a few user-drawn hints, and can combine those conditions for global-plus-local control.
  • Because the method inherits Stable Video Diffusion's temporal prior, it can colorize longer clips without the frame-by-frame error accumulation shown by exemplar-based pipelines.
  • The luma replacement technique converts video-VAE reconstruction flicker in high-frequency regions into a fixable structural issue, since the grayscale input supplies the luma channel.
  • The optical-flow loss gives a training signal that penalizes color crossing object boundaries under large motion, addressing a failure mode the qualitative comparisons show in L-CAD and ColorMNet.
  • The Colorfulness/FVMD ratio, if accepted, gives video colorization a single score that rewards both vivid color and temporal consistency, avoiding the paradox where gray videos win on FVMD alone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the shared feature space transfers, the same Dual Qformer conditioning could be applied to other video diffusion tasks that combine global semantics with local spatial control, such as video inpainting or editing.
  • The luma replacement works because the grayscale input preserves structure; a stronger claim would be to maintain temporal consistency without relying on that known luma.
  • A perceptual calibration of the Colorfulness/FVMD ratio against human flicker and color-bleeding judgments, on a larger sample than 20 questionnaires, would decide whether the metric is generally useful.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces VanGogh, a unified multimodal diffusion-based framework for video colorization built on Stable Video Diffusion. The method accepts text, exemplar, hint, or combined conditions through a Dual Qformer for cross-modal fusion, and incorporates a Color Projector, Depth Guider, optical flow loss, and luma-channel replacement to reduce color bleeding and flicker. Training is performed in two stages on a private image-text dataset and OpenVid-1M. The authors report qualitative comparisons, quantitative metrics on DAVIS, ablations, and a user study, and claim state-of-the-art performance and superior temporal consistency and color fidelity.

Significance. If substantiated, the framework would be a valuable step toward a single, user-controllable video colorization model that handles multiple condition modalities without per-modality re-engineering. The architecture is reasonable and the qualitative results, including the ablations in Sec. 4.4, are internally consistent. However, the central quantitative claim rests on a newly introduced Colorfulness/FVMD metric that is not validated against human perception or established benchmarks, and the reported numbers lack error bars and significance testing. The paper is therefore promising but needs substantial evaluation strengthening before the state-of-the-art claim can be accepted.

major comments (4)
  1. [Sec. 4.3, Table 1] The proposed Colorfulness/FVMD ratio is introduced without any validation. No calibration against human preference, established colorization benchmarks, or existing temporal-consistency metrics is provided, and no error bars or significance tests are reported. Since this ratio is used to support the headline claims of 'superior temporal consistency and color fidelity' and state-of-the-art performance, the quantitative basis for those claims is not established. The authors should validate the metric (e.g., by correlating it with user preferences across methods) and supplement it with established per-frame and temporal metrics.
  2. [Sec. 4.3, Table 1] The text and Table 1 are internally inconsistent regarding FVMD. The paper states that 'Grayish videos tend to achieve higher FVMD scores,' yet the Grayscale row in Table 1 reports the lowest FVMD value (596.1689), which is described as 'the best' under the convention that lower FVMD is better. This contradiction indicates a misunderstanding of the metric's direction and undermines the stated rationale for the Colorfulness/FVMD ratio. Please clarify the direction of FVMD and justify why dividing colorfulness by FVMD yields a meaningful quality measure rather than an artifact of the metric's scale.
  3. [Secs. 4.3 and 4.5] All quantitative results in Tables 1-4 are point estimates with no standard deviations, no number of evaluation videos, and no significance tests. Some reported differences are small (e.g., LPIPS 0.1908 vs. 0.1912 in Table 1), and the user study in Sec. 4.5 is based on only 20 questionnaires with no confidence intervals or statistical test. Without this information, the claimed superiority over baselines is not statistically supported. Please report variance, test-set size, and appropriate significance tests, or qualify the claims accordingly.
  4. [Sec. 4.3, Tables 1-4] The comparability of baseline evaluations is not fully specified. Table 1 is described as an 'automatic colorization' comparison, yet SVCNet is a scribble-based method and ColorMNet is exemplar-based; it is unclear what conditions or initialization were provided to each baseline. Similarly, the text-based comparison in Table 2 relies on video captioning models to generate prompts, but the prompts and alignment protocol are not described in enough detail to assess fairness. Please specify the exact input conditions and evaluation protocol used for each baseline.
minor comments (4)
  1. [Table 1] The word 'Grayscle' is a typo and should read 'Grayscale.'
  2. [Sec. 4.3] The reference 'As shown in Tab. S1' appears to point to the supplementary table, but the table being discussed is Table 1 in the main text; please correct the cross-reference.
  3. [Sec. 3.4] 'Inspired techniques in image colorization' should be 'Inspired by techniques in image colorization.'
  4. [Sec. 4.1] The in-house image dataset is not described or released, which limits reproducibility; please provide dataset details or a release plan.

Circularity Check

1 steps flagged · score 2.0 of 10

Mild evaluation-loop concern: the SOTA claim leans on a self-defined Colorfulness/FVMD ratio, but the core framework has no circular derivation.

  1. other [Sec. 4.3 (Quantitative Comparison), Table 1 and surrounding text]
    "Grayish videos tend to achieve higher FVMD scores. As observed in our experiments, the FVMD metric for grayscale videos is the best, yet it does not satisfy the task of colorization. Therefore, we propose a new metric, denoted as Colorfulness / FVMD, which effectively takes into account both color and temporal relationships. As shown in Tab. S1, our method is not only the most versatile but also achieves state-of-the-art performance across multiple metrics."

    The metric used to support the headline SOTA claim is defined by the same paper, after observing that FVMD alone favors grayscale inputs, and it is not calibrated against human perception or any established colorization benchmark. Colorfulness / FVMD is literally a ratio of the two quantities the method is engineered to improve, and it assigns the grayscale input a score of zero by construction. Thus the favorable ranking of Ours on this metric is a self-consistent definition rather than an independent test. This is not a fitted-parameter-as-prediction case, and the paper also reports external metrics (LPIPS, PSNR, user study), so the circularity is mild and localized to the quantitative SOTA framing.

full rationale

The paper is an empirical systems paper: there is no analytical derivation in which an output is shown to be equivalent to an input, no fitted parameter is renamed as a prediction, and no uniqueness theorem or ansatz is imported from the authors' prior work. The architecture choices (Dual Qformer, Depth Guider, optical flow loss, luma replacement) are validated by ablations and external metrics. The only load-bearing self-referential element is the newly proposed Colorfulness / FVMD metric in Sec. 4.3, which the authors introduce and then use to claim state-of-the-art performance. Because this metric is defined by the authors without external validation and directly combines the quantities their method is designed to maximize and minimize, it creates a mild evaluation loop. However, the paper's other quantitative results (LPIPS, PSNR, SSIM, CLIP score, user study) are independent of this metric and provide partial support for the central claims. Overall, no significant circularity in the method itself; score 2 reflects the localized self-defined metric concern.

Assumptions & free parameters 4 free parameters · 4 assumptions · 4 invented entities

Free parameters are mostly architectural losses and scales set by hand. The axioms are assumptions about the reliability of pretrained models, the link between optical flow error and color overflow, and the validity of the custom evaluation metric. The invented entities are new network modules and a new metric, none of which has evidence independent of this paper.

free parameters (4)
  • lambda1 and lambda2 (text and image scaling factors) = 1.0 during training; adjustable at inference
    Hand-chosen fusion weights for text and image features in the Dual Qformer (Sec. 3.2); they control condition strength and were set to 1 during training without a sensitivity study.
  • Optical flow loss weight gamma = 1
    Hand-set hyperparameter in Eq. (3); no sensitivity analysis is provided.
  • Contextual loss hyperparameters h and w_l = h=0.1; w_l=8,4,2 for layers 5,4,3
    Set manually following prior work; the paper does not examine how robust the results are to these values.
  • Hint sampling parameters K and d = K ~ U(0,150), d ~ U(10,20)
    Randomly sampled during training; these control hint density and cell size and are not justified by any analysis.
assumptions (4)
  • domain assumption Pretrained foundation models (SVD, CLIP, Depth Anything V2, GMFlow, CoTracker) provide reliable and unbiased priors for colorization.
    The pipeline reuses these models without fine-tuning them for colorization; if their priors are biased or fail on grayscale input, the colorization quality claim is weakened. Invoked throughout Sec. 3.2.
  • domain assumption Color overflow manifests as optical flow estimation errors, so minimizing flow error reduces color bleeding.
    Fig. 3 shows one example; the optical flow loss in Eq. (3) assumes this relationship holds broadly, including for large motions.
  • domain assumption Replacing the luma channel of the generated output with the grayscale input luma always improves or preserves quality.
    Sec. 3.4 and Fig. 4 argue that the grayscale luma is structurally correct and the video VAE only errs in chroma; this post-processing is applied during every inference.
  • ad hoc to paper The proposed Colorfulness/FVMD ratio is a valid evaluation metric for video colorization.
    Introduced in Sec. 4.3 without external validation, yet used to conclude that the method is state-of-the-art.
invented entities (4)
  • Dual QFormer
    purpose: Aligns and fuses text and image color features into a shared space for SVD cross-attention.
    Ablation in the supplement supports its contribution, but no independent follow-up or formalized definition outside this paper.
  • Color Projector
    purpose: Extracts color distribution from an exemplar while weakening structural information via multi-scale patches and CLIP.
    Ablation shows an effect, but the claimed decoupling of color from structure is not independently measured.
  • Depth Guider
    purpose: Injects depth features into the denoising UNet to improve spatial-temporal consistency.
    This small convolutional module is supported only by the paper's own ablations.
  • Colorfulness/FVMD metric
    purpose: Composite evaluation metric intended to capture both color richness and temporal consistency.
    Introduced and used only in this paper; no validation against human judgments or other benchmarks is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VanGogh: A Unified Multimodal Diffusion-based Framework for Video Colorization." pith.science (2026). https://pith.science/paper/ZRBPFVOV

@misc{pith2026250109499,
  author       = {Pith},
  title        = {Pith review of: VanGogh: A Unified Multimodal Diffusion-based Framework for Video Colorization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZRBPFVOV}},
  note         = {Machine review of arXiv:2501.09499}
}
read the original abstract

Video colorization aims to transform grayscale videos into vivid color representations while maintaining temporal consistency and structural integrity. Existing video colorization methods often suffer from color bleeding and lack comprehensive control, particularly under complex motion or diverse semantic cues. To this end, we introduce VanGogh, a unified multimodal diffusion-based framework for video colorization. VanGogh tackles these challenges using a Dual Qformer to align and fuse features from multiple modalities, complemented by a depth-guided generation process and an optical flow loss, which help reduce color overflow. Additionally, a color injection strategy and luma channel replacement are implemented to improve generalization and mitigate flickering artifacts. Thanks to this design, users can exercise both global and local control over the generation process, resulting in higher-quality colorized videos. Extensive qualitative and quantitative evaluations, and user studies, demonstrate that VanGogh achieves superior temporal consistency and color fidelity.Project page: https://becauseimbatman0.github.io/VanGogh.

Figures

Figures reproduced from arXiv: 2501.09499 by the authors.

Figure 1
Figure 1. We present VanGogh, a novel multimodal video colorization method that accepts one or more conditional inputs to generate high-quality, vivid colorization results. Given the grayscale video in (a), our method can take a single condition as input to produce the result shown in (b), or accept multiple conditions for joint control, as demonstrated in (c). Please zoom in for more details. Abstract Video colorization aims… view at source ↗
Figure 2
Figure 2. Overall pipeline. We omit the depiction of the VAE encoder and decoder for simplicity. Given a source color video I 1:N gt , we first randomly select one frame as the exemplar image and feed it into the Color Projector, where the exemplar is divided into three groups of patches and then passed through the ResBlocks and CLIP image encoder to obtain the color features. The color features are sent to the Dual Qformer a… view at source ↗
Figure 3
Figure 3. Color overflow caused by large motion results in optical [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: The reconstruction results of the video VAE exhibit [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Comparison results for automatic video colorization. VCGAN and SVCNet exhibit severe grayish issues. L-CAD suffers from flickering artifacts; even though it is post-processed by DVP, color bleeding still persists. ColorMNet heavily relies on the colored exemplar frame,…
Figure 6
Figure 6. Figure 6: Comparison for text-based video colorization. L￾CAD+DVP exhibits color bleeding and temporal incoherence. In contrast, our method can generate vivid and natural results that align with given prompts. AdamW [33] as the optimizer, with a learning rate set to 1 × 10−5 . W…
Figure 8
Figure 8. Figure 8: Comparison for hints-based video colorization. SVCNet suffers from grayish issues and fails to align with the given hints, while our method can synthesize diverse results that align with the provided hints. For text-based colorization, we compare our method with L-CAD …
Figure 9
Figure 9. Figure 9: Ablation studies. Removing the Color Projector results in inaccurate color allocation. Without the optical flow loss, color overflow effects occur. Eliminating the Depth Guider leads to spatial-temporal inconsistency. In contrast, our full method achieves smooth and co…
Figure 10
Figure 10. Figure 10: User study results. The vast majority of users prefer our model. leaves. Removing the Color Projector results in ineffective extraction of color information from the exemplar. Further￾more, omitting the optical flow loss leads to color overflow during significant moti…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

65 extracted references · 50 canonical work pages

  1. [1]

    https : / / huggingface

    Stable-Video-Diffusion-img2vid-xt. https : / / huggingface . co / stabilityai / stable - video-diffusion-img2vid-xt , 2023. 5, 1

    work page 2023

  2. [2]

    Semantic-sparse coloriza- tion network for deep exemplar-based colorization

    Yunpeng Bai, Chao Dong, Zenghao Chai, Andong Wang, Zhengzhuo Xu, and Chun Yuan. Semantic-sparse coloriza- tion network for deep exemplar-based colorization. In Eur. Conf. Comput. Vis., pages 505–521. Springer, 2022. 2, 3

    work page 2022

  3. [3]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023. 2, 4, 1

    arXiv 2023

  4. [4]

    Versatile Vision Foundation Model for Image and Video Colorization

    Vukasin Bozic, Abdelaziz Djelouah, Yang Zhang, Radu Timofte, Markus Gross, and Christopher Schroers. Versatile Vision Foundation Model for Image and Video Colorization. In ACM SIGGRAPH 2024 Conference Papers , pages 1–11,

    work page 2024

  5. [5]

    L-CoIns: Language-based colorization with instance awareness

    Zheng Chang, Shuchen Weng, Peixuan Zhang, Yu Li, Si Li, and Boxin Shi. L-CoIns: Language-based colorization with instance awareness. In IEEE Conf. Comput. Vis. Pattern Recog., pages 19221–19230, 2023. 2

    work page 2023

  6. [6]

    Deep colorization

    Zezhou Cheng, Qingxiong Yang, and Bin Sheng. Deep colorization. In Int. Conf. Comput. Vis. , pages 415–423,

    work page

  7. [7]

    Automatic Controllable Colorization via Imagination

    Xiaoyan Cong, Yue Wu, Qifeng Chen, and Chenyang Lei. Automatic Controllable Colorization via Imagination. In IEEE Conf. Comput. Vis. Pattern Recog., pages 2609–2619,

    work page

  8. [8]

    Learn- ing large-scale automatic image colorization

    Aditya Deshpande, Jason Rock, and David Forsyth. Learn- ing large-scale automatic image colorization. In Int. Conf. Comput. Vis., pages 567–575, 2015

    work page 2015

Show all 65 references
  1. [9]

    Learning diverse image colorization

    Aditya Deshpande, Jiajun Lu, Mao-Chuang Yeh, Min Jin Chong, and David Forsyth. Learning diverse image colorization. In IEEE Conf. Comput. Vis. Pattern Recog. , pages 6837–6845, 2017. 2

    2017

  2. [10]

    A superpixel-based variational model for image colorization

    Faming Fang, Tingting Wang, Tieyong Zeng, and Guixu Zhang. A superpixel-based variational model for image colorization. IEEE Trans. Vis. Comput. Graph. , 26(10): 2931–2943, 2019. 2

    2019

  3. [11]

    A neural algorithm of artistic style

    Leon A Gatys. A neural algorithm of artistic style. arXiv preprint arXiv:1508.06576, 2015. 3

    2015 arXiv

  4. [12]

    Long video generation with time-agnostic vqgan and time- sensitive transformer

    Songwei Ge, Thomas Hayes, Harry Yang, Xi Yin, Guan Pang, David Jacobs, Jia-Bin Huang, and Devi Parikh. Long video generation with time-agnostic vqgan and time- sensitive transformer. InEur. Conf. Comput. Vis., pages 102–

  5. [13]

    Measuring color- fulness in natural images

    David Hasler and Sabine E Suesstrunk. Measuring color- fulness in natural images. In Human vision and electronic imaging VIII, pages 87–95. SPIE, 2003. 7, 1

    2003

  6. [14]

    CLIPscore: A reference-free evaluation met- ric for image captioning

    Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPscore: A reference-free evaluation met- ric for image captioning. arXiv preprint arXiv:2104.08718,

    arXiv

  7. [15]

    Unicolor: A unified framework for multi-modal colorization with trans- former

    Zhitong Huang, Nanxuan Zhao, and Jing Liao. Unicolor: A unified framework for multi-modal colorization with trans- former. ACM Trans. Graph., 41(6):1–16, 2022. 2, 3

    2022

  8. [16]

    Deepremaster: tem- poral source-reference attention networks for comprehensive video enhancement

    Satoshi Iizuka and Edgar Simo-Serra. Deepremaster: tem- poral source-reference attention networks for comprehensive video enhancement. ACM Trans. Graph., 38(6):1–13, 2019. 2, 3

    2019

  9. [17]

    Colorformer: Image colorization via color memory assisted hybrid-attention transformer

    Xiaozhong Ji, Boyuan Jiang, Donghao Luo, Guangpin Tao, Wenqing Chu, Zhifeng Xie, Chengjie Wang, and Ying Tai. Colorformer: Image colorization via color memory assisted hybrid-attention transformer. In Eur. Conf. Comput. Vis. , pages 20–36. Springer, 2022. 2

    2022

  10. [18]

    DDColor: Towards photo- realistic image colorization via dual decoders

    Xiaoyang Kang, Tao Yang, Wenqi Ouyang, Peiran Ren, Lingzhi Li, and Xuansong Xie. DDColor: Towards photo- realistic image colorization via dual decoders. In Int. Conf. Comput. Vis., pages 328–338, 2023. 2, 5, 1

    2023

  11. [19]

    Co- tracker: It is better to track together

    Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Co- tracker: It is better to track together. arXiv preprint arXiv:2307.07635, 2023. 4, 7

    2023 arXiv

  12. [20]

    Neural preset for color style transfer

    Zhanghan Ke, Yuhao Liu, Lei Zhu, Nanxuan Zhao, and Rynson WH Lau. Neural preset for color style transfer. In IEEE Conf. Comput. Vis. Pattern Recog. , pages 14173– 14182, 2023. 3

    2023

  13. [21]

    Slic: Self-supervised learning with iterative clus- tering for human action videos

    Salar Hosseini Khorasgani, Yuxuan Chen, and Florian Shkurti. Slic: Self-supervised learning with iterative clus- tering for human action videos. In IEEE Conf. Comput. Vis. Pattern Recog., pages 16091–16101, 2022. 3

    2022

  14. [22]

    Auto-encoding variational bayes

    Diederik P Kingma. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013. 3

    2013 arXiv

  15. [23]

    Learning representations for automatic colorization

    Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Learning representations for automatic colorization. In Eur. Conf. Comput. Vis. , pages 577–593. Springer, 2016. 2

    2016

  16. [24]

    Fully automatic video colorization with self-regularization and diversity

    Chenyang Lei and Qifeng Chen. Fully automatic video colorization with self-regularization and diversity. In IEEE Conf. Comput. Vis. Pattern Recog., pages 3753–3761, 2019. 2, 3

    2019

  17. [25]

    Deep video prior for video consistency and propagation

    Chenyang Lei, Yazhou Xing, Hao Ouyang, and Qifeng Chen. Deep video prior for video consistency and propagation. IEEE Trans. Pattern Anal. Mach. Intell. , 45(1):356–371,

  18. [26]

    Automatic example-based image colorization using location- aware cross-scale matching

    Bo Li, Yu-Kun Lai, Matthew John, and Paul L Rosin. Automatic example-based image colorization using location- aware cross-scale matching. IEEE Trans. Image Process., 28 (9):4606–4619, 2019. 2 9

    2019

  19. [27]

    Towards Photorealistic Video Colorization via Gated Color- Guided Image Diffusion Models

    Jiaxing Li, Hongbo Zhao, Yijun Wang, and Jianxin Lin. Towards Photorealistic Video Colorization via Gated Color- Guided Image Diffusion Models. In ACM Int. Conf. Multi- media, 2024. 2, 3

    2024

  20. [28]

    Videogen: A reference-guided latent diffusion approach for high definition text-to-video generation

    Xin Li, Wenqing Chu, Ye Wu, Weihang Yuan, Fanglong Liu, Qi Zhang, Fu Li, Haocheng Feng, Errui Ding, and Jingdong Wang. Videogen: A reference-guided latent diffusion approach for high definition text-to-video generation. arXiv preprint arXiv:2309.00398, 2023. 2

    2023 arXiv

  21. [29]

    Control Color: Multimodal Diffusion-based Interactive Image Colorization

    Zhexin Liang, Zhaochen Li, Shangchen Zhou, Chongyi Li, and Chen Change Loy. Control Color: Multimodal Diffusion-based Interactive Image Colorization. arXiv preprint arXiv:2402.10855, 2024. 2, 3, 5, 1

    2024 arXiv

  22. [30]

    Video colorization with pre-trained text-to- image diffusion models

    Hanyuan Liu, Minshan Xie, Jinbo Xing, Chengze Li, and Tien-Tsin Wong. Video colorization with pre-trained text-to- image diffusion models. arXiv preprint arXiv:2306.01732,

    arXiv

  23. [31]

    Fr ´echet Video Motion Distance: A Metric for Evaluating Motion Consistency in Videos.arXiv preprint arXiv:2407.16124, 2024

    Jiahe Liu, Youran Qu, Qi Yan, Xiaohui Zeng, Lele Wang, and Renjie Liao. Fr ´echet Video Motion Distance: A Metric for Evaluating Motion Consistency in Videos.arXiv preprint arXiv:2407.16124, 2024. 7, 1

    2024 arXiv

  24. [32]

    Switch- able temporal propagation network

    Sifei Liu, Guangyu Zhong, Shalini De Mello, Jinwei Gu, Varun Jampani, Ming-Hsuan Yang, and Jan Kautz. Switch- able temporal propagation network. In Eur. Conf. Comput. Vis., pages 87–102, 2018. 2, 3

    2018

  25. [33]

    Decoupled weight decay regularization

    I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 6

    2017 arXiv

  26. [34]

    Unpaired cartoon image synthesis via gated cycle mapping

    Yifang Men, Yuan Yao, Miaomiao Cui, Zhouhui Lian, Xuansong Xie, and Xian-Sheng Hua. Unpaired cartoon image synthesis via gated cycle mapping. In IEEE Conf. Comput. Vis. Pattern Recog., pages 3501–3510, 2022. 3

    2022

  27. [35]

    OpenVid-1M: A large-scale high-quality dataset for text-to- video generation

    Kepan Nan, Rui Xie, Penghao Zhou, Tiehan Fan, Zhenheng Yang, Zhijie Chen, Xiang Li, Jian Yang, and Ying Tai. OpenVid-1M: A large-scale high-quality dataset for text-to- video generation. arXiv preprint arXiv:2407.02371 , 2024. 5

    2024 arXiv

  28. [36]

    Perazzi, J

    F. Perazzi, J. Pont-Tuset, B. McWilliams, L. Van Gool, M. Gross, and A. Sorkine-Hornung. A Benchmark Dataset and Evaluation Methodology for Video Object Segmentation. In IEEE Conf. Comput. Vis. Pattern Recog., 2016. 7, 8, 1

    2016

  29. [37]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In Int. Conf. Mach. Learn., pages 8748–8763. PMLR,

  30. [38]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In IEEE Conf. Comput. Vis. Pattern Recog., pages 10684–10695, 2022. 3

    2022

  31. [39]

    Instance- aware image colorization

    Jheng-Wei Su, Hung-Kuo Chu, and Jia-Bin Huang. Instance- aware image colorization. In IEEE Conf. Comput. Vis. Pattern Recog., pages 7968–7977, 2020. 2

    2020

  32. [40]

    Raft: Recurrent all-pairs field transforms for optical flow

    Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23– 28, 2020, Proceedings, Part II 16, pages 402–419. Springer,

    2020

  33. [41]

    Towards accurate generative models of video: A new metric & challenges

    Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Rapha¨el Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. ArXiv, abs/1812.01717, 2018. 1

    2018 arXiv

  34. [42]

    Tracking emerges by colorizing videos

    Carl V ondrick, Abhinav Shrivastava, Alireza Fathi, Sergio Guadarrama, and Kevin Murphy. Tracking emerges by colorizing videos. In Eur. Conf. Comput. Vis., pages 391– 408, 2018. 2, 3

    2018

  35. [43]

    Unsupervised deep exemplar colorization via pyramid dual non-local attention

    Hanzhang Wang, Deming Zhai, Xianming Liu, Junjun Jiang, and Wen Gao. Unsupervised deep exemplar colorization via pyramid dual non-local attention. IEEE Trans. Image Process., 2023. 2

    2023

  36. [44]

    CT 2: Colorization transformer via color tokens

    Shuchen Weng, Jimeng Sun, Yu Li, Si Li, and Boxin Shi. CT 2: Colorization transformer via color tokens. In Eur. Conf. Comput. Vis., pages 1–16. Springer, 2022. 2

    2022

  37. [45]

    L-CAD: Language-based colorization with any-level descriptions using diffusion priors

    Shuchen Weng, Peixuan Zhang, Yu Li, Si Li, Boxin Shi, et al. L-CAD: Language-based colorization with any-level descriptions using diffusion priors. Adv. Neural Inform. Process. Syst., 36, 2024. 2, 6, 7, 8, 1

    2024

  38. [46]

    Dynamicrafter: Animating open-domain images with video diffusion priors

    Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Wangbo Yu, Hanyuan Liu, Gongye Liu, Xintao Wang, Ying Shan, and Tien-Tsin Wong. Dynamicrafter: Animating open-domain images with video diffusion priors. In Eur. Conf. Comput. Vis., pages 399–417. Springer, 2025. 2

    2025

  39. [47]

    GMFlow: Learning optical flow via global matching

    Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, and Dacheng Tao. GMFlow: Learning optical flow via global matching. In IEEE Conf. Comput. Vis. Pattern Recog., pages 8121–8130, 2022. 2, 5

    2022

  40. [48]

    Stylization-based architecture for fast deep exemplar colorization

    Zhongyou Xu, Tingting Wang, Faming Fang, Yun Sheng, and Guixu Zhang. Stylization-based architecture for fast deep exemplar colorization. In IEEE Conf. Comput. Vis. Pattern Recog., pages 9363–9372, 2020. 2

    2020

  41. [49]

    Depth Anything V2

    Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xi- aogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth Anything V2. arXiv preprint arXiv:2406.09414, 2024. 5

    2024 arXiv

  42. [50]

    Bistnet: Semantic image prior guided bidirectional temporal feature fusion for deep exemplar-based video colorization

    Yixin Yang, Jinshan Pan, Zhongzheng Peng, Xiaoyu Du, Zhulin Tao, and Jinhui Tang. Bistnet: Semantic image prior guided bidirectional temporal feature fusion for deep exemplar-based video colorization. IEEE Trans. Pattern Anal. Mach. Intell., 2024. 2, 3

    2024

  43. [51]

    ColorMNet: A Memory-based Deep Spatial-Temporal Fea- ture Propagation Network for Video Colorization

    Yixin Yang, Jiangxin Dong, Jinhui Tang, and Jinshan Pan. ColorMNet: A Memory-based Deep Spatial-Temporal Fea- ture Propagation Network for Video Colorization. In Eur. Conf. Comput. Vis., pages 336–352. Springer, 2025. 2, 3, 6, 7, 8

    2025

  44. [52]

    Photorealistic style transfer via wavelet transforms

    Jaejun Yoo, Youngjung Uh, Sanghyuk Chun, Byeongkyu Kang, and Jung-Woo Ha. Photorealistic style transfer via wavelet transforms. In Int. Conf. Comput. Vis., pages 9036– 9045, 2019. 3

    2019

  45. [53]

    iColoriT: Towards propagating local hints to the right region in interactive colorization by leveraging vision transformer

    Jooyeol Yun, Sanghyeon Lee, Minho Park, and Jaegul Choo. iColoriT: Towards propagating local hints to the right region in interactive colorization by leveraging vision transformer. In IEEE Winter Conf. Appl. Comput. Vis., pages 1787–1796,

  46. [54]

    Diffusing Colors: Image Colorization with Text Guided Diffusion

    Nir Zabari, Aharon Azulay, Alexey Gorkor, Tavi Halperin, and Ohad Fried. Diffusing Colors: Image Colorization with Text Guided Diffusion. In SIGGRAPH Asia 2023 Conference Papers, pages 1–11, 2023. 2, 5, 1 10

    2023

  47. [55]

    Deep exemplar- based video colorization

    Bo Zhang, Mingming He, Jing Liao, Pedro V Sander, Lu Yuan, Amine Bermak, and Dong Chen. Deep exemplar- based video colorization. InIEEE Conf. Comput. Vis. Pattern Recog., pages 8052–8061, 2019. 2, 3, 5

    2019

  48. [56]

    A tale of two features: Stable diffusion complements dino for zero-shot semantic correspondence

    Junyi Zhang, Charles Herrmann, Junhwa Hur, Luisa Pola- nia Cabrera, Varun Jampani, Deqing Sun, and Ming-Hsuan Yang. A tale of two features: Stable diffusion complements dino for zero-shot semantic correspondence. Adv. Neural Inform. Process. Syst., 36, 2024. 6

    2024

  49. [57]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Int. Conf. Comput. Vis., pages 3836–3847, 2023. 2

    2023

  50. [58]

    Colorful image colorization

    Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In Eur. Conf. Comput. Vis., pages 649–

  51. [59]

    CV- V AE: A Compatible Video V AE for Latent Generative Video Models

    Sijie Zhao, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Muyao Niu, Xiaoyu Li, Wenbo Hu, and Ying Shan. CV- V AE: A Compatible Video V AE for Latent Generative Video Models. arXiv preprint arXiv:2405.20279, 2024. 2, 5, 1

    2024 arXiv

  52. [60]

    VC- GAN: Video colorization with hybrid generative adversarial network

    Yuzhi Zhao, Lai-Man Po, Wing-Yin Yu, Yasar Abbas Ur Rehman, Mengyang Liu, Yujia Zhang, and Weifeng Ou. VC- GAN: Video colorization with hybrid generative adversarial network. IEEE Trans. Multimedia, 25:3017–3032, 2022. 2, 3, 6

    2022

  53. [61]

    SVCNet: Scribble-based video colorization network with temporal aggregation

    Yuzhi Zhao, Lai-Man Po, Kangcheng Liu, Xuehui Wang, Wing-Yin Yu, Pengfei Xian, Yujia Zhang, and Mengyang Liu. SVCNet: Scribble-based video colorization network with temporal aggregation. IEEE Trans. Image Process. ,

  54. [63]

    Luma: This represents the lightness of the image, ranging from 0 (black) to 100 (white)

  55. [64]

    Negative values indicate green, and positive values indicate red

    a: This axis represents the color’s position between green and red. Negative values indicate green, and positive values indicate red

  56. [65]

    Red ducks

    b: This axis represents the position between blue and yellow. Negative values indicate blue, and positive values indicate yellow. In the task of colorization, the luma channel of the grayscale input retains the structural information intact, which is why many image-based color...

    1990

  57. [2023]

    A, including the parameter settings of the network during training and the analysis of luma channel replacement

    2, 3, 6, 7, 8 11 VanGogh: A Unified Multimodal Diffusion-based Framework for Video Colorization Supplementary Material In this supplementary material, we first describe the specific details of our implementation in App. A, including the parameter settings of the network during...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.