REVIEW 6 major objections 5 minor 26 references
SketchColour: Channel Concat Guided DiT-based Sketch-to-Colour Pipeline for 2D Animation
T0 review · 6 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read The paper claims that a diffusion transformer with channel-concatenated sketch latents produces the best sketch-to-colour results on the SAKUGA dataset while training on roughly half the data of earlier models.
desk verdict A sensible DiT-based sketch colorization idea with a clean architecture story, but the evaluation protocol and reported statistics do not support the abstract's across-all-metrics claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the channel-concatenation adapter on a DiT backbone. The colored first frame, the noisy video latent, and the sketch sequence are each encoded by the same frozen 3D VAE, then concatenated along the channel dimension; the projection for the new sketch channel is zero-initialized in the style of ControlNet, while LoRA updates the attention projections and feedforward layers. This replaces the duplicated ControlNet that earlier U-Net pipelines required, and it is what the paper credits for lower parameter count, lower GPU memory, and fewer colour-bleed artifacts.
What would settle it
Decode the frozen VAE latent of a sketch back to pixels and compare it with the original line art: if thin strokes or small features vanish, the conditioning channel is already corrupted, and an ablation with a separately trained sketch encoder would reveal what the pipeline is losing.
Extended reading notes
Core claim
On the paper's own terms, the central claim is that a frozen 3D VAE encodes line-art sketches faithfully enough to serve as a conditioning signal, so the sketch latent can be concatenated channel-wise with the colored reference latent and the noisy video latent as input to a pretrained DiT video model. A zero-initialized patch projection adds the new sketch channel, and a small LoRA (about 10 million parameters) fine-tunes attention. With this setup, SketchColour reports the best MSCE, PSNR, SSIM, LPIPS, and FVD results at matched frame counts against AniDoc, LVCD, and ToonCrafter on SAKUGA, and attributes the gain to the DiT's global attention scope, which avoids the latent-gap colour bleed seen in ControlNet baselines.
Load-bearing premise
The load-bearing premise is that the frozen 3D VAE encodes sketch line art into latents that still carry enough detail, so channel-wise concatenation can supply the colouring condition without a dedicated sketch encoder.
Editorial extensions
If this is right
- With one colored keyframe and a sketch sequence as input, the pipeline produces a fully colored, temporally coherent animation clip.
- Because conditioning is injected by channel concatenation plus a roughly 10-million-parameter LoRA, the method avoids the billions of parameters a duplicated ControlNet adds.
- The reported SAKUGA results indicate the DiT backbone can outperform U-Net/ControlNet baselines across all five metrics while training on about 80K video clips.
- The frozen 3D VAE is claimed to need no dedicated sketch encoder, so a full sketch-conditioned model can be obtained by fine-tuning a small adapter set.
- Global attention in the DiT reduces colour bleed and object deformation compared with the spatial down-sampling U-Net baselines.
Reading between the lines
- Beyond the paper, the same channel-concat recipe could carry other sparse conditioning modalities, such as depth, pose, or motion hints, into DiT video generators without a ControlNet.
- If the frozen-VAE claim holds, the cost of adding a new sparse control signal to a new DiT model drops to a small adapter and LoRA training run.
- A natural extension the paper does not test is sliding-window propagation of the colored frame to much longer clips, which would test whether the temporal coherence holds beyond 17 frames.
- The method does not address deliberate palette changes mid-scene; combining it with text descriptions attached to the dataset could give animators explicit color control.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. SketchColour proposes a DiT-based pipeline for sketch-to-colour animation generation. The method replaces the U-Net denoiser of prior sketch-colorization systems with CogVideoX-I2V, a diffusion transformer, and injects sketch conditioning by channel-wise concatenation of the frozen VAE latents of the sketch sequence with the reference-frame and noisy-video latents, plus LoRA fine-tuning of attention and feed-forward layers. Evaluated on a subsample of the SAKUGA dataset, the paper reports that SketchColour outperforms AniDoc, LVCD, and ToonCrafter on MSCE, PSNR, SSIM, LPIPS, and FVD at frame lengths matching the baselines, while using fewer trainable parameters (10M LoRA) and, per the abstract, only half the training data.
Significance. If the reported results are reproducible, the paper makes a useful contribution to animation production: it is the first sketch-to-colour pipeline on a DiT backbone, it avoids a duplicated ControlNet, and the quantitative gains on colorization fidelity metrics are meaningful. The method is parameter-efficient and could lower the barrier for practical adoption. However, the contribution is primarily empirical, and several load-bearing claims—the across-all-metrics superiority, the data-efficiency comparison, and the frozen-VAE fidelity—are not sufficiently verified by the manuscript as written. The paper also lacks evaluation artifacts (seed, clip list, code) needed to confirm the numbers.
major comments (6)
- [Abstract; §4.3, Table 1] The abstract claims SketchColour 'outperforms previous state-of-the-art video colourization methods across all metrics,' but Table 1 and §4.3 show that at the model's native 17-frame length, SketchColour's FVD is 918.70±771.13, which is worse than AniDoc's 14-frame FVD of 898.19±704.30; the text itself states that on FVD the model 'only loses to AniDoc with a slight margin.' Because the 17-frame row is the native output of the model and the comparison against AniDoc is at a different frame length, the across-all-metrics claim is contradicted by the paper's own results and must be qualified or removed.
- [§4.1, Table 1] The evaluation protocol is not reproducible as described. The paper uses a random 1K-clip subsample of a filtered SAKUGA test set, but it does not provide the sampling seed, the list of clips, or the evaluation code; the provided link is a project page, not a code repository. It is also not stated whether the AniDoc, LVCD, and ToonCrafter numbers in Table 1 were produced by the authors on this same 1K split using official checkpoints and hyperparameters, or whether they were transcribed from prior papers. Without this information, the quantitative comparison cannot be independently verified.
- [Abstract; §4.1] The abstract's claim that SketchColour uses 'only half the training data of competing models' is unsupported. Section 4.1 reports that SketchColour was trained on 80K sampled SAKUGA videos, but the training dataset sizes for AniDoc, LVCD, and ToonCrafter are never given. The relative data-efficiency claim should be substantiated with concrete numbers for the baselines or removed.
- [§4.2, Figure 4] The claim that the frozen 3D VAE preserves sketch information is supported only by a PCA visualization, which shows qualitative resemblance but provides no quantitative measure of reconstruction fidelity or of the specific line-art details needed for colorization. This assumption is load-bearing because channel-concat conditioning depends on the sketch latent retaining fine structure. The paper should provide quantitative evidence of sketch-latent fidelity (e.g., reconstruction error or downstream metric comparison) or an ablation, such as training with a dedicated sketch encoder, to justify the assertion that 'there is no need to fine-tune a specialized sketch encoder.'
- [§4.1, Table 1] The table reports SketchColour results at 14, 16, and 17 frames, while the model is trained to generate clips of 17 frames. The paper does not explain how the 14- and 16-frame outputs are produced (truncation, padding, re-generation, or a separate inference protocol), nor does it state whether the baselines were evaluated on the same 1K clips at native resolution with identical preprocessing. This ambiguity undermines the 'fair comparison' statement in the table caption.
- [§3.1, §4.1] The parameter-efficiency claim—'LoRA of only 10 million parameters compared to the billions of parameters used by ControlNet'—is incomplete. The method also trains new zero-initialized patch-projection weights for the added sketch channel; whether these weights are included in the 10M count is not specified. No direct comparison of total trainable parameters, GPU memory usage, or training time against the baselines is reported, so the stated contribution of 'reducing parameter count and GPU memory usage' is not fully quantified.
minor comments (5)
- [Abstract] 'The production of high-quality 2D animation is highly labor-intensive process' should read 'a highly labor-intensive process.'
- [§4.1] 'we used a Lora of rank 192' should be 'LoRA,' and 'DDP training' should be spelled out as distributed data-parallel training.
- [Figure 4 caption] 'repsectively' is a typo for 'respectively'; the abbreviation 'VAE' is inconsistently spaced as 'V AE' throughout the paper.
- [§4.3] The sentence 'with the score difference against the baselines being half of more of those baselines' standard deviation' is unclear and should be rewritten to specify which differences are being compared.
- [References, §1] Reference [18] has a malformed author list, and [25] appears to list author names in the wrong order; the project page link in the abstract is described as 'code' but does not appear to link to an implementation.
Circularity Check
No significant circularity; the paper's claims rest on external benchmarks and architectural comparisons, not on self-referential derivations.
full rationale
SketchColour is an empirical systems paper. The proposed method concatenates frozen 3D VAE latents of the colored reference frame and sketch frames, fine-tunes a DiT backbone with LoRA, and evaluates on external SAKUGA test clips using standard metrics. None of the core claims reduce to the paper's inputs by construction: the quantitative results in Table 1 come from held-out test videos and public baseline models, not from fitted parameters that are then renamed as predictions. The claim that a frozen VAE can encode sketch line art is supported by a PCA visualization rather than a rigorous proof, but this is an evidence-strength limitation, not a circular step. The paper also contains an internal inconsistency: the abstract claims improvement 'across all metrics,' while Section 4.3 admits that on FVD at 17 frames the method 'only loses to AniDoc with a slight margin.' That is an overclaim or a reporting error, not a circularity. The comparison protocol is under-specified (custom 1K sample, no sampling seed, unspecified baseline reproduction conditions), which threatens verifiability but does not make the derivation circular. There are no load-bearing self-citations, no imported uniqueness theorems, and no ansatz smuggled in via citation of the authors' own prior work. Accordingly, the circularity score is 0.
Assumptions & free parameters
free parameters (5)
- LoRA rank =
192
- Learning rate =
1e-4
- Training steps =
40,000
- Batch size =
2
- Frame length =
17
assumptions (5)
- domain assumption The frozen 3D VAE preserves sketch line-art structure well enough for channel-concat conditioning.
- domain assumption CogVideoX-I2V-5B provides a suitable pretrained prior for 2D animation colorization.
- domain assumption Anime2Sketch plus binarization produces sketches comparable to those used by prior baselines.
- domain assumption The sampled 80K/1K split of SAKUGA is a fair basis for comparing against baselines.
- domain assumption Standard metrics (MSCE, PSNR, SSIM, LPIPS, FVD) capture the practical quality of animation colorization.
Cite this review
Pith. "Pith review of SketchColour: Channel Concat Guided DiT-based Sketch-to-Colour Pipeline for 2D Animation." pith.science (2026). https://pith.science/paper/V3UBBF54
@misc{pith2026250701586,
author = {Pith},
title = {Pith review of: SketchColour: Channel Concat Guided DiT-based Sketch-to-Colour Pipeline for 2D Animation},
year = {2026},
howpublished = {\url{https://pith.science/paper/V3UBBF54}},
note = {Machine review of arXiv:2507.01586}
}
read the original abstract
The production of high-quality 2D animation is highly labor-intensive process, as animators are currently required to draw and color a large number of frames by hand. We present SketchColour, the first sketch-to-colour pipeline for 2D animation built on a diffusion transformer (DiT) backbone. By replacing the conventional U-Net denoiser with a DiT-style architecture and injecting sketch information via lightweight channel-concatenation adapters accompanied with LoRA finetuning, our method natively integrates conditioning without the parameter and memory bloat of a duplicated ControlNet, greatly reducing parameter count and GPU memory usage. Evaluated on the SAKUGA dataset, SketchColour outperforms previous state-of-the-art video colourization methods across all metrics, despite using only half the training data of competing models. Our approach produces temporally coherent animations with minimal artifacts such as colour bleeding or object deformation. Our code is available at: https://bconstantine.github.io/SketchColour .
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
https://zhenglinpan.github.io/ AnitaDataset_homepage/
Anita dataset. https://zhenglinpan.github.io/ AnitaDataset_homepage/. Accessed: 2025-06-13. 1
work page 2025
-
[2]
Stable video diffusion: Scaling latent video diffusion models to large datasets, 2023
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, Varun Jampani, and Robin Rombach. Stable video diffusion: Scaling latent video diffusion models to large datasets, 2023. 2, 3
work page 2023
-
[3]
Yu Cao, Xiangqiao Meng, P. Y . Mok, Tong-Yee Lee, Xuet- ing Liu, and Ping Li. Animediffusion: Anime diffusion col- orization. IEEE Transactions on Visualization and Computer Graphics, 30(10):6956–6969, Oct. 2024. 2
work page 2024
-
[4]
Latent video diffusion models for high-fidelity long video generation
Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity long video generation. 2022. 2
2022
-
[5]
Lvcd: Reference-based lineart video colorization with diffusion models
Zhitong Huang, Mohan Zhang, and Jing Liao. Lvcd: Reference-based lineart video colorization with diffusion models. ACM Transactions on Graphics, 43(6):1–11, Nov
-
[6]
Tag2pix: Line art colorization using text tag with secat and changing loss
Hyunsu Kim, Ho Young Jhoo, Eunhyeok Park, and Sungjoo Yoo. Tag2pix: Line art colorization using text tag with secat and changing loss. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 9056–9065, Seoul, South Korea, October 2019. 3
work page 2019
-
[7]
Hunyuanvideo: A systematic framework for large video generative models
Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. CoRR, abs/2412.03603, 2024. 2
arXiv 2024
-
[8]
Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In In- ternational conference on machine learning , pages 19730– 19742. PMLR, 2023. 4
2023
Show all 26 references
-
[9]
Magicmotion: Controllable video generation with dense-to-sparse trajectory guidance
Quanhao Li, Zhen Xing, Rui Wang, Hui Zhang, Qi Dai, and Zuxuan Wu. Magicmotion: Controllable video generation with dense-to-sparse trajectory guidance. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Honolulu, Hawai‘i, USA, October 2025. 1
2025
-
[10]
Eliminating gradient conflict in reference-based line-art colorization
Zekun Li, Zhengyang Geng, Zhao Kang, Wenyu Chen, and Yibo Yang. Eliminating gradient conflict in reference-based line-art colorization. In European Conference on Computer Vision, pages 579–596. Springer, 2022. 2
2022
-
[11]
Motionclone: Training-free motion cloning for controllable video generation
Pengyang Ling, Jiazi Bu, Pan Zhang, Xiaoyi Dong, Yuhang Zang, Tong Wu, Huaian Chen, Jiaqi Wang, and Yi Jin. Motionclone: Training-free motion cloning for controllable video generation. In The Thirteenth International Confer- ence on Learning Representations, 2025. 1
2025
-
[12]
Sora: A review on background, technology, limitations, and opportunities of large vision models, 2024
Yixin Liu, Kai Zhang, Yuan Li, Zhiling Yan, Chujie Gao, Ruoxi Chen, Zhengqing Yuan, Yue Huang, Hanchi Sun, Jian- feng Gao, Lifang He, and Lichao Sun. Sora: A review on background, technology, limitations, and opportunities of large vision models, 2024. 2
2024
-
[13]
Anidoc: Animation creation made easier
Yihao Meng, Hao Ouyang, Hanlin Wang, Qiuyu Wang, Wen Wang, Ka Leong Cheng, Zhiheng Liu, Yujun Shen, and Huamin Qu. Anidoc: Animation creation made easier. In Proceedings of the Computer Vision and Pattern Recogni- tion Conference (CVPR), pages 18187–18197, June 2025. 1, 2, 3, 4, 6
2025
-
[14]
Scalable diffusion mod- els with transformers
William Peebles and Saining Xie. Scalable diffusion mod- els with transformers. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision (ICCV) , pages 4195–4205, October 2023. 2
2023
-
[15]
Fvd: A new metric for video generation
Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Rapha¨el Marinier, Marcin Michalski, and Sylvain Gelly. Fvd: A new metric for video generation. 2019. 5
2019
-
[16]
Wan: Open and advanced large-scale video generative models
Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianx- iao Yang, Jianyuan Zeng, Jiayu Wang, Jingfeng Zhang, Jin- gren Zhou, Jinkai Wang, Jixuan Chen, Kai Zhu, Kang Zhao, Keyu Yan, Lianghua Huang, Mengyang Feng, Ningyi Zhang, P...
2025 arXiv
-
[17]
Framer: Interactive video interpolation
Wen Wang, Qiuyu Wang, Kecheng Zheng, Hao Ouyang, Zhekai Chen, Biao Gong, Hao Chen, Yujun Shen, and Chun- hua Shen. Framer: Interactive video interpolation. In In- ternational Conference on Learning Representations (ICLR),
-
[18]
Anime2sketch: A sketch extractor for anime arts with deep networks
Xiao Yang Yiheng Zhu Xiaohui Shen Xiaoyu Xiang, Ding Liu. Anime2sketch: A sketch extractor for anime arts with deep networks. https : / / github . com / Mukosame/Anime2Sketch, 2021. 4
2021
-
[19]
Tooncrafter: Generative cartoon interpolation
Jinbo Xing, Hanyuan Liu, Menghan Xia, Yong Zhang, Xin- tao Wang, Ying Shan, and Tien-Tsin Wong. Tooncrafter: Generative cartoon interpolation. ACM Transactions on Graphics (TOG), 43(6):1–11, 2024. 2, 3, 6
2024
-
[20]
Dynamicrafter: Animating open-domain im- ages with video diffusion priors
Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Wangbo Yu, Hanyuan Liu, Xintao Wang, Tien-Tsin Wong, and Ying Shan. Dynamicrafter: Animating open-domain im- ages with video diffusion priors. InComputer Vision – ECCV 2024, pages 399–417. Springer, 2024. 3
2024
-
[21]
Colorizediffusion: Improv- ing reference-based sketch colorization with latent diffusion model
Dingkun Yan, Liang Yuan, Erwin Wu, Yuma Nishioka, Is- sei Fujishiro, and Suguru Saito. Colorizediffusion: Improv- ing reference-based sketch colorization with latent diffusion model. In Proceedings of the Winter Conference on Applica- tions of Computer Vision (WACV), pages 509...
2025
-
[22]
Cogvideox: Text-to-video diffusion models with an expert transformer
Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiao- han Zhang, Guanyu Feng, Da Yin, Yuxuan Zhang, Weihan Wang, Yean Cheng, Bin Xu, Xiaotao Gu, Yuxiao Dong, and Jie Tang. Cogvideox: Text-to-video diffusion models with an ...
2025
-
[23]
Adding conditional control to text-to-image diffusion models, 2023
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models, 2023. 2, 3
2023
-
[24]
Magiccolor: Multi-instance sketch colorization
Yinhan Zhang, Yue Ma, Bingyuan Wang, Qifeng Chen, and Zeyu Wang. Magiccolor: Multi-instance sketch colorization. CoRR, abs/2503.16948, 2025. 3
2025 arXiv
-
[25]
Sakuga-42m dataset: Scaling up cartoon research
Yuxuan Mu Zhenglin Pan, Yu Zhu. Sakuga-42m dataset: Scaling up cartoon research. arXiv preprint arXiv:2405.07425, 2024. 4
2024 arXiv
-
[26]
Cobra: Efficient line art colorization with broader references
Junhao Zhuang, Lingen Li, Xuan Ju, Zhaoyang Zhang, Chun Yuan, and Ying Shan. Cobra: Efficient line art colorization with broader references. CoRR, abs/2504.12240, 2025. 3 8
2025 arXiv
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.