REVIEW 4 major objections 7 minor 40 references
Enabling Region-Specific Control via Lassos in Point-Based Colorization
T0 review · 4 major / 7 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read This paper introduces a lasso tool for point-based interactive colorization that confines each color hint to a user-defined region via a localization attention mask, claiming one lasso is as effective as 4.18 color points and reduces time…
desk verdict The paper's central mask equation doesn't actually hard-mask as claimed, which undermines the main result, but the idea is sound enough to warrant a careful revision rather than a desk reject. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the localization attention mask, a binary mask built from each user lasso and applied inside a Transformer decoder's cross-attention layer. In that layer, grayscale image patches are the queries and color-hint tokens are the keys and values; the mask multiplies the attention logits $\frac{QK^T}{\sqrt{d}}$ before softmax, setting attention to zero outside each lasso and to one inside, while an unconditional mask lets un-lassoed areas be colored by the unconditional token. This masking is what confines color propagation to the user-defined region and, the paper argues, is what lets the model handle loose boundaries and vary lasso sizes.
What would settle it
A controlled comparison would settle it: on the collected repetitive-pattern images, have participants colorize with freehand lassos that are allowed to be non-rectangular, off-center, or not fully enclosing the target region, and measure PSNR over time against the paper's reported numbers. If imprecise or incomplete lassos produce color bleeding comparable to the point-only baseline, the simulation-to-real generalization fails.
Extended reading notes
Core claim
The paper introduces a lasso interaction for point-based interactive colorization and a localization attention mask that makes it work. In the model, grayscale image patches act as queries and color hints as keys and values in a Transformer decoder's cross-attention layer; a binary mask derived from each lasso is multiplied onto the attention logits $\frac{QK^T}{\sqrt{d}}$ before softmax, so a hint can only spread inside the user's encircled region. Training simulates lassos as random rectangles centered on the hint with the ground-truth color always inside, and the objective is plain Huber loss—no auxiliary region-control loss—because the mask itself is the control mechanism. The paper reports that on a user study over 98 repetitive-pattern images, the lasso setup reached the same PSNR as the point-only approach in about 30% less time, using 3.64 lassos plus 5.69 points versus 20.9 points when only points were available, which the authors translate to one lasso being as effective as about 4.18 points. The paper also claims the model remains competitive with the previous state-of-the-art when lassos are not used.
Load-bearing premise
Everything rests on the assumption that lassos drawn by real people behave like the synthetic rectangles used in training—centered on the hint, with the intended color region fully enclosed—so that the mask alone, with no extra loss, is enough to control propagation.
Editorial extensions
If this is right
- Users can colorize repetitive patterns such as flower petals, tiles, or fruit baskets with significantly fewer interactions, because each lasso confines a color to its intended region and prevents color collapse.
- The lasso tool yields the same quality as point-only interaction in about 30% less time, based on a user study with 98 challenging images, making interactive colorization more practical for real-world editing.
- Model performance without lassos stays comparable to the previous state-of-the-art, so the lasso is an optional precision tool rather than a trade-off against point-based speed.
- Lasso size controls the boldness of a color stroke, letting users adjust how far a hint propagates per interaction and enabling personalized recolorization of similar objects in different colors.
- The authors suggest that the same cross-attention masking mechanism can be extended to a broader range of user hints, pointing toward a general way to inject region constraints into interactive image editing.
Reading between the lines
- The 4.18-points-per-lasso equivalence likely underestimates the gain in scenes with higher repetition or in deliberate recolorization tasks, because the lasso removes the need to add corrective points that a point-only workflow requires.
- The training simulation could be made more realistic by sampling freehand or partially occluding lassos; if that closes the simulation-to-real gap, the approach may still need an auxiliary loss or a mask-refinement module to handle imprecise inputs.
- The same localization-mask idea could transfer to other point-based image editing tasks such as local inpainting, dehazing, or text-guided region editing, where users want to specify a region of effect without drawing precise masks.
- An interesting testable extension is applying the mask at inference time to an off-the-shelf colorizer without training; the paper's ablation suggests training the mask in is necessary, but that leaves open whether lightweight fine-tuning could retrofit existing models.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper augments point-based interactive colorization with a lasso interaction that supplies a localization attention mask. Grayscale image patches serve as queries and color-hint tokens as keys/values in a transformer decoder; the mask is multiplied into the pre-softmax attention logits so that, according to the claim, each hint only colors inside its lasso. Training simulates lassos as random rectangles around sampled hints; at inference a fixed-size pre-defined lasso is used when the user draws none. The authors report point-only performance on par with iColoriT, gains on a synthetic color-collapse grid, and a user study claiming one lasso is worth about 4.18 points and saves roughly 30% of interaction time.
Significance. The lasso-conditioned cross-attention mask is a natural and potentially practical extension of point-based colorization, and the paper includes useful assets: a synthetic color-collapse benchmark, HPR-based analysis of lasso size, and a user study with challenging natural images. If the implementation matches the intended hard masking, the method could be a solid contribution. However, Eq. (1) as written does not implement hard masking, and the two headline efficiency numbers (4.18 points and 30% time saving) are supported only by internally inconsistent arithmetic and under-reported user-study statistics. These issues are central, not cosmetic.
major comments (4)
- [Method, Localized cross-attention, Eq. (1)] The mask M_l is described as 1 inside the lasso and 0 outside, and Eq. (1) applies it as an elementwise product with QK^T before softmax. Under the standard softmax, positions with logit 0 still receive probability exp(0) in the numerator and denominator, so outside-lasso keys are not suppressed; they simply receive equal unnormalized weight with any other zero-logit position. The stated mechanism, restricting the cross-attention map to only attend within the user-provided lasso, therefore does not follow from the equation. If the actual implementation uses masked_fill(-inf) or an equivalent additive bias, Eq. (1) and the mask description are incorrect as written; if it literally multiplies by 0/1, the central claim is unsupported. Please state the exact masking operation and, if it is the 0/1 multiply, provide attention-map visualizations or an ablation that demonstrates the mask's causal effect.
- [Method, Overall Workflow; Supplementary Table 2] The fixed pre-defined lasso size is selected by testing point increments and measuring PSNR on the benchmark dataset, and the same benchmark is then used for the comparisons in Figure 7 and for the predefined-lasso condition in Figure 8. This is test-set selection, which can only inflate the reported point-only performance and also affects the user study's default-lasso condition. The authors should select r on a held-out validation split, or show that the main conclusions are stable across a wide range of r.
- [Experiments, Effectiveness Evaluation; Figure 6] The efficiency claims do not add up: 43.76 - 26.7 = 17.06 s, not 29.50 s; 20.9 - 5.69 = 15.21 points, not 15.12; and 15.12/3.64 = 4.15, not 4.18 (while 15.21/3.64 = 4.18). Because the abstract's '4.18 points' and '30% less time' claims come from these numbers, the authors must reconcile the reported means and clarify how the comparisons were computed. They should also report per-participant distributions and significance tests, since the current user-study section gives no number of participants, no variance, and no statistical test.
- [Experiments, User study] The user study is the sole evidence for the central interaction-efficiency claim, but it reports only aggregate PSNR-over-time curves and a single '85.7% of participants' percentage, without N, standard errors, or inferential statistics. As a result, the claims 'as effective as 4.18 color points' and '30% less time' are not statistically substantiated. Please report participant counts, number of images per participant, per-condition means with confidence intervals, and appropriate significance tests such as paired tests for time and interactions.
minor comments (7)
- [Supplementary Materials, Additional quantitative results] The LPIPS citation is left as an unfilled '(?)' placeholder and should be completed.
- [Method, Localization attention mask] The sentence 'final localization attention mask, Ml, is constructed by concatenating Mc and Mh' refers to an undefined Mh; it should presumably be Mu.
- [Method, Overall Workflow] The notation 'Ml ∈ N(h+1)×N' should be written as a real-valued matrix, e.g., R^(h+1)×N, and the phrase 'explicitly masking the attention map QK^T from the lassos' is grammatically unclear.
- [Experiments, Ablation studies] The phrase 'without the lasso, which is the same as using the image lasso' is confusing; please clarify whether 'no lasso' means an all-ones mask or no mask at all.
- [Experiments, User study] The sentence 'If users did not provide a lasso interaction for the corresponding color hint, our model used a fixed-size, pre-defined lasso' makes the point-only condition ambiguous; specify exactly what mask is used in the point-only arm.
- [Supplementary Materials, Implementation Details] The lasso size range is given as U(4, 64) while the main text defines the pre-defined lasso size as (P × r)^2; clarify the units and relationship between these two formulations.
- [Experiments, Evaluation on colorization benchmarks] The phrase 'systemic quantitative comparison' should be 'systematic quantitative comparison'.
Circularity Check
No significant circularity: the lasso-effectiveness claims rest on measured user studies and independent baselines.
full rationale
No circularity found. The paper's central claims — that lassos localize color propagation, that one lasso is as effective as 4.18 points, and that users finish 30% faster — are supported by measured user studies and benchmark comparisons against external baselines (iColoriT, UniColor, Zhang et al., Su et al., Yin et al.), not by derivation from the model's own fitted parameters. The 4.18 ratio is computed from observed interaction counts (20.9 points versus 5.69 points plus 3.64 lassos) and is therefore an empirical equivalence, not a constructed identity. The training-time lasso simulation is a stated assumption about generalization from rectangles to freehand lassos; it is a weakness, not a circular reduction. Self-citations to Yun et al. (2023) supply the iColoriT baseline, pixel-shuffling upsampling, and the HPR metric; these are implementation and comparison details, and the lasso result does not rest on the truth of that cited work. The equation-level concern that Eq. (1) multiplies logits by a 0/1 mask before softmax, so masked positions still receive nonzero probability, is a correctness/consistency issue that would need implementation clarification; it is not an instance of a result reducing to its inputs by construction.
Assumptions & free parameters
free parameters (3)
- pre-defined lasso size scaling factor r =
1 (from {0.25, 0.5, 1, 4, 16, 64})
- lasso training size range =
U(4,64)
- number of hints h distribution =
U(0,150)
assumptions (3)
- domain assumption The ground truth color for each simulated hint is always enclosed within its lasso region during training.
- domain assumption Cross-attention masking alone is sufficient to control color propagation; no explicit boundary or regularization loss is needed.
- ad hoc to paper Simulated rectangular lassos transfer to freehand user lassos.
Cite this review
Pith. "Pith review of Enabling Region-Specific Control via Lassos in Point-Based Colorization." pith.science (2026). https://pith.science/paper/MF2MONW6
@misc{pith2026241213469,
author = {Pith},
title = {Pith review of: Enabling Region-Specific Control via Lassos in Point-Based Colorization},
year = {2026},
howpublished = {\url{https://pith.science/paper/MF2MONW6}},
note = {Machine review of arXiv:2412.13469}
}
read the original abstract
Point-based interactive colorization techniques allow users to effortlessly colorize grayscale images using user-provided color hints. However, point-based methods often face challenges when different colors are given to semantically similar areas, leading to color intermingling and unsatisfactory results-an issue we refer to as color collapse. The fundamental cause of color collapse is the inadequacy of points for defining the boundaries for each color. To mitigate color collapse, we introduce a lasso tool that can control the scope of each color hint. Additionally, we design a framework that leverages the user-provided lassos to localize the attention masks. The experimental results show that using a single lasso is as effective as applying 4.18 individual color hints and can achieve the desired outcomes in 30% less time than using points alone.
Figures
Figures from the paper (16 more)
Reference graph
Works this paper leans on
-
[1]
K.; Wu, Z.; Ma, X.; and Choo, J
Bahng, H.; Yoo, S.; Cho, W.; Park, D. K.; Wu, Z.; Ma, X.; and Choo, J. 2018. Coloring with words: Guiding image colorization through text-based palette generation. In Proceedings of the european conference on computer vision (eccv), 431--447
work page 2018
-
[2]
Bai, Y.; Dong, C.; Chai, Z.; Wang, A.; Xu, Z.; and Yuan, C. 2022. Semantic-sparse colorization network for deep exemplar-based colorization. In European Conference on Computer Vision, 505--521. Springer
work page 2022
-
[3]
Chang, Z.; Weng, S.; Li, Y.; Li, S.; and Shi, B. 2022. L-CoDer: Language-based colorization with color-object decoupling transformer. In European Conference on Computer Vision, 360--375. Springer
work page 2022
-
[4]
Chang, Z.; Weng, S.; Zhang, P.; Li, Y.; Li, S.; and Shi, B. 2023. L-CoIns: Language-Based Colorization With Instance Awareness. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 19221--19230
work page 2023
-
[5]
Chen, J.; Shen, Y.; Gao, J.; Liu, J.; and Liu, X. 2018. Language-based image editing with recurrent attentive models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 8721--8729
work page 2018
-
[6]
Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. 2020. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In ICLR
2020
-
[7]
He, M.; Chen, D.; Liao, J.; Sander, P. V.; and Yuan, L. 2018. Deep exemplar-based colorization. ACM Transactions on Graphics (TOG), 37(4): 1--16
work page 2018
-
[8]
Hertz, A.; Mokady, R.; Tenenbaum, J.; Aberman, K.; Pritch, Y.; and Cohen-Or, D. 2022. Prompt-to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.01626
arXiv 2022
Show all 40 references
-
[9]
Huang, Z.; Zhao, N.; and Liao, J. 2022. Unicolor: A unified framework for multi-modal colorization with transformer. ACM Transactions on Graphics (TOG), 41(6): 1--16
2022
-
[10]
Huber, P. J. 1992. Robust estimation of a location parameter. In Breakthroughs in statistics, 492--518. Springer
1992
-
[11]
Kim, E.; Lee, S.; Park, J.; Choi, S.; Seo, C.; and Choo, J. 2021. Deep Edge-Aware Interactive Colorization against Color-Bleeding Effects. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 14667--14676
2021
-
[12]
Larsson, G.; Maire, M.; and Shakhnarovich, G. 2016. Learning Representations for Automatic Colorization. In European Conference on Computer Vision (ECCV)
2016
-
[13]
Levin, A.; Lischinski, D.; and Weiss, Y. 2004. Colorization Using Optimization. ACM Transactions on Graphics, 23: 689–694
2004
-
[14]
Li, B.; Lai, Y.-K.; John, M.; and Rosin, P. L. 2019. Automatic example-based image colorization using location-aware cross-scale matching. IEEE Transactions on Image Processing, 28(9): 4606--4619
2019
-
[15]
Li, H.; Sheng, B.; Li, P.; Ali, R.; and Chen, C. P. 2021. Globally and Locally Semantic Colorization via Exemplar-Based Broad-GAN. IEEE Transactions on Image Processing, 30: 8526--8539
2021
-
[16]
Liang, Z.; Li, Z.; Zhou, S.; Li, C.; and Loy, C. C. 2024. Control Color: Multimodal Diffusion-based Interactive Image Colorization. arXiv preprint arXiv:2402.10855
2024 arXiv
-
[17]
Lu, P.; Yu, J.; Peng, X.; Zhao, Z.; and Wang, X. 2020. Gray2colornet: Transfer more colors from reference image. In Proceedings of the 28th ACM International Conference on Multimedia, 3210--3218
2020
-
[18]
Manjunatha, V.; Iyyer, M.; Boyd-Graber, J.; and Davis, L. 2018. Learning to color from language. arXiv preprint arXiv:1804.06026
2018 arXiv
-
[19]
Nilsback, M.-E.; and Zisserman, A. 2008. Automated Flower Classification over a Large Number of Classes. In Indian Conference on Computer Vision, Graphics and Image Processing
2008
-
[20]
H.; Luo, G.; Toste, C.; Azadi, S.; Liu, X.; Karalashvili, M.; Rohrbach, A.; and Darrell, T
Park, D. H.; Luo, G.; Toste, C.; Azadi, S.; Liu, X.; Karalashvili, M.; Rohrbach, A.; and Darrell, T. 2022. Shape-Guided Diffusion with Inside-Outside Attention. arXiv preprint arXiv:2212.00210
2022 arXiv
-
[21]
Rombach, R.; Blattmann, A.; Lorenz, D.; Esser, P.; and Ommer, B. 2022. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 10684--10695
2022
-
[22]
C.; and Fei-Fei, L
Russakovsky, O.; Deng, J.; Su, H.; Krause, J.; Satheesh, S.; Ma, S.; Huang, Z.; Karpathy, A.; Khosla, A.; Bernstein, M.; Berg, A. C.; and Fei-Fei, L. 2015. ImageNet Large Scale Visual Recognition Challenge . International Journal of Computer Vision (IJCV), 115(3): 211--252
2015
-
[23]
Su, J.-W.; Chu, H.-K.; and Huang, J.-B. 2020. Instance-aware image colorization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 7968--7977
2020
-
[24]
N.; Kaiser, .; and Polosukhin, I
Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, .; and Polosukhin, I. 2017. Attention is all you need. Advances in neural information processing systems, 30
2017
-
[25]
Wang, Y.; Xia, M.; Qi, L.; Shao, J.; and Qiao, Y. 2022. PalGAN: Image colorization with palette generative adversarial networks. In European Conference on Computer Vision, 271--288. Springer
2022
-
[26]
Welinder, P.; Branson, S.; Mita, T.; Wah, C.; Schroff, F.; Belongie, S.; and Perona, P. 2010. Caltech-UCSD Birds 200. Technical Report CNS-TR-2010-001, California Institute of Technology
2010
-
[27]
Weng, S.; Sun, J.; Li, Y.; Li, S.; and Shi, B. 2022 a . CT 2: Colorization transformer via color tokens. In European Conference on Computer Vision, 1--16. Springer
2022
-
[28]
Weng, S.; Wu, H.; Chang, Z.; Tang, J.; Li, S.; and Shi, B. 2022 b . L-code: Language-based colorization using color-object decoupled conditions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, 2677--2684
2022
-
[29]
Wu, S.; Yang, Y.; Xu, S.; Liu, W.; Yan, X.; and Zhang, S. 2023. FlexIcon: Flexible Icon Colorization via Guided Images and Palettes. In Proceedings of the 31st ACM International Conference on Multimedia, 8662--8673
2023
-
[30]
Xiao, C.; Han, C.; Zhang, Z.; Qin, J.; Wong, T.-T.; Han, G.; and He, S. 2020. Example-Based Colourization Via Dense Encoding Pyramids. In Computer Graphics Forum, volume 39, 20--33. Wiley Online Library
2020
-
[31]
T.; Durand, F.; and Han, S
Xiao, G.; Yin, T.; Freeman, W. T.; Durand, F.; and Han, S. 2024. Fastcomposer: Tuning-free multi-subject image generation with localized attention. International Journal of Computer Vision, 1--20
2024
-
[32]
Xu, Z.; Wang, T.; Fang, F.; Sheng, Y.; and Zhang, G. 2020. Stylization-based architecture for fast deep exemplar colorization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 9363--9372
2020
-
[33]
Yin, H.; Gong, Y.; and Qiu, G. 2019. Side window filtering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 8758--8766
2019
-
[34]
Attention Is All You Need
Yin, W.; Lu, P.; Zhao, Z.; and Peng, X. 2021. Yes," Attention Is All You Need", for Exemplar based Colorization. In Proceedings of the 29th ACM International Conference on Multimedia, 2243--2251
2021
-
[35]
Yun, J.; Lee, S.; Park, M.; and Choo, J. 2023. iColoriT: Towards Propagating Local Hints to the Right Region in Interactive Colorization by Leveraging Vision Transformer. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 1787--1796
2023
-
[36]
V.; Yuan, L.; Bermak, A.; and Chen, D
Zhang, B.; He, M.; Liao, J.; Sander, P. V.; Yuan, L.; Bermak, A.; and Chen, D. 2019. Deep exemplar-based video colorization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 8052--8061
2019
-
[37]
Zhang, R.; Isola, P.; and Efros, A. A. 2016. Colorful image colorization. In European conference on computer vision, 649--666. Springer
2016
-
[38]
S.; Yu, T.; and Efros, A
Zhang, R.; Zhu, J.-Y.; Isola, P.; Geng, X.; Lin, A. S.; Yu, T.; and Efros, A. A. 2017. Real-time user-guided image colorization with learned deep priors. ACM TOG
2017
-
[39]
, " * write output.state after.block = add.period write newline
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...
-
[40]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.