REVIEW 3 major objections 4 minor 6 references
Instance Segmentation of Scene Sketches Using Natural Image Priors
T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read InkLayer adapts natural-image detection and segmentation models—via class-agnostic fine-tuning, mask-based overlap suppression, and depth-guided refinement—to perform instance segmentation of scene sketches, reporting an average recall of…
desk verdict Useful benchmark and a promising method, but the headline gains on synthetic styles rest on unmeasured ground-truth alignment—worth a careful revision, not a desk reject. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is class-agnostic fine-tuning combined with depth-grounded overlap resolution. Class-agnostic fine-tuning replaces the usual category recognition loss with a single 'object' label and trains only the box-prediction head, forcing the model to separate instances by visual grouping rather than by recognizing classes; this is what lifts detection AP from 26% to 74% on SketchyScene sketches. Then SAM extracts masks from the detected boxes, and the refinement stage computes a mask-intersection-over-union overlap score between candidate boxes, keeps the highest-confidence detection for redundant pairs, samples the depth map from DepthAnythingV2 along sketch pixels to assign each mask a modal depth score, and gives overlapping pixels to the front-most object; a watershed step propagates labels to unlabeled sketch pixels. Finally, a pretrained SDXL inpainting model fills occluded regions so each instance becomes a complete layer.
What would settle it
Compute, for each generated sketch in InkScenes, the overlap between the rasterized generated object and its inherited ground-truth box/mask (e.g., mask IoU). If a large share of CLIPasso and SketchAgent samples show low alignment while the reported segmentation metrics stay high, then the benchmark's labels are partly easy targets and the gains over baselines would shrink under fresh, human-verified annotations.
Extended reading notes
Core claim
The paper's central discovery is that the domain gap between natural images and sketches can be largely closed without sketch-specific architecture changes. The authors fine-tune Grounding DINO's detection head on 5,000 SketchyScene sketches, collapsing all class labels into a single 'object' label and dropping the class recognition loss; this pushes the detector to rely on Gestalt grouping cues rather than category knowledge. Detected boxes feed SAM to produce masks, and the paper shows that mask-based overlap suppression plus depth-based assignment of ambiguous pixels from DepthAnythingV2, followed by watershed propagation, refines segmentation substantially. On the InkScenes benchmark the method claims consistent performance across all datasets, with average AR 0.79 and average AP 0.74, and it outperforms prior sketch-specific methods even on the SketchyScene dataset that those methods were built for.
Load-bearing premise
The evaluation assumes that the ground-truth boxes and masks inherited from SketchyScene and Visual Genome layouts stay correct after each new sketch style is generated, even though CLIPasso, SketchAgent, and InstantStyle objects may drift from their assigned positions or vanish when small.
Editorial extensions
If this is right
- Sketch editing becomes practical without manual layer separation: users can move, scale, or delete segmented objects, with occluded regions inpainted automatically.
- The method generalizes to sketch styles and object categories it was never trained on, including 53 novel categories added via InstantStyle and human-drawn sketches from other datasets.
- Class-agnostic fine-tuning with a single 'object' label is sufficient to transfer open-vocabulary detection from natural images to sketches, reducing the need for large sketch-specific annotated datasets.
- The released InkScenes dataset (20,542 annotated scene sketches) gives the community a common benchmark with explicit style and category variation for future sketch segmentation work.
Reading between the lines
- The reported gains on the CLIPasso and SketchAgent styles may partly reflect easy labels: generated objects are placed at ground-truth layout positions, and the paper never measures how far generated strokes drift from those annotations, so a style with larger drift could lower real-world performance.
- Because the depth-ordering step only needs relative object order, not accurate metric depth, the approach could transfer to other sparse-image tasks such as vector art or technical drawings, where depth priors are noisy but ordering is still recoverable.
- A natural testable extension is to replace the synthetic depth proxy with human-annotated occlusion order on a subset of InkScenes; if ordering agreement is high, depth-based refinement is validated beyond the paper's Kendall's Tau proxy on InstantStyle.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper addresses instance segmentation of raster scene sketches. The proposed pipeline, InkLayer, fine-tunes Grounding DINO in a class-agnostic manner on a subset of SketchyScene, uses SAM to produce masks from detected boxes, resolves overlapping regions using depth maps from DepthAnythingV2, and finally creates inpainted ordered layers for editing. To evaluate across styles, the authors construct the InkScenes benchmark by re-rendering SketchyScene layouts with CLIPasso and SketchAgent object sketches and three brush styles, and by transferring Visual Genome images into sketches with InstantStyle. On this benchmark they report large gains over unadapted Grounding DINO (absolute gains of 0.38 IoU, 0.48 AR, and 0.48 AP on average) and better segmentation accuracy than several baselines, including on the human-drawn Zhang et al. dataset. Code and data are released.
Significance. The paper's contribution is mainly empirical and engineering-oriented. If the benchmark labels are valid, the reported result is a strong practical baseline for sketch instance segmentation and sketch editing, and the class-agnostic fine-tuning recipe plus depth-based mask refinement is simple and reproducible. The release of code and data is a clear strength, as is the breadth of qualitative evaluation. The main caveat is that the central quantitative claims rest on synthetic benchmark labels whose alignment with the generated strokes is not verified; until that is checked, the size of the reported gains should be treated as an upper bound. With that validation, the work would be a solid contribution to sketch understanding and editing.
major comments (3)
- [Section 4, Tables 1-3] The ground-truth masks for the CLIPasso, SketchAgent, and InstantStyle subsets are inherited from SketchyScene layout annotations or Visual Genome segmentations after placing and scaling generated object sketches into layout boxes. The paper states this placement procedure but never measures whether the rendered strokes actually align with the inherited mask. CLIPasso sketches often contain empty margins and SketchAgent sketches are highly abstract, so a predictor that outputs the layout region can score well on Acc/IoU even when it does not segment the strokes that are present. Because the headline 0.38/0.48/0.48 gains are computed on this benchmark, I ask for a direct alignment check (e.g., stroke-to-mask IoU, fraction of instances with visible strokes exceeding a threshold) and re-reporting of metrics on instances that pass the check.
- [Section 4, Section 3.4] The paper does not state whether the CLIPasso and SketchAgent evaluation scenes are constructed from the same SketchyScene layouts used for fine-tuning. If they are, the detector could exploit layout-level regularities rather than generalize to new compositions. Please clarify the split provenance for each of the eight evaluation sets, and if necessary re-run the comparisons on layouts that are disjoint from the fine-tuning split.
- [Section 5.1, Table 1] The claim of "consistent performance across all datasets" is not fully supported by the row-level numbers: on InstantStyle, the method's IoU (0.45) is below unadapted Grounding DINO (0.49), and the reported standard deviations across datasets (e.g., ±0.12 for AP) are large relative to some between-method gaps. Please report per-dataset error bars across multiple runs or a significance test, and soften or qualify the consistency claim accordingly.
minor comments (4)
- [Section 5.1] The phrase "an increase of 38% in IoU, 48% in AR, and 48% in AP" should be stated as absolute percentage-point increases (0.38, 0.48, 0.48) to avoid confusion with relative improvements.
- [Section 3.2] The duplicate-box merge threshold O(i,j) > 0.5 and the detection confidence threshold are introduced without a sensitivity analysis; a short ablation would help the reader understand their influence on the reported results.
- [Section 4] The filtering criterion for InstantStyle is described only as "five to ten distinct objects"; please also report how many Visual Genome images were discarded by this filter and whether any per-instance visibility filter was applied before inheriting the segmentation masks.
- [Tables 1-3] The table headers are confusing because the name "SketchyScene" is used both for a dataset and for one of the baseline methods; please use distinct labels such as "SketchyScene baseline" or "Zou et al.".
Circularity Check
No significant circularity: the headline gains are empirical test-set measurements, and the central robustness claim is supported by external human-drawn benchmarks independent of the self-cited sketch-generation tools.
full rationale
The paper's contribution is an empirical pipeline (class-agnostic fine-tuning of Grounding DINO, SAM mask extraction, depth-guided refinement, and inpainting), not an analytic derivation in which an output is defined in terms of an input. The central quantitative claim (38% IoU, 48% AR, 48% AP gains over unadapted Grounding DINO) is a held-out test-set measurement, and the model is fine-tuned on only 5,000 SketchyScene training samples while evaluated on held-out SketchyScene test data, human-drawn Zhang et al. sketches, FSCOCO-Seg, and OpenSketch. The robustness claim therefore has independent content beyond the training distribution. The use of CLIPasso and SketchAgent as dataset-generation tools involves a coauthor citation, but it is not load-bearing: the same qualitative and quantitative robustness appears on human-drawn external benchmarks, and no parameter is fitted to the generated CLIPasso/SketchAgent labels. The concern that generated stroke placements may drift from inherited SketchyScene/Visual Genome ground-truth boxes is a benchmark-validity issue, not a circularity of the kind where a reported result reduces by construction to a fitted input or to a self-citation chain. No uniqueness theorem is imported, no ansatz is smuggled in via citation, and no known result is merely renamed. Thus the derivation chain is self-contained for the purpose of circularity analysis.
Assumptions & free parameters
free parameters (6)
- Duplicate box merge threshold O(i,j) =
0.5
- Detection confidence threshold =
not stated
- Depth sample point layout =
not stated
- Brush stroke widths for InkScenes =
manually selected per style
- Fine-tuning subset size =
5,000 sketches
- InstantStyle scene filter =
5 to 10 objects
assumptions (4)
- domain assumption Pretrained natural-image models (Grounding DINO, SAM, DepthAnythingV2, SDXL inpainting) provide reliable priors on sketches.
- domain assumption Generated vector sketches preserve the instance identity and layout position of their source annotations.
- domain assumption DepthAnythingV2 depth order on sparse sketches is accurate enough to resolve occlusion.
- ad hoc to paper Consolidating all class labels into 'object' preserves the detector's ability to group instances by Gestalt cues.
Cite this review
Pith. "Pith review of Instance Segmentation of Scene Sketches Using Natural Image Priors." pith.science (2026). https://pith.science/paper/TTVJSXMS
@misc{pith2026250209608,
author = {Pith},
title = {Pith review of: Instance Segmentation of Scene Sketches Using Natural Image Priors},
year = {2026},
howpublished = {\url{https://pith.science/paper/TTVJSXMS}},
note = {Machine review of arXiv:2502.09608}
}
read the original abstract
Sketch segmentation involves grouping pixels within a sketch that belong to the same object or instance. It serves as a valuable tool for sketch editing tasks, such as moving, scaling, or removing specific components. While image segmentation models have demonstrated remarkable capabilities in recent years, sketches present unique challenges for these models due to their sparse nature and wide variation in styles. We introduce InkLayer, a method for instance segmentation of raster scene sketches. Our approach adapts state-of-the-art image segmentation and object detection models to the sketch domain by employing class-agnostic fine-tuning and refining segmentation masks using depth cues. Furthermore, our method organizes sketches into sorted layers, where occluded instances are inpainted, enabling advanced sketch editing applications. As existing datasets in this domain lack variation in sketch styles, we construct a synthetic scene sketch segmentation dataset, InkScenes, featuring sketches with diverse brush strokes and varying levels of detail. We use this dataset to demonstrate the robustness of our approach.
Figures
Figures from the paper (28 more)
Reference graph
Works this paper leans on
-
[6]
A realistic image of a {class_name} with a blank background
ENDE-GNN: An Encoder-decoder GNN Framework for Sketch Semantic 9 SIGGRAPH Conference Papers ’25, August 10–14, 2025, Vancouver, BC, Canada Mia Tang, Yael Vinker, Chuan Yan, Lvmin Zhang, and Maneesh Agrawala Segmentation. In 2022 IEEE International Conference on Visual Communications and Image Processing (VCIP). 1–5. https://doi.org/10.1109/VCIP56404.2022....
-
[2018]
construct the SketchyScene dataset, providing annotated scene sketches with meaningful layouts of object interactions, and use it to train an instance segmentation model based on the Mask R-CNN architecture [He et al. 2018]. However, their method is lim- ited to the predefined categories included in the dataset, and the proposed dataset contains sketches ...
work page 2018
-
[2020]
Neurocomputing 392 (2020), 50–59
Multi-column point-CNN for sketch segmentation. Neurocomputing 392 (2020), 50–59. https://doi.org/10.1016/j.neucom.2019.12.117 Haofan Wang, Matteo Spinelli, Qixun Wang, Xu Bai, Zekui Qin, and Anthony Chen. 2024. InstantStyle: Free Lunch towards Style-Preserving in Text-to-Image Generation. ArXiv abs/2404.02733 (2024). https://api.semanticscholar.org/Corpu...
arXiv 2020
-
[2022]
Simple Open-Vocabulary Object Detection with Vision Transformers. arXiv:2205.06230 [cs.CV] https://arxiv.org/abs/2205.06230 Anran Qi, Yulia Gryaditskaya, Tao Xiang, and Yi-Zhe Song. 2021. One Sketch for All: One-Shot Personalized Sketch Segmentation. CoRR abs/2112.10838 (2021). arXiv:2112.10838 https://arxiv.org/abs/2112.10838 Yonggang Qi, Yi-Zhe Song, Ta...
arXiv 2021
-
[2023]
are shown separately in Table 3 since it performs semantic segmentation and requires dataset filtering. As shown, our method outperforms alternative approaches across both metrics, with a par- ticularly notable advantage over Grounded SAM. While SketchSeger performs well on SketchyScene dataset, its performance degrades significantly on the symbolic style...
arXiv 2012
-
[2025]
Instance Segmentation of Scene Sketches Using Natural Image Priors. In Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Papers (SIGGRAPH Conference Papers ’25), August 10–14, 2025, Vancouver, BC, Canada. ACM, New York, NY, USA, 29 pages. https://doi.org/10.1145/3721238.3730606 1 INTRODUCTION Sketches serve as a ...
arXiv 2025
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.