IAM: Enhancing RGB-D Instance Segmentation with New Benchmarks

Aecheon Jung; junhong Min; Soyun Choi; Sungeun Hong

REVIEW 4 major objections 5 minor 56 references

IAM: Enhancing RGB-D Instance Segmentation with New Benchmarks

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A simple intra-modal attention mixing module, evaluated on three new RGB-D instance segmentation benchmarks, consistently outperforms prior fusion methods such as early fusion, late fusion, and attention-based approaches.

desk verdict The new benchmarks are the real contribution, but unvalidated mask conversion and one contradictory table undercut the central claims; worth a serious referee, not a desk reject. read the letter →

arxiv 2501.01685 v1 pith:6SEGHER5 submitted 2025-01-03 cs.CV

Aecheon Jung , Soyun Choi , Junhong Min , Sungeun Hong This is my paper · ORCID

classification cs.CV

keywords RGB-Dinstancesegmentationbenchmarkdatasetmultimodalfusionintra-modalattentionmechanismdepthsensingindoorsceneunderstandingDETR

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces three RGB-D instance segmentation benchmarks — NYUDv2-IS, SUN-RGBD-IS, and Box-IS — and a lightweight fusion module called Intra-modal Attention Mix (IAM) for combining RGB and depth features. The authors claim that IAM consistently outperforms existing fusion methods, including early fusion, late fusion, and attention-based approaches, with gains such as 6.7% average-precision improvement over early fusion on NYUDv2-IS. If the benchmarks are valid, they provide reusable evaluation resources for a field that previously lacked real-world RGB-D instance segmentation datasets. If the module performs as reported, it offers a simple and efficient way to exploit depth for instance segmentation in indoor scenes.

What carries the argument

The central mechanism is the Intra-modal Attention Mix (IAM) block, which works on concatenated RGB and depth feature maps written as block matrices $Q = [Q_{rgb}, Q_d]$ and $K = [K_{rgb}, K_d]$. The attention score $QK^T$ then splits exactly into $Q_{rgb}K_{rgb}^T + Q_d K_d^T$, so the model learns spatial relationships inside each modality separately, with no direct cross-modal attention. A mixup-inspired summation blends these intra-modal maps, a channel-wise dynamic fusion module recalibrates the modalities per channel, and the block outputs enhanced RGB features, enhanced depth features, and an aggregated fused feature that feed the detection and segmentation heads of the DETR or SOLQ backbone.

What would settle it

Take a random sample of images from NYUDv2-IS and SUN-RGBD-IS, have annotators draw instance masks by hand, and compute agreement with the connected-component-derived masks. If a substantial fraction of instances (for example more than 10%) differ in whether they are merged or split, the benchmarks would not measure what they claim, and model rankings on them could be reversed.

Watch

Extended reading notes

Core claim

The paper's central claim is that RGB-D instance segmentation is held back by a lack of real-world instance-level datasets, and that both this gap and the challenge of fusing RGB and depth features can be addressed by two contributions: three new benchmarks (NYUDv2-IS, SUN-RGBD-IS, Box-IS) and the Intra-modal Attention Mix (IAM) module. IAM computes self-attention within each modality by decomposing the query-key product into $Q_{rgb}K_{rgb}^T + Q_dK_d^T$, then recalibrates channel importance with a dynamic fusion step. Across DETR and SOLQ backbones and all three datasets, the paper reports that IAM achieves the highest AP among compared fusion methods, including SA-Gate and CMX, while being computationally lighter because it avoids cross-modal attention terms.

Load-bearing premise

The ground-truth instance masks for NYUDv2-IS and SUN-RGBD-IS are generated by extracting connected components from semantic segmentation labels, with no human verification, so touching objects of the same category or occluded objects may be merged or split incorrectly, making the benchmarks unreliable measures of true instance segmentation.

Editorial extensions

If this is right

Standardized evaluation becomes possible: future RGB-D instance segmentation work can report numbers on NYUDv2-IS, SUN-RGBD-IS, and Box-IS, which cover real indoor scenes and multiple depth sensors.
IAM can be inserted into existing two-stream detectors as a drop-in block, giving practitioners a cheaper alternative to inter-modal attention that still improves accuracy on small objects and boundaries.
If the gains replicate, depth information becomes more useful for segmentation under low light and texture-ambiguous conditions, where RGB-only features are noisy.
The Box-IS dataset gives the robotics community a dedicated single-class benchmark for box depalletizing, with depth computed from stereo IR images rather than consumer depth post-processing.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the NYUDv2-IS and SUN-RGBD-IS masks are derived from connected components of semantic labels, a natural next step is to validate a random subset against human-annotated instances; the paper reports no such verification, and the rankings on these benchmarks could shift if merged or split objects are common.
The same intra-modal decomposition could be applied to other aligned modality pairs such as RGB-Thermal or RGB-surface normals, since the block matrices do not rely on RGB-D-specific assumptions; this is a direct testable extension the paper does not mention.
The baselines SA-Gate and CMX are evaluated with default hyperparameters; a stronger comparison would tune them per dataset, which could reduce the reported margins.
The paper does not report whether the two backbones share weights or train separately; testing weight sharing versus independence would clarify where IAM's benefit comes from.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a useful dataset paper wrapped around a modest method. The three RGB-D instance segmentation benchmarks are new artifacts, and Box-IS fills a real gap for robotic box depalletizing. The GitHub releases make the resources usable. The IAM module is a minor variant of existing attention fusion, with a mixup-inspired intra-modal term plus a channel-wise gate; nothing wrong with that, but it is not a conceptual leap.

The soft spots are where the reader's stress-test lands. First, the conversion of NYUDv2 and SUN-RGBD semantic labels into instance masks is described in one sentence ('extracting binary instance masks') with no detail on how same-class touching objects are separated, no human verification, no inter-annotator agreement, and no error bars. That is load-bearing because every AP number in the paper is computed against those masks. The intro says 're-annotating,' but the methods section does not substantiate it. Second, Table 4 directly contradicts the 'consistent improvements over existing fusion methods' claim: DETR on SUN-RGBD-IS gives Ours 22.9 APseg versus Intra 23.6 and Inter 23.4. The text's hand-wave about DETR's architecture does not fix the inconsistency. On Box-IS, the SOLQ gain over late fusion is 0.5 AP, and over RGB it is 0.4, so the 'notable enhancements' are sometimes marginal.

I would push back mildly on the reader's circularity score: there is no circularity loop here, just a data provenance issue, and the method's parameters are learned from the training split. That worry is overblown.

Who this is for: people building RGB-D instance segmentation baselines and robotics manipulation researchers will get value from the released benchmarks, if the masks hold up. The method section is a reasonable baseline for future fusion work. It deserves a serious referee, but the authors should be pushed to (a) document the mask-generation pipeline precisely, ideally with human verification or at least a random-sample audit, (b) fix the 'consistently outperforms' claim or the table, and (c) release code and error bars. If those are addressed, it is a publishable contribution; as is, the central claims are not fully supported.

Referee Report

4 major / 5 minor

Summary. The manuscript introduces three RGB-D instance segmentation benchmarks (NYUDv2-IS, SUN-RGBD-IS, and Box-IS) and a fusion module, Intra-modal Attention Mix (IAM), inserted into two-stream DETR and SOLQ architectures. The first two benchmarks are derived from existing semantic-segmentation datasets via COCO-format conversion, while the third is newly captured with an Intel RealSense D455. The paper reports instance-segmentation AP comparisons against early/late fusion, intra/inter-modal attention, SA-Gate, and CMX, together with ablations on block placement, feature-integration design, and depth-sensor quality. The two central claims are that the derived datasets are valid instance-level benchmark resources and that IAM consistently outperforms existing RGB-D fusion methods.

Significance. If the benchmark masks are valid, the datasets would fill a genuine gap: public real-world RGB-D instance segmentation benchmarks are scarce, and the paper provides statistics, splits, and public links for all three datasets. The method side is modest but useful: IAM is a light-weight module that improves SOLQ in most reported settings, and the paper includes multiple baselines and ablations. However, the significance of the benchmark contribution is conditional on the unverified instance-mask construction described in Section 3.1, and the 'consistent improvement' claim is contradicted by Table 4. The absence of any human verification or inter-annotator agreement for the derived instance masks is the main correctness risk for the dataset half of the paper.

major comments (4)

[Section 3.1] The benchmark construction does not establish that the derived masks are true object instances. The text states only that instance masks were generated from the original object-class annotations by 'extracting binary instance masks' and converting them to COCO-style polygons; there is no step that separates distinct touching objects of the same class, no handling of occluders that split a single object, and no human verification or inter-annotator agreement. Since every AP number in Tables 3-5 is computed against these masks, the reusable-benchmark contribution and all derived comparisons rest on unvalidated ground truth. The Introduction's claim of 're-annotating' (Section 1) is not substantiated by the described pipeline. The authors should provide a validation study, per-image instance-count comparisons, or use existing genuinely instance-level annotations before the benchmarks can be accepted as instance segmentation resources.
[Section 5.3, Table 4] The claim that 'models using our IAM module consistently outperformed' intra- and inter-modal attention is contradicted by the DETR rows on SUN-RGBD-IS: Ours achieves APseg = 22.9, while Intra achieves 23.6 and Inter achieves 23.4; Ours is also lower in APdet (29.3 vs. 30.2 and 30.1). The paper must either scope the consistency claim to SOLQ or to the datasets where it holds, and it should analyze the negative DETR result instead of asserting universal superiority. This is load-bearing because the abstract, introduction, and Section 5.3 all advertise consistent gains over fusion baselines.
[Section 5.4.1 and 5.4.3, Tables 6 and 8] The ablation baseline is ambiguous and inconsistent with Table 3. Table 6 reports 'Backbone' APseg = 28.6, which matches the 'Early' fusion result in Table 3 (28.6), not the RGB-only SOLQ result (33.1). Table 8 similarly uses SOLQ = 28.6 as the starting point. The 7.2% gain attributed to IAM+CDF is therefore a gain over early fusion, not over the RGB-only or late-fusion baselines that the text elsewhere compares against. The ablation protocol should be aligned with the comparison protocol in Table 3, or the text should explicitly define which baseline is being augmented.
[Section 4.2.1, Eq. (7)] The claimed 'mixup-inspired' mechanism is not reflected in the algebra. Expanding the block matrices yields QK^T = Qrgb Krgb^T + Qd Kd^T, which is a block-diagonal attention with no cross-modal terms; this is equivalent to computing attention within each modality separately and summing the results, not to mixing or blending features across modalities. If the intended operation is exactly this per-modality attention sum, the mixup analogy should be removed or substantially qualified, because the current description overstates the conceptual novelty of the module.

minor comments (5)

[Section 1, contribution bullet] The phrase 'we construct two public RGB-D instance segmentation dataset' should be 'datasets', and the sentence should be reworded for grammatical consistency.
[Section 3.1] The paper provides public repository links, but it does not state the licenses, file formats, or evaluation scripts included in those repositories; adding this information would improve reproducibility.
[Section 4.2.2, Eqs. (10)-(12)] The shape and broadcasting of Wn are not specified: after global average pooling and sigmoid, Wn appears to be a per-channel vector, but the text should state its dimensionality and how it is applied to Frgb and Fd in Eqs. (11) and (12).
[Section 5.3, Table 5] Box-IS results are reported only for SOLQ, with no DETR table or a clear statement that DETR was omitted due to the difficulty described in the text; please include the DETR numbers or an explicit note on their absence.
[References] Reference [6] lists the venue as 'Winter Conf. on Application of Computer Vision (ECCV)'; the correct venue is WACV, and the citation should be corrected.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: IAM is an empirical fusion module evaluated on held-out splits; benchmark provenance is a data-validity issue, not a definitional loop.

full rationale

The paper's central claim is that the IAM module improves RGB-D instance segmentation over baselines. The module is defined by Equations 1-13 as a network component with learnable parameters, and its performance is measured on held-out validation splits against early fusion, late fusion, intra-modal attention, inter-modal attention, SA-Gate, and CMX. No fitted parameter is renamed as a prediction, and no equation defines the reported AP numbers in terms of the module's own construction. The benchmarks NYUDv2-IS and SUN-RGBD-IS are built by converting existing semantic segmentation labels into instance masks (Section 3.1), but this is a data provenance and annotation-validity issue, not a circularity loop: the model's AP is computed against those masks rather than derived from them by construction. The paper does not rely on a self-citation chain or an author-imported uniqueness theorem; references such as [13], [14], [15], [30], and [31] are external dataset and architecture sources. The apparent contradiction in Table 4, where DETR on SUN-RGBD-IS gives Ours 22.9 AP^seg versus Intra 23.6 and Inter 23.4, undermines the 'consistently outperforms' claim as a correctness matter, not a circularity matter. Likewise, the unvalidated connected-component assumption about instance masks is a benchmark-validity concern. No load-bearing step reduces by definition to its own inputs, so the appropriate circularity finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central empirical claims rest on the assumption that automatically generated connected-component masks are valid instance annotations, plus the usual transferability of DETR and SOLQ training protocols. There are no fitted scalar free parameters in the ledger sense; the method's weights are learned from training data.

assumptions (3)

domain assumption Connected components of semantic segmentation labels correspond to distinct object instances.
Section 3.1 constructs instance masks by extracting binary masks from NYUDv2 and SUN-RGBD class labels; adjacent same-class objects are not separated, so instance boundaries may be wrong. This is load-bearing for the benchmark claims.
domain assumption Converting binary masks to polygon representations and COCO format preserves instance geometry.
The paper states that masks are converted into polygons and COCO-style annotations without reporting any verification of geometric fidelity.
domain assumption UniMatch stereo depth computed from Box-IS IR images is accurate enough for instance-level annotations.
Box-IS depth is derived by bypassing the sensor's default depth processing and applying UniMatch to IR stereo pairs; no depth accuracy or validation against the sensor output is reported.

how reviews work

0 comments

Cite this review

Pith. "Pith review of IAM: Enhancing RGB-D Instance Segmentation with New Benchmarks." pith.science (2026). https://pith.science/paper/6SEGHER5

@misc{pith2026250101685,
  author       = {Pith},
  title        = {Pith review of: IAM: Enhancing RGB-D Instance Segmentation with New Benchmarks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6SEGHER5}},
  note         = {Machine review of arXiv:2501.01685}
}

read the original abstract

Image segmentation is a vital task for providing human assistance and enhancing autonomy in our daily lives. In particular, RGB-D segmentation-leveraging both visual and depth cues-has attracted increasing attention as it promises richer scene understanding than RGB-only methods. However, most existing efforts have primarily focused on semantic segmentation and thus leave a critical gap. There is a relative scarcity of instance-level RGB-D segmentation datasets, which restricts current methods to broad category distinctions rather than fully capturing the fine-grained details required for recognizing individual objects. To bridge this gap, we introduce three RGB-D instance segmentation benchmarks, distinguished at the instance level. These datasets are versatile, supporting a wide range of applications from indoor navigation to robotic manipulation. In addition, we present an extensive evaluation of various baseline models on these benchmarks. This comprehensive analysis identifies both their strengths and shortcomings, guiding future work toward more robust, generalizable solutions. Finally, we propose a simple yet effective method for RGB-D data integration. Extensive evaluations affirm the effectiveness of our approach, offering a robust framework for advancing toward more nuanced scene understanding.

Figures

Figures reproduced from arXiv: 2501.01685 by the authors.

**Figure 2.** Examples of datasets: RGB image, depth map, and ground truth labels. [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

**Figure 3.** Examples from the constructed datasets. (a) Diversity of categories per image, [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Figures from the paper (9 more)

**Figure 4.** Figure 4: Cumulative distribution function showing the relative scale of objects. [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]

**Figure 5.** Figure 5: Scatter plots illustrating the relative proportions of bounding boxes for each [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]

**Figure 6.** Figure 6: Overview of baseline models with the proposed method for RGB-D instance segmentation. data only once, and intra-modal and inter-modal attention is limited to focusing exclusively on within or between modality relationships. This oversight may ultimately hinder the mod…

**Figure 7.** Figure 7: The details of the Intra-modal Attention Mix (IAM) architecture. IAM is [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]

**Figure 8.** Figure 8: Channel-wise dynamic fusion architecture. It dynamically refines feature inter [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]

**Figure 9.** Figure 9: Structure of the adaptable IAM module within dual-stream architectures for [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]

**Figure 10.** Figure 10: The network structure reconfigured to incorporate the proposed method into [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]

**Figure 11.** Figure 11: Comparison of DETR structures that fuse depth with RGB. [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]

**Figure 12.** Figure 12: A visual comparison of the early fusion, late fusion, intra-modal, inter-modal, [PITH_FULL_IMAGE:figures/full_fig_p023_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

56 extracted references · 54 canonical work pages

[1]

Tsutsui, M

K.-I. Tsutsui, M. Taira, H. Sakata, Neural mechanisms of three- dimensional vision, Neuroscience research 51 (3) (2005) 221–229

work page 2005
[2]

X. Qi, R. Liao, J. Jia, S. Fidler, R. Urtasun, 3d graph neural networks for rgbd semantic segmentation, in: Proc. of Int’l Conf. on Computer Vision (ICCV), 2017, pp. 5199–5208

work page 2017
[3]

Zhang, H

J. Zhang, H. Liu, K. Yang, X. Hu, R. Liu, R. Stiefelhagen, Cmx: Cross- modal fusion for rgb-x semantic segmentation with transformers, arXiv preprint arXiv:2203.04838 (2022)

arXiv 2022
[4]

Chen, K.-Y

X. Chen, K.-Y. Lin, J. Wang, W. Wu, C. Qian, H. Li, G. Zeng, Bi-directional cross-modality feature propagation with separation-and- aggregation gate for rgb-d semantic segmentation, in: Proc. of European Conf. on Computer Vision (ECCV), Springer, 2020, pp. 561–577

work page 2020
[5]

C. Tang, H. Chen, X. Li, J. Li, Z. Zhang, X. Hu, Look closer to segment better: Boundary patch refinement for instance segmentation, in: Proc. of Computer Vision and Pattern Recognition (CVPR), 2021, pp. 13926– 13935

work page 2021
[6]

M. Kim, S. Woo, D. Kim, I. S. Kweon, The devil is in the boundary: Ex- ploiting boundary representation for basis-based instance segmentation, in: Proc. of Winter Conf. on Application of Computer Vision (ECCV), 2021, pp. 929–938

work page 2021
[7]

W. Wang, U. Neumann, Depth-aware cnn for rgb-d segmentation, in: Proc. of European Conf. on Computer Vision (ECCV), 2018, pp. 135– 150

work page 2018
[8]

J. Cao, H. Leng, D. Lischinski, D. Cohen-Or, C. Tu, Y. Li, Shapeconv: Shape-aware convolutional layer for indoor rgb-d semantic segmentation, in: Proc. of Int’l Conf. on Computer Vision (ICCV), 2021, pp. 7088– 7097

work page 2021

Show all 56 references

[9]

Zhang, S

Z. Zhang, S. Fidler, R. Urtasun, Instance-level segmentation for au- tonomous driving with deep densely connected mrfs, in: Proc. of Com- puter Vision and Pattern Recognition (CVPR), 2016, pp. 669–677. 24

2016
[10]

De Brabandere, D

B. De Brabandere, D. Neven, L. Van Gool, Semantic instance segmenta- tion for autonomous driving, in: Proc. of Computer Vision and Pattern Recognition Workshops (CVPR W), 2017, pp. 7–9

2017
[11]

S. Back, J. Lee, T. Kim, S. Noh, R. Kang, S. Bak, K. Lee, Unseen object amodal instance segmentation via hierarchical occlusion modeling, in: IEEE Int’l Conf. on Robotics and Automation (ICRA), IEEE, 2022, pp. 5085–5092

2022
[12]

S. Ito, S. Kubota, Point proposal based instance segmentation with rect- angular masks for robot picking task, in: Proc. of Asian Conf. on Com- puter Vision (ACCV), 2020

2020
[13]

Silberman, D

N. Silberman, D. Hoiem, P. Kohli, R. Fergus, Indoor segmentation and support inference from rgbd images, in: Proc. of European Conf. on Computer Vision (ECCV), Springer, 2012, pp. 746–760

2012
[14]

S. Song, S. P. Lichtenberg, J. Xiao, Sun rgb-d: A rgb-d scene under- standing benchmark suite, in: Proc. of Computer Vision and Pattern Recognition (CVPR), 2015, pp. 567–576

2015
[15]

Zhang, M

H. Zhang, M. Cisse, Y. N. Dauphin, D. Lopez-Paz, mixup: Beyond empirical risk minimization, in: Proc. of Int’l Conf. on Learning Repre- sentations (ICLR) , 2018

2018
[16]

X. Hu, K. Yang, L. Fei, K. Wang, Acnet: Attention based network to exploit complementary features for rgbd semantic segmentation, in: IEEE Int’l Conf. on Image Processing (ICIP), IEEE, 2019, pp. 1440– 1444

2019
[17]

H. Zhou, L. Qi, Z. Wan, H. Huang, X. Yang, Rgb-d co-attention network for semantic segmentation, in: Proc. of Asian Conf. on Computer Vision (ACCV), 2020

2020
[18]

Park, K.-S

S.-J. Park, K.-S. Hong, S. Lee, Rdfnet: Rgb-d multi-level residual fea- ture fusion for indoor semantic segmentation, in: Proc. of Int’l Conf. on Computer Vision (ICCV), 2017, pp. 4980–4989

2017
[19]

X. Ying, M. C. Chuah, Uctnet: Uncertainty-aware cross-modal trans- former network for indoor rgb-d semantic segmentation, in: Proc. of 25 European Conf. on Computer Vision (ECCV), Springer, 2022, pp. 20– 37

2022
[20]

Dosovitskiy, L

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al., An image is worth 16x16 words: Transformers for image recognition at scale, in: Proc. of Int’l Conf. on Learning Representations (ICLR) , 2021

2021
[21]

S. Dong, Y. Feng, Q. Yang, Y. Huang, D. Liu, H. Fan, Efficient multi- modal semantic segmentation via dual-prompt learning, IEEE/RSJ Int’l Conf. on Intelligent Robots and Systems (IROS) (2023)

2023
[22]

K. He, G. Gkioxari, P. Doll´ ar, R. Girshick, Mask r-cnn, in: Proc. of Int’l Conf. on Computer Vision (ICCV), 2017, pp. 2961–2969

2017
[23]

K. Wang, J. H. Liew, Y. Zou, D. Zhou, J. Feng, Panet: Few-shot image semantic segmentation with prototype alignment, in: Proc. of Int’l Conf. on Computer Vision (ICCV), 2019, pp. 9197–9206

2019
[24]

Liang, L

X. Liang, L. Lin, Y. Wei, X. Shen, J. Yang, S. Yan, Proposal-free net- work for instance-level object segmentation, IEEE Trans. on Pattern Anal. Mach. Intell. (TPAMI) 40 (12) (2017) 2978–2991

2017
[25]

Huang, L

Z. Huang, L. Huang, Y. Gong, C. Huang, X. Wang, Mask scoring r-cnn, in: Proc. of Computer Vision and Pattern Recognition (CVPR), 2019, pp. 6409–6418

2019
[26]

Z. Cai, N. Vasconcelos, Cascade r-cnn: High quality object detection and instance segmentation, IEEE Trans. on Pattern Anal. Mach. Intell. (TPAMI) 43 (5) (2019) 1483–1498

2019
[27]

K. Chen, J. Pang, J. Wang, Y. Xiong, X. Li, S. Sun, W. Feng, Z. Liu, J. Shi, W. Ouyang, et al., Hybrid task cascade for instance segmentation, in: Proc. of Computer Vision and Pattern Recognition (CVPR), 2019, pp. 4974–4983

2019
[28]

Bolya, C

D. Bolya, C. Zhou, F. Xiao, Y. J. Lee, Yolact: Real-time instance seg- mentation, in: Proc. of Int’l Conf. on Computer Vision (ICCV), 2019, pp. 9157–9166. 26

2019
[29]

Z. Tian, C. Shen, H. Chen, Conditional convolutions for instance seg- mentation, in: Proc. of European Conf. on Computer Vision (ECCV), Springer, 2020, pp. 282–298

2020
[30]

Carion, F

N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, S. Zagoruyko, End-to-end object detection with transformers, in: Proc. of European Conf. on Computer Vision (ECCV), Springer, 2020, pp. 213–229

2020
[31]

B. Dong, F. Zeng, T. Wang, X. Zhang, Y. Wei, Solq: Segmenting objects by learning queries, Proc. of Neural Information Processing Systems (NeurIPS) 34 (2021) 21898–21909

2021
[32]

Dosovitskiy, L

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, N. Houlsby, An image is worth 16x16 words: Transform- ers for image recognition at scale, in: Proc. of Int’l Conf. on Learning Repres...

2021
[33]

J. Hou, S. Xie, B. Graham, A. Dai, M. Nießner, Pri3d: Can 3d priors help 2d representation learning?, in: Proc. of Int’l Conf. on Computer Vision (ICCV), 2021, pp. 5693–5702

2021
[34]

M. Wang, L. Hu, Y. Bai, X. Yao, J. Hu, S. Zhang, Amnet: a new rgb-d instance segmentation network based on attention and multi-modality, The Visual Computer (2023) 1–15

2023
[35]

Hazirbas, L

C. Hazirbas, L. Ma, C. Domokos, D. Cremers, Fusenet: Incorporating depth into semantic segmentation via fusion-based cnn architecture, in: Proc. of Asian Conf. on Computer Vision (ACCV), Springer, 2016, pp. 213–228

2016
[36]

Jiang, L

J. Jiang, L. Zheng, F. Luo, Z. Zhang, Rednet: Residual encoder- decoder network for indoor rgb-d semantic segmentation, arXiv preprint arXiv:1806.01054 (2018)

2018 arXiv
[37]

Zhang, J.-H

G. Zhang, J.-H. Xue, P. Xie, S. Yang, G. Wang, Non-local aggregation for rgb-d semantic segmentation, IEEE Signal Processing Letters 28 (2021) 658–662. 27

2021
[38]

E. Yang, W. Zhou, X. Qian, L. Yu, Mgcnet: Multilevel gated collabo- rative network for rgb-d semantic segmentation of indoor scene, IEEE Signal Processing Letters 29 (2022) 2567–2571

2022
[39]

W. Zhou, E. Yang, J. Lei, J. Wan, L. Yu, Pgdenet: Progressive guided fusion and depth enhancement network for rgb-d indoor scene parsing, IEEE Transactions on Multimedia 25 (2022) 3483–3494

2022
[40]

Y. Lv, Z. Liu, G. Li, Context-aware interaction network for rgb-t se- mantic segmentation, IEEE Transactions on Multimedia (2024)

2024
[41]

Caesar, J

H. Caesar, J. Uijlings, V. Ferrari, Coco-stuff: Thing and stuff classes in context, in: Proc. of Computer Vision and Pattern Recognition (CVPR), 2018, pp. 1209–1218

2018
[42]

B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, A. Torralba, Scene parsing through ade20k dataset, in: Proc. of Computer Vision and Pat- tern Recognition (CVPR), 2017, pp. 633–641

2017
[43]

Cordts, M

M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benen- son, U. Franke, S. Roth, B. Schiele, The cityscapes dataset for semantic urban scene understanding, in: Proc. of Computer Vision and Pattern Recognition (CVPR), 2016, pp. 3213–3223

2016
[44]

T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll´ ar, C. L. Zitnick, Microsoft coco: Common objects in context, in: Proc. of European Conf. on Computer Vision (ECCV), Springer, 2014, pp. 740–755

2014
[45]

Roberts, J

M. Roberts, J. Ramapuram, A. Ranjan, A. Kumar, M. A. Bautista, N. Paczan, R. Webb, J. M. Susskind, Hypersim: A photorealistic syn- thetic dataset for holistic indoor scene understanding, in: Proc. of Int’l Conf. on Computer Vision (ICCV), 2021, pp. 10912–10922

2021
[46]

G. Ros, L. Sellart, J. Materzynska, D. Vazquez, A. M. Lopez, The syn- thia dataset: A large collection of synthetic images for semantic seg- mentation of urban scenes, in: Proc. of Computer Vision and Pattern Recognition (CVPR), 2016, pp. 3234–3243. 28

2016
[47]

Gaidon, Q

A. Gaidon, Q. Wang, Y. Cabon, E. Vig, Virtual worlds as proxy for multi-object tracking analysis, in: Proc. of Computer Vision and Pattern Recognition (CVPR), 2016, pp. 4340–4349

2016
[48]

Yogamani, C

S. Yogamani, C. Hughes, J. Horgan, G. Sistu, P. Varley, D. O’Dea, M. Uric´ ar, S. Milz, M. Simon, K. Amende, et al., Woodscape: A multi- task, multi-camera fisheye dataset for autonomous driving, in: Proc. of Int’l Conf. on Computer Vision (ICCV), 2019, pp. 9308–9318

2019
[49]

Richtsfeld, T

A. Richtsfeld, T. M¨ orwald, J. Prankl, M. Zillich, M. Vincze, Segmen- tation of unknown objects in indoor environments, in: IEEE/RSJ Int’l Conf. on Intelligent Robots and Systems (IROS), IEEE, 2012, pp. 4791– 4796

2012
[50]

Calli, A

B. Calli, A. Singh, A. Walsman, S. Srinivasa, P. Abbeel, A. M. Dollar, The ycb object and model set: Towards common benchmarks for ma- nipulation research, in: IEEE Int’l Conf. on Robotics and Automation (ICRA), IEEE, 2015, pp. 510–517

2015
[51]

H. Xu, J. Zhang, J. Cai, H. Rezatofighi, F. Yu, D. Tao, A. Geiger, Uni- fying flow, stereo and depth estimation, IEEE Transactions on Pattern Analysis and Machine Intelligence (2023)

2023
[52]

Singh, L

B. Singh, L. S. Davis, An analysis of scale invariance in object detection snip, in: Proc. of Computer Vision and Pattern Recognition (CVPR), 2018, pp. 3578–3587

2018
[53]

C. Du, J. Teng, T. Li, Y. Liu, T. Yuan, Y. Wang, Y. Yuan, H. Zhao, On uni-modal feature learning in supervised multi-modal learning, in: Proc. of Int’l Conf. on Machine Learning (ICML), 2023, pp. 8632–8656

2023
[54]

X. Peng, Y. Wei, A. Deng, D. Wang, D. Hu, Balanced multimodal learn- ing via on-the-fly gradient modulation, in: Proc. of Computer Vision and Pattern Recognition (CVPR), 2022, pp. 8238–8247

2022
[55]

M. D. Zeiler, D. Krishnan, G. W. Taylor, R. Fergus, Deconvolu- tional networks, in: Proc. of Computer Vision and Pattern Recognition (CVPR), IEEE, 2010, pp. 2528–2535

2010
[56]

E. Xie, W. Wang, Z. Yu, A. Anandkumar, J. M. Alvarez, P. Luo, Segformer: Simple and efficient design for semantic segmentation 29 with transformers, Proc. of Neural Information Processing Systems (NeurIPS) 34 (2021) 12077–12090. 30

2021

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

Tsutsui, M

K.-I. Tsutsui, M. Taira, H. Sakata, Neural mechanisms of three- dimensional vision, Neuroscience research 51 (3) (2005) 221–229

work page 2005

[2] [2]

X. Qi, R. Liao, J. Jia, S. Fidler, R. Urtasun, 3d graph neural networks for rgbd semantic segmentation, in: Proc. of Int’l Conf. on Computer Vision (ICCV), 2017, pp. 5199–5208

work page 2017

[3] [3]

Zhang, H

J. Zhang, H. Liu, K. Yang, X. Hu, R. Liu, R. Stiefelhagen, Cmx: Cross- modal fusion for rgb-x semantic segmentation with transformers, arXiv preprint arXiv:2203.04838 (2022)

arXiv 2022

[4] [4]

Chen, K.-Y

X. Chen, K.-Y. Lin, J. Wang, W. Wu, C. Qian, H. Li, G. Zeng, Bi-directional cross-modality feature propagation with separation-and- aggregation gate for rgb-d semantic segmentation, in: Proc. of European Conf. on Computer Vision (ECCV), Springer, 2020, pp. 561–577

work page 2020

[5] [5]

C. Tang, H. Chen, X. Li, J. Li, Z. Zhang, X. Hu, Look closer to segment better: Boundary patch refinement for instance segmentation, in: Proc. of Computer Vision and Pattern Recognition (CVPR), 2021, pp. 13926– 13935

work page 2021

[6] [6]

M. Kim, S. Woo, D. Kim, I. S. Kweon, The devil is in the boundary: Ex- ploiting boundary representation for basis-based instance segmentation, in: Proc. of Winter Conf. on Application of Computer Vision (ECCV), 2021, pp. 929–938

work page 2021

[7] [7]

W. Wang, U. Neumann, Depth-aware cnn for rgb-d segmentation, in: Proc. of European Conf. on Computer Vision (ECCV), 2018, pp. 135– 150

work page 2018

[8] [8]

J. Cao, H. Leng, D. Lischinski, D. Cohen-Or, C. Tu, Y. Li, Shapeconv: Shape-aware convolutional layer for indoor rgb-d semantic segmentation, in: Proc. of Int’l Conf. on Computer Vision (ICCV), 2021, pp. 7088– 7097

work page 2021