Pith. sign in

REVIEW 3 major objections 4 minor 74 references

Native Segmentation Vision Transformers

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that replacing uniform downsampling with a differentiable content-aware grouping layer lets a vision backbone produce hierarchical segmentation masks natively, without dedicated segmentation heads and without mask…

desk verdict SeNaTra is a genuinely new backbone design with strong empirical support; the 'native segmentation' claim is mostly earned but needs direct grouping-quality evidence and code release. read the letter →

arxiv 2505.16993 v1 pith:ARQD372A submitted 2025-05-22 cs.CV cs.LG

Guillem Brasó , Aljoša Ošep , Laura Leal-Taixé This is my paper · ORCID
classification cs.CVcs.LG
keywords nativesegmentationvisiontransformerbackbonespatialgroupinglayerdifferentiableclusteringzero-shotsemanticlearneddownsamplinghierarchicalMarkovchaintokenassignment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Modern vision backbones shrink feature maps with grid-based pooling or strided convolution that ignores image content, and segmentation then relies on heavy decoder heads to repair the resulting misalignment. This paper proposes a hierarchical vision transformer whose downsampling steps are instead a learned spatial grouping layer: tokens are iteratively assigned to a smaller set of output tokens by feature similarity, in the style of differentiable clustering. Because the assignments from every stage compose, the backbone itself carries a hierarchy of pixel-to-region maps, so segmentation masks can be read directly from the backbone. The paper shows these backbone-level masks give strong zero-shot segmentation after image-text training, with no mask supervision and no postprocessing, and that the same backbone improves semantic and panoptic segmentation when a standard head is attached.

What carries the argument

The central object is the spatial grouping layer, a fully differentiable iterative clustering module that treats downsampled tokens as cluster centroids. It initializes centroids with a strided convolution, then for a few iterations computes soft assignments from input tokens to centroids, renormalizes the assignments, and updates each centroid as the weighted mean of inputs assigned to it. Early stages restrict assignments to a small local window, keeping the cost linear in resolution; the final stage uses dense assignment so regions can merge across the whole image. Because each assignment matrix spreads a token's weight across outputs in probabilities that sum to one, the per-stage matrices compose like a Markov chain, which turns downsampling into native, hierarchy-preserving segmentation masks.

What would settle it

One decisive experiment is to take a mask-free SeNaTra model and run it on images where two adjacent objects share nearly identical texture and color but belong to different classes, with no low-level edge between them. If the final dense grouping layer consistently merges them into one mask, the similarity-grouping assumption fails at exactly the point the native-segmentation claim depends on; a small benchmark of such adversarial pairs would quantify where the grouping stops being semantic.

Watch

Extended reading notes

Core claim

The central claim is that segmentation can be a native property of the backbone rather than a separate decoding stage. SeNaTra replaces each uniform downsampling layer with a spatial grouping layer that initializes output tokens by a strided convolution, then alternates between computing soft assignments from input tokens to those output tokens and updating the output tokens as weighted means of their assigned inputs. All except the final grouping layer restrict assignments to small local windows, which keeps complexity linear in input resolution; the final dense layer lets output tokens merge regions across the whole image. The learned assignment matrices are stochastic, so each input token splits its weight across output tokens in probabilities that sum to one, and they compose across stages like a Markov chain, giving a principled operator for upsampling and downsampling features. The paper reports that with only image-level supervision, superpixel-like groups emerge in early stages and semantically coherent regions emerge in the final stage, and that these native masks outperform prior zero-shot segmentation methods without any mask supervision.

Load-bearing premise

The load-bearing premise is that image tokens with similar internal feature vectors belong to the same object or semantically meaningful region, so grouping by similarity respects real boundaries; if intermediate features do not align with semantic boundaries, the native masks would fragment or merge objects even when the classification head is right.

Editorial extensions

If this is right

  • With the backbone's own assignments available, semantic segmentation can be done by a two-layer MLP on final tokens plus an upsampling step, removing dedicated decoder heads and their parameter and FLOP cost.
  • On zero-shot text-supervised segmentation, native masks from the backbone outperform prior methods on several benchmarks without CRF or PAMR postprocessing, and even beat models pre-trained on 20 times more image-text data on most datasets.
  • Replacing uniform upsampling with learned assignment-based upsampling improves standard segmentation heads such as Mask2Former when SeNaTra is used as a drop-in backbone.
  • The entire architecture stays end-to-end trainable on image-level losses, so mask labels are not required for coherent masks to emerge.
  • With mask supervision, native masks from a tiny variant already surpass a MaskFormer with a Swin-T backbone on COCO panoptic segmentation, and the backbone improves state-of-the-art performance when combined with dedicated heads.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper's experiments, the per-stage assignment maps could be read as boundary signals for other dense tasks, such as edge detection or depth discontinuities.
  • A test the paper does not run: varying the number of grouping iterations and local-window size should reveal a direct trade-off between boundary precision and semantic coherence if similarity grouping is the active mechanism.
  • The paper's own comparison of semantic versus panoptic gains suggests an untested route: an instance-oriented pre-training objective might reduce the instance-level gap.
  • Since early local grouping keeps cost linear in resolution, a resolution-ablation study would show whether mask boundary quality improves predictably as input size grows.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces SeNaTra, a hierarchical vision backbone in which uniform grid-based downsampling is replaced by a content-aware spatial grouping layer. The layer iteratively computes soft assignments between input tokens and a reduced set of output tokens via cross-attention, with local 3x3 windows in early stages for efficiency and dense grouping in the final stage. The authors argue that composing these assignment matrices across stages yields multi-scale segmentation masks natively in the backbone, without dedicated segmentation decoders. They evaluate SeNaTra in three regimes: ImageNet classification, zero-shot text-supervised semantic segmentation, and fully supervised semantic and panoptic segmentation on ADE20k and COCO, reporting strong results both for native MLP-based masks and as a drop-in backbone for Mask2Former.

Significance. If the central claim holds, this is a meaningful conceptual contribution: it shows that a backbone can produce semantic masks as a byproduct of learned downsampling, potentially simplifying the standard encoder-decoder segmentation pipeline and improving data efficiency. The experimental scope is broad and mostly well controlled: comparisons include standard and grouping-based backbones, several segmentation heads, zero-shot and fully supervised settings, and ablations of the grouping layer design. The efficient sparse implementation is also a practical strength, as it makes the method scalable to high resolutions. However, the strongest interpretation of the paper's claim—that the learned assignments are themselves semantically coherent segmentation masks—is supported only indirectly, and the manuscript does not currently release code, checkpoints, or seed-variance information. These gaps matter because several headline improvements are small and because a classification head can compensate for imperfect grouping.

major comments (3)
  1. [Section 3.1, Eq. (1), Tables 1 and 2a] The paper's central claim is that the learned assignment matrices form semantically coherent segmentation masks, yet all quantitative evidence is downstream mIoU obtained by classifying final group tokens and upsampling the resulting logits. A classifier can assign a mixed group to its dominant class, so high mIoU does not isolate assignment quality. This is load-bearing because early grouping errors are irreversible: once an early local layer merges patches across an object boundary, the final dense grouping layer operates on already-merged tokens and cannot split them. Please add direct measures of group quality, for example achievable segmentation accuracy when each final group is labeled by its majority ground-truth class, boundary precision/recall or contour IoU, and group purity, computed on held-out images from ADE20k and COCO at stages 3 and 4.
  2. [Section 4.2, Appendix D.3] The claim that segmentation arises 'solely from grouping layers' and 'without dedicated segmentation heads' is softened by the actual native pipeline: semantic segmentation uses a 2-layer MLP classifier, an auxiliary loss at the penultimate stage, and for panoptic segmentation an additional 2-layer MLP over the top-100 final tokens supervised with bipartite matching, plus a recomputed final assignment via dot-product with projected penultimate-stage features. These components are not necessarily objectionable, but the wording overstates the result. Please define precisely what counts as a segmentation head and restate the contribution as: the grouping layers provide the masks, while lightweight per-token classification MLPs provide labels.
  3. [Appendix D and Tables 1, 2] Empirical claims rely on single runs with no reported variance across seeds, and the appendix states that code and pre-trained models 'will be made publicly available' without providing release artifacts. Given that several reported gains are small (e.g., +1.0 mIoU for SeNaTra-T + M2F over Swin-T + M2F on ADE20k, and +0.7 PQ for SeNaTra-L + M2F over Swin-L + M2F on COCO), the lack of code, checkpoints, and seed-level results makes it difficult to distinguish genuine improvement from training noise. Please release the code and models and report results over multiple seeds, or state if the reported numbers are single-run and include error bars where feasible.
minor comments (4)
  1. [Appendix E.2] The text says 'In Table 8 we compare three implementation approaches' but the table reporting None, Naive, and CUDA implementations is Table 9; Table 8 compares NAT-B with UperNet against SeNaTra-B with native segmentation. Please correct the cross-reference.
  2. [Appendix D.1] The discussion of ImageNet results ends with 'as it can be observed qualitatively in ??', which is a missing cross-reference to Figure 3.
  3. [Section 4.2.2, Table 2b] The sentence 'our native results surpass consolidated baselines' is too broad: SeNaTra-T Native (49.2 PQ) is below Swin-T with M2F (53.2 PQ) and NAT-T with M2F (54.3 PQ), although it exceeds MaskFormer with Swin-T (47.7 PQ). Please specify that the comparison is against MaskFormer-based decoders and other native or head-free approaches.
  4. [Table 3a] The column labels S1, S2, S3 are not defined in the table caption; please clarify whether they refer to grouping layers after each of the first three backbone stages or to grouping at stages 1-3, and specify the baseline in the first row.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: native segmentation is defined by the grouping mechanism, but its quality is validated on external benchmarks.

full rationale

The paper's derivation chain is self-contained rather than circular. The spatial grouping layer is specified by an explicit algorithm (Algorithm 1) using cross-attention-like soft assignments and centroid updates, and the 'native segmentation' masks are formally defined as the composition of the resulting assignment matrices in Eq. 1. The claim that these masks are semantically meaningful is not obtained by definition or by fitting a target quantity; it is tested against external benchmarks (ADE20k, COCO-panoptic, Pascal VOC, COCO-Stuff, Cityscapes, etc.) under both mask-free and mask-supervised settings. No fitted parameter is relabeled as a prediction, no load-bearing result is justified solely by a self-citation, and no uniqueness theorem from the authors is invoked to force the design. The only definitional overlap is that the paper calls its learned assignments 'segmentation masks,' which is a naming choice; the empirical strength of those masks remains an independent, falsifiable claim. Therefore no circular step meets the evidentiary bar required by the analysis rules.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No new physical or latent entities are postulated; the grouping layer is a differentiable module, and 'native segmentation' is a capability, not an entity. Free parameters are standard hyperparameters chosen by hand, not fitted to explain a specific result.

free parameters (4)
  • Number of grouping iterations L = 3
    Chosen by hand for all experiments; not ablated. Controls refinement depth in Algorithm 1.
  • Local window size = 3x3
    Used for grouping in stages 2 and 3; set to 3x3, a design choice limiting cross-attention context.
  • Temperature tau in cross-attention = not specified
    Used in Algorithm 1 L3 to scale attention logits; value not reported, affecting assignment sharpness.
  • Top-100 token selection for panoptic things MLP = 100
    In panoptic native model, only top-100 final tokens with largest assignments are used as object candidates; selected by hand.
assumptions (4)
  • domain assumption Tokens with similar feature embeddings belong to the same object or semantically meaningful region.
    Stated in Section 3.1, this is the core assumption that makes grouping by feature similarity yield semantic segments.
  • ad hoc to paper The 3x3 local window in early grouping layers is sufficient to capture semantic boundaries.
    Locality prior in Section 3.1; enables linear complexity but assumes small windows cover meaningful structures.
  • domain assumption Iterative soft-assignment refinement converges to useful centroids with skip connections instead of GRU.
    Algorithm 1 and ablations show empirical benefit, but convergence to semantic groups is not proven analytically.
  • standard math Composing row-stochastic assignment matrices yields meaningful upsampling maps.
    Markov chain composition in Section 3.2 is mathematically defined; meaningfulness for segmentation is assumed and validated empirically.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Native Segmentation Vision Transformers." pith.science (2026). https://pith.science/paper/ARQD372A

@misc{pith2026250516993,
  author       = {Pith},
  title        = {Pith review of: Native Segmentation Vision Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ARQD372A}},
  note         = {Machine review of arXiv:2505.16993}
}
read the original abstract

Uniform downsampling remains the de facto standard for reducing spatial resolution in vision backbones. In this work, we propose an alternative design built around a content-aware spatial grouping layer, that dynamically assigns tokens to a reduced set based on image boundaries and their semantic content. Stacking our grouping layer across consecutive backbone stages results in hierarchical segmentation that arises natively in the feature extraction process, resulting in our coined Native Segmentation Vision Transformer. We show that a careful design of our architecture enables the emergence of strong segmentation masks solely from grouping layers, that is, without additional segmentation-specific heads. This sets the foundation for a new paradigm of native, backbone-level segmentation, which enables strong zero-shot results without mask supervision, as well as a minimal and efficient standalone model design for downstream segmentation tasks. Our project page is https://research.nvidia.com/labs/dvl/projects/native-segmentation.

Figures

Figures reproduced from arXiv: 2505.16993 by the authors.

Figure 1
Figure 1. Downsampling in vision backbones via uniform downsampling (top) v.s. learned downsampling (bottom, this work): Vision backbones downsample feature maps using uniform-grid operators (e.g., pooling, top) and rely on uniform upsampling (e.g., bilinear interpolation, top) for image segmentation tasks. Our new backbone with spatial grouping layers learns to map pixels to a reduced set of tokens, aligning with image bound… view at source ↗
Figure 2
Figure 2. Overall model design. Visualization of our hierarchical architecture and its key components. (a) Our backbone architecture consists of four processing stages interconnected by content-aware grouping layers for downsampling. (b) Core operations of our Spatial Grouping Layer, which computes soft token assignments and updates group features iteratively (detailed in Algorithm 1). (c) The composition of learned assignmen… view at source ↗
Figure 3
Figure 3. Segmentation emerges from ImageNet pre-training. We visualize group decompositions across each backbone stage, along with their upsampled activations over the predicted class. We observe that even in the absence of mask supervision, super-pixel-like structures emerge in earlier layers, and are eventually grouped into semantically coherent regions in dense grouping layers. 7 [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Qualitative zero-shot segmentation learned from image-text contrastive pre-training. We visualize hierarchical final decompositions along with their predicted semantic masks, obtained in a zero-shot setting on Pascal VOC validation images [46], and corresponding ground…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

74 extracted references · 59 canonical work pages

  1. [1]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021

    2021

  2. [2]

    Convnext v2: Co-designing and scaling convnets with masked autoencoders

    Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, and Saining Xie. Convnext v2: Co-designing and scaling convnets with masked autoencoders. In CVPR, 2023

    work page 2023

  3. [3]

    Neighborhood attention transformer

    Ali Hassani, Steven Walton, Jiachen Li, Shen Li, and Humphrey Shi. Neighborhood attention transformer. In CVPR, 2023. 10

    work page 2023

  4. [4]

    Backpropagation applied to handwritten zip code recognition

    Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 1(4):541–551, 1989

    1989

  5. [5]

    K. He, G. Gkioxari, P. Dollár, and R. Girshick. Mask R-CNN. In ICCV, 2017

    2017

  6. [6]

    Schwing, Alexander Kirillov, and Rohit Girdhar

    Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In CVPR, 2022

    2022

  7. [7]

    Feature pyramid networks for object detection

    Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, 2017

    2017

  8. [8]

    FaPN: Feature-aligned pyramid network for dense image prediction

    Shihua Huang, Zhichao Lu, Ran Cheng, and Cheng He. FaPN: Feature-aligned pyramid network for dense image prediction. In ICCV, 2021

    work page 2021

Show all 74 references
  1. [9]

    Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and X. Wang. Groupvit: Semantic segmentation emerges from text supervision. In CVPR, 2022

    2022

  2. [10]

    Clusterformer: Clustering as a universal visual learner

    James C Liang, Yiming Cui, Qifan Wang, Tong Geng, Wenguan Wang, and Dongfang Liu. Clusterformer: Clustering as a universal visual learner. In Adv. Neural Inform. Process. Syst., 2023

    2023

  3. [11]

    Learning hierarchical image segmentation for recognition and by recognition

    Tsung-Wei Ke, Sangwoo Mo, and X Yu Stella. Learning hierarchical image segmentation for recognition and by recognition. In ICLR, 2023

    2023

  4. [12]

    Tcformer: Visual recognition via token clustering transformer

    Wang Zeng, Sheng Jin, Lumin Xu, Wentao Liu, Chen Qian, Wanli Ouyang, Ping Luo, and Xiaogang Wang. Tcformer: Visual recognition via token clustering transformer. IEEE Trans. Pattern Anal. Mach. Intell., 2024

    2024

  5. [13]

    Schwing, and Alexander Kirillov

    Bowen Cheng, Alexander G. Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. In Adv. Neural Inform. Process. Syst., 2021

    2021

  6. [14]

    Slic superpixels compared to state-of-the-art superpixel methods

    Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süsstrunk. Slic superpixels compared to state-of-the-art superpixel methods. IEEE TPAMI, 34(11):2274–2282, 2012

    2012

  7. [15]

    Object-centric learning with slot attention

    Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. In Adv. Neural Inform. Process. Syst., 2020

    2020

  8. [16]

    Mean shift: A robust approach toward feature space analysis

    Dorin Comaniciu and Peter Meer. Mean shift: A robust approach toward feature space analysis. IEEE TPAMI, 24(5):603–619, 2002

    2002

  9. [17]

    Felzenszwalb and Daniel P

    Pedro F. Felzenszwalb and Daniel P. Huttenlocher. Efficient graph-based image segmentation. IJCV, 59(2):167–181, 2004

    2004

  10. [18]

    Contour detection and hierarchical image segmentation

    Pablo Arbelaez, Michael Maire, Charless Fowlkes, and Jitendra Malik. Contour detection and hierarchical image segmentation. IEEE TPAMI, 33(5):898–916, 2011

    2011

  11. [19]

    Seeds: Superpixels extracted via energy-driven sampling

    Michael Van den Bergh, Xavier Boix, Gemma Roig, Benjamin de Capitani, and Luc Van Gool. Seeds: Superpixels extracted via energy-driven sampling. In ECCV, 2012

    2012

  12. [20]

    Semantic understanding of scenes through the ade20k dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. IJCV, 2019

    2019

  13. [21]

    Panoptic segmentation

    Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár. Panoptic segmentation. In CVPR, 2019

    2019

  14. [22]

    Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position

    Kunihiko Fukushima. Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biological cybernetics, 36(4):193–202, 1980

    1980

  15. [23]

    Gradient-based learning applied to document recognition

    Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998. 11

    1998

  16. [24]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at...

    2021

  17. [25]

    A convnet for the 2020s

    Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In CVPR, 2022

    2022

  18. [26]

    Sam 2: Segment anything in images and videos

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenh...

    2024 arXiv

  19. [27]

    Fully convolutional networks for semantic segmentation

    Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR, 2015

    2015

  20. [28]

    U-net: Convolutional networks for biomedical image segmentation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, 2015

    2015

  21. [29]

    Rich feature hierarchies for accurate object detection and semantic segmentation

    Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, 2014

    2014

  22. [30]

    Fast r-cnn

    Ross Girshick. Fast r-cnn. In ICCV, pages 1440–1448, 2015

    2015

  23. [31]

    End-to-end object detection with transformers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020

    2020

  24. [32]

    Normalized cuts and image segmentation

    Jianbo Shi and Jitendra Malik. Normalized cuts and image segmentation. IEEE TPAMI, 22(8):888–905, 2000

    2000

  25. [33]

    Multiscale combinatorial grouping

    Pablo Arbeláez, Jordi Pont-Tuset, Jon Barron, Ferran Marques, and Jitendra Malik. Multiscale combinatorial grouping. CVPR, 2014

    2014

  26. [34]

    Superpixel samping networks

    Varun Jampani, Deqing Sun, Ming-Yu Liu, Ming-Hsuan Yang, and Jan Kautz. Superpixel samping networks. In ECCV, 2018

    2018

  27. [35]

    S. P. Lloyd. Least squares quantization in pcm. IEEE Transactions on Information Theory, 28(2):129–137, 1957

    1957

  28. [36]

    J. B. MacQueen. Some methods for classification and analysis of multivariate observations. Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability , 1:281–297, 1967

    1967

  29. [37]

    Unified perceptual parsing for scene understanding

    Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In ECCV. Springer, 2018

    2018

  30. [38]

    Berg, and Li Fei-Fei

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. IJCV, 115(3):211–252, 2015

    2015

  31. [39]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Int. Conf. Mach. Learn., 2021

    2021

  32. [40]

    Le, Yun- Hsuan Sung, Zhen Li, and Tom Duerig

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V . Le, Yun- Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In Int. Conf. Mach. Learn., 2021

    2021

  33. [41]

    A simple framework for text-supervised semantic segmentation

    Muyang Yi, Quan Cui, Hao Wu, Cheng Yang, Osamu Yoshie, and Hongtao Lu. A simple framework for text-supervised semantic segmentation. In CVPR, 2023. 12

    2023

  34. [42]

    Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning

    Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Assoc. for Comp. Ling., 2018

    2018

  35. [43]

    Conceptual 12m: Push- ing web-scale image-text pre-training to recognize long-tail visual concepts

    Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Push- ing web-scale image-text pre-training to recognize long-tail visual concepts. arXiv preprint arXiv:2102.08981, 2021

    2021 arXiv

  36. [44]

    Redcaps: Web-crawled image-text data created by the people, for the people

    Karan Desai and Justin Johnson. Redcaps: Web-crawled image-text data created by the people, for the people. In Adv. Neural Inform. Process. Syst., 2021

    2021

  37. [45]

    Learning to generate text-grounded mask for open-world semantic segmentation from only image-text pairs

    Junbum Cha, Jonghwan Mun, and Byungseok Roh. Learning to generate text-grounded mask for open-world semantic segmentation from only image-text pairs. In CVPR, 2023

    2023

  38. [46]

    Everingham, L

    M. Everingham, L. Van Gool, C.K.I. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (VOC) challenge. IJCV, 88(2):303–338, 2010

    2010

  39. [47]

    The role of context for object detection and semantic segmentation in the wild

    Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In CVPR, 2014

    2014

  40. [48]

    T.Y . Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014

    2014

  41. [49]

    Coco-stuff: Thing and stuff classes in context

    Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In CVPR, 2018

    2018

  42. [50]

    Cordts, M

    M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016

    2016

  43. [51]

    Open- world semantic segmentation via contrasting and clustering vision-language embedding

    Quande Liu, Youpeng Wen, Jianhua Han, Chunjing Xu, Hang Xu, and Xiaodan Liang. Open- world semantic segmentation via contrasting and clustering vision-language embedding. In ECCV, 2022

    2022

  44. [52]

    Segclip: Patch aggregation with learnable centers for open-vocabulary semantic segmentation

    Huaishao Luo, Junwei Bao, Youzheng Wu, Xiaodong He, and Tianrui Li. Segclip: Patch aggregation with learnable centers for open-vocabulary semantic segmentation. In Int. Conf. Mach. Learn., 2023

    2023

  45. [53]

    Image-text co-decomposition for text-supervised semantic segmentation

    Ji-Jia Wu, Andy Chia-Hao Chang, Chieh-Yu Chuang, Chun-Pei Chen, Yu-Lun Liu, Min-Hung Chen, Hou-Ning Hu, Yung-Yu Chuang, and Yen-Yu Lin. Image-text co-decomposition for text-supervised semantic segmentation. In CVPR, 2024

    2024

  46. [54]

    Viewco: Discovering text-supervised segmentation masks via multi-view semantic consistency

    Pengzhen Ren, Changlin Li, Hang Xu, Yi Zhu, Guangrun Wang, Jianzhuang Liu, Xiaojun Chang, and Xiaodan Liang. Viewco: Discovering text-supervised segmentation masks via multi-view semantic consistency. In ICLR, 2023

    2023

  47. [55]

    Rewrite caption semantics: Bridging semantic gaps for language-supervised semantic segmentation

    Yun Xing, Jian Kang, Aoran Xiao, Jiahao Nie, Shao Ling, and Shijian Lu. Rewrite caption semantics: Bridging semantic gaps for language-supervised semantic segmentation. In NeurIPS, 2023

    2023

  48. [56]

    Uncovering prototypical knowledge for weakly open-vocabulary semantic segmentation

    Fei Zhang, Tianfei Zhou, Boyang Li, Hao He, Chaofan Ma, Tianjiao Zhang, Jiangchao Yao, Ya Zhang, and Yanfeng Wang. Uncovering prototypical knowledge for weakly open-vocabulary semantic segmentation. In NeurIPS, 2023

    2023

  49. [57]

    Efficient inference in fully connected crfs with gaussian edge potentials

    Philipp Krähenbühl and Vladlen Koltun. Efficient inference in fully connected crfs with gaussian edge potentials. In Adv. Neural Inform. Process. Syst., 2011

    2011

  50. [58]

    Single-stage semantic segmentation from image labels

    Nikita Araslanov and Stefan Roth. Single-stage semantic segmentation from image labels. In CVPR, 2020

    2020

  51. [59]

    Vmamba: Visual state space model

    Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, Jianbin Jiao, and Yunfan Liu. Vmamba: Visual state space model. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information P...

    2024

  52. [60]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In CVPR, pages 24185–24198, 2024

    2024

  53. [61]

    Panoptic feature pyramid networks

    Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollar. Panoptic feature pyramid networks. In CVPR, June 2019

    2019

  54. [62]

    Segmenter: Transformer for semantic segmentation

    Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 7262–7272, October 2021

    2021

  55. [63]

    Ondrej Biza, Sjoerd Van Steenkiste, Mehdi S. M. Sajjadi, Gamaleldin F. Elsayed, Aravindh Mahendran, and Thomas Kipf. Invariant slot attention: object discovery with slot-centric reference frames. In Int. Conf. Mach. Learn., 2023

    2023

  56. [64]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. ECCV, 2024

    2024

  57. [65]

    Self-attention with relative position rep- resentations

    Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position rep- resentations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers),...

    2018

  58. [66]

    Roformer: Enhanced transformer with rotary position embedding

    Hangbo Su, Zhi Yang, Zhiwei Gao, Nannan Zheng, Song Bai, Wei Tan, Huan Li, Chao Qian, and Jianlong Fu. Roformer: Enhanced transformer with rotary position embedding. In ICLR, 2021

    2021

  59. [67]

    Rotary position embedding for vision transformer

    Byeongho Heo, Song Park, Dongyoon Han, and Sangdoo Yun. Rotary position embedding for vision transformer. In ECCV, 2024

    2024

  60. [68]

    Flashattention: Fast and memory-efficient exact attention with io-awareness

    Thang Dao, Han Hu, Hyungwoo Kwon, Xuehai Zhang, Lisha Song, Dmitriy Vasilenko, Yi Gu, Xinyu Yang, Khac Duy Nguyen, and Kyunghyun Lee. Flashattention: Fast and memory-efficient exact attention with io-awareness. In NeurIPS, 2022

    2022

  61. [69]

    Faster neighborhood attention: Reducing the o(n^2) cost of self attention at the threadblock level

    Ali Hassani, Wen mei Hwu, and Humphrey Shi. Faster neighborhood attention: Reducing the o(n^2) cost of self attention at the threadblock level. In Adv. Neural Inform. Process. Syst., 2024

    2024

  62. [70]

    Training data-efficient image transformers; distillation through attention

    Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers; distillation through attention. In Int. Conf. Mach. Learn., 2021

    2021

  63. [71]

    Weinberger

    Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger. Deep networks with stochastic depth. In European Conference on Computer Vision (ECCV), pages 646–661. Springer, 2016. 14 Appendix Overview. We structure the appendix as follows: in Appendix A and Appendix B...

    2016

  64. [72]

    Empirically, we observe a negligible decrease of downstream classification and segmentation performance, and a significant increase in speed

    with RoPe, and leverage the recently proposed fused kernels from [69]. Empirically, we observe a negligible decrease of downstream classification and segmentation performance, and a significant increase in speed. Particularly on newer hardware, e.g., A100s, this change results...

  65. [73]

    An image of {CLASS }

    produces coarse patch-class activa- tions and relies on postprocessing with Conditional Random Fields [57] to obtain pixel-precise masks. Our method does not require such heuristic postprocessing and instead utilizes our upsampling operations (Section 3.2) to produce pixel-lev...

  66. [74]

    library. In Algorithm 2, we outline the high-level implementation of the sparse variant of the cross- attention and re-normalization operations described in Algorithm 1 (L3-8, excluding the use of LN) using PyTorch. Abusing notation, we denote q = q(Xout), k = k(Xin), v = v(Xi...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.