Pith. sign in

REVIEW 3 major objections 6 minor 47 references

Incorporating Feature Pyramid Tokenization and Open Vocabulary Semantic Segmentation

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Tokenizing a frozen vision-language model's feature pyramid into semantic codebooks improves open-vocabulary segmentation.

desk verdict A solid incremental architecture for open-vocab segmentation with a careful ablation study; the semantic-bridge narrative is under-evidenced but the core empirical claim holds. read the letter →

arxiv 2412.14145 v1 pith:TIA5MDFC submitted 2024-12-18 cs.CV

Jianyu Zhang , Li Zhang , Shijian Li This is my paper · ORCID
classification cs.CV
keywords featurepyramidtokenizationopen-vocabularysemanticsegmentationvectorquantizationvision-languagemodelvisualvMFclusteringdecoupledlearningCOCO-Stuff
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that open-vocabulary semantic segmentation can be improved by treating segmentation as tokenization: rather than feeding a frozen vision-language model's raw features to a segmenter, PAT clusters the multi-resolution feature pyramid into learnable codebooks, so each codebook entry becomes an intermediate 'meta semantic' between pixels and object labels. These tokens are learned end-to-end by jointly reconstructing pixels and predicting segmentation masks, with separate pixel and semantic branches to stop the two objectives from fighting. The result, if correct, is that the cleaned token pyramid visibly carries more semantic structure than the raw VLM features and lifts the baseline segmenter on four open-vocabulary benchmarks by about 0.8 and 1.6 mIoU on average with CLIP and EVA-CLIP backbones, while adding only 23M trainable parameters. The deeper claim is that segmentation and visual tokenization are the same operation at different granularities.

What carries the argument

The load-bearing component is the stage-wise PAT VQ module, which combines vector quantization with von Mises-Fisher meanshift attention (HSAttn) to cluster CLIP features into 32-dimensional codebook tokens at 4x, 2x, and 1x resolutions. The paper argues that VQ can be viewed as hard attention that assigns features to nearest centroids, while HSAttn performs a soft meanshift update of those centroids, so the codebook entries become 'meta semantic' clusters. Around this, the method builds two loosely coupled streams: a pixel branch that carries residual features and a semantic branch that feeds tokens into a Side Adapter network, plus a shared U-Net-like SPADE decoder that fuses all token stages for both reconstruction and mask prediction. The ablation singles out the vMF prior as the critical design choice, since removing it drops Pascal Context-59 mIoU from 57.9 to 53.0.

What would settle it

Train the same PAT architecture on a VLM whose intermediate layer order has been shuffled or whose weights are randomly initialized, and compare segmentation mIoU; if the method still improves over its baseline or produces separable tokens, the 'progressive semantic pyramid' premise is not load-bearing. A cheaper check is to measure class separability, such as linear probe accuracy, of the upscaled VLM features at each pyramid level and show that it must increase monotonically with depth for the method's bridge assumption to hold.

Watch

Extended reading notes

Core claim

PAT's central discovery is that a pretrained VLM's intermediate features, when upscaled and quantized stage-by-stage with vMF-VQ codebooks, can be turned into tokens that are both perceptually faithful and semantically meaningful, and that these tokens serve as a bridge from image-level to pixel-level understanding. The paper shows this by visualizing the tokenized feature pyramid, where low-level stages capture edges and colors and later stages capture parts and objects, and by measuring segmentation gains: PAT improves the SAN baseline from 55.1 to 57.9 mIoU on Pascal Context-59 and from 30.4 to 31.6 on ADE20K-150 with EVA-CLIP, with comparable gains on the two full-vocabulary sets. It also finds that a single shared decoder can decode both pixels and semantics from the same token pyramid, and that the tokenizer remains detachable from the segmentation head after training.

Load-bearing premise

The whole method rests on the premise that a frozen vision-language model's feature pyramid, once upscaled, contains clean, progressively more semantic clusters that the stage-wise codebooks can capture; if those features were not already hierarchically organized, the tokenization would have nothing meaningful to compress.

Editorial extensions

If this is right

  • A detached PAT tokenizer could be reused for generation or multimodal interaction, giving spatial-semantic control over the frozen VLM's features without retraining them.
  • The decoupled pixel/semantic design implies that perceptual and semantic compression conflicts can be managed architecturally, which may transfer to other dense prediction tasks that reuse frozen features.
  • Because training uses only COCO-Stuff and evaluation transfers to four held-out datasets, successful tokenization means strong annotation efficiency for adapting VLMs to pixel-level tasks.
  • If gains grow with clearer VLM representations, as the CLIP-versus-EVA-CLIP comparison shows, PAT's benefit should compound as future VLMs improve.
  • Tokenizing instead of directly decoding raw features reduces resolution noise, which suggests the method also serves as a form of feature denoising for dense prediction.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One could probe the codebook with text embeddings to name each meta-token, turning PAT into an interpretable concept inventory that is not claimed in the paper.
  • The ablation comparing 4x and 1x pyramids implies the upsampled resolutions carry complementary granularities; a direct test would retune codebook sizes per backbone and check whether gains persist across VLM architectures.
  • A stronger claim than the paper's is that any dense prediction head consuming VLM features would benefit from PAT tokenization; that is testable by swapping the Side Adapter for other heads.
  • The paper trains one decoder for both reconstruction and segmentation; measuring whether the shared decoder transfers to unseen tasks (e.g., depth or edges) would clarify how 'united' the perceptual-semantic encoding really is.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes Feature Pyramid Tokenization (PAT), a method for open-vocabulary semantic segmentation that tokenizes the multi-resolution feature pyramid of a frozen VLM (CLIP/EVA-CLIP) using stage-wise vMF vector-quantization codebooks. A decoupled pixel branch reconstructs the input image and a semantic branch fuses the hierarchical tokens with a Side Adapter Network (SAN) backbone; a shared decoder unifies pixel and semantic decoding. The model is trained on COCO-Stuff and evaluated on Pascal Context-59, ADE20K-150, Pascal Context-459, and ADE20K-847. The authors report consistent improvements over the SAN baseline and competitive performance against prior methods. The central conceptual claim is that PAT 'enhances the semantic intuition' of the VLM feature pyramid, turning the pyramid into a sequence of meta-semantic tokens that compose from low-level patterns to high-level abstractions and act as a bridge between pixel-level and image-level understanding.

Significance. If the central claims hold, PAT would be a useful contribution to open-vocabulary segmentation: it demonstrates a lightweight way to extract semantic structure from pretrained VLM feature pyramids and connects visual tokenization with dense prediction. The paper's strengths include a clear architecture, a fairly extensive ablation study (Table II), evaluation on four standard benchmarks, and the explicit attempt to cast segmentation as semantic tokenization. The main conceptual claim of a progressive semantic ladder is, however, supported only by qualitative visualizations and indirect ablations, and the quantitative results are reported without variance. These gaps currently prevent full confidence in the paper's central narrative, although the empirical improvements over the baseline are plausible.

major comments (3)
  1. [III-A, Figs. 1 and 6] The claim that PAT reveals a stage-wise semantic ladder in the VLM feature pyramid is not quantitatively tested. The evidence is visual (Fig. 1 row 2, Fig. 6) and indirect ablations in Table II (Scale variants, No vMF meanshift). These results are also consistent with a purely resolution-based explanation: high-resolution features are noisy for clustering and low-resolution features lack detail. The authors should provide a direct measure of per-stage semantic coherence, e.g., linear probing each tokenized stage against segmentation labels, or measuring cluster purity or vMF concentration per stage, and ideally compare with a control that shuffles or reverses the stage ordering while keeping the multi-resolution pyramid. Without such a test, the central 'semantic bridge' narrative is underdetermined.
  2. [IV (first paragraph) and Table I] The paper states that variance is smoothed by a 5-run average, but no standard deviations or confidence intervals are reported anywhere. The reported gains over SAN are small (e.g., +0.78 and +1.6 mIoU averaged over datasets, and differences of 0.8 or less on several individual datasets), and ablation differences in Table II (e.g., No Spatial Alignment vs. Baseline, 57.1 vs 57.9) are of similar magnitude. Without error bars or a significance test, these differences could be within run-to-run noise. Please report per-run numbers or standard deviations for all main tables and ablations.
  3. [III-A, Eq. (2) and Table II] The vMFVQ module is load-bearing (removing the vMF meanshift drops PC59 by 4.9 points), yet the paper never defines it. The text says 'vMFVQ is VQ with vMF prior' and refers to Eq. (1), but it does not specify how the vMF prior is imposed, how the meanshift is integrated with codebook learning, or what loss terms are used for codebook updates. This makes the central technique impossible to reproduce from the manuscript. Please provide a precise formulation of vMFVQ, including the meanshift update and any commitment or alignment losses.
minor comments (6)
  1. [Table II] The four rows after 'Separate Decoding' are all labeled 'FPN = (Early, Mid, Late)'; presumably they correspond to removing individual stages, but as printed the ablation cannot be interpreted. Please fix the labels.
  2. [Abstract] The sentence 'The visual understanding are often approached' should be 'Visual understanding is often approached.'
  3. [IV (first paragraph)] The phrase 'smooth the variance by 5-run average' is unclear; it likely means 'we report the average over 5 runs.'
  4. [Fig. 5 caption] The caption 'Accumulated SAN baseline improvements' is not defined; please specify how the accumulated gain is computed (e.g., sum over the four datasets).
  5. [IV-B] The statement 'PAT with EVACLIP demonstrates competitive performances compared to the current state-of-the-art (SOTA)' is too vague; please specify which SOTA methods and on which datasets PAT is competitive.
  6. [III-B] The fragment 'The Up/DownSampler is DySample [39] and convolution for scaling features' should be integrated into a full sentence.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: PAT's segmentation gains are measured on held-out datasets after COCO-Stuff training, and the token semantics are learned under explicit supervision rather than derived from a fitted parameter or self-citation.

full rationale

The paper's central claims are empirical and self-contained. PAT is trained on COCO-Stuff and evaluated on Pascal Context and ADE20K variants, so the reported mIoU improvements over the SAN baseline are not fitted to the test benchmarks. The semantic structure of the learned codebook tokens is a consequence of training with both reconstruction and segmentation losses, not a quantity that is predicted from a parameter fitted to the same data. The 'progressive semantics' assumption about the pretrained VLM feature pyramid is motivated by external citations ([14]-[16]) and is treated as a design hypothesis, with ablations showing sensitivity to the pyramid scales and vMF clustering; this assumption is not established by direct measurement, but that is an evidence-strength limitation rather than a circular derivation. There is no load-bearing self-citation chain: the baseline SAN and the HSAttn module are external prior works, and no uniqueness theorem from the authors is invoked to force the design choice. The qualitative claim that PAT 'enhances the semantic intuition of the VLM feature pyramid' is partially self-fulfilling because the tokens are supervised by segmentation labels, but the paper does not present this as a prediction derived from an input; it presents it as an observed property of the trained model. Overall, the derivation chain does not reduce to its inputs by construction.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The paper's central claim rests on hand-chosen hyperparameters (codebook sizes, loss weights) and on the assumption that pretrained VLM features form a semantically composable pyramid. No new physical or conceptual entities beyond learned token representations are introduced.

free parameters (4)
  • Stage-wise codebook sizes = 128, 64, 32, 256 for latent/mid/late stages
    Chosen by hand in Section IV-A; the size controls the granularity of semantic tokens and is not derived from data.
  • Codebook token dimension = 32
    Set to 32 for all codebooks; a design choice that affects reconstruction and segmentation quality.
  • Loss weights = 0.1, 0.1, 1.0 for VQ, spatial alignment, and reconstruction losses
    Empirically chosen; the reconstruction loss dominates. No sensitivity analysis is provided.
  • Training iterations = 120K
    Standard choice; no early stopping or convergence analysis shown.
assumptions (3)
  • domain assumption Pretrained VLM feature pyramid exhibits progressive composition from low-level patterns to high-level semantics
    Invoked in the introduction and Section III-A to justify tokenizing the pyramid. The paper cites prior studies [14]-[16] for this property.
  • standard math HSAttn with a vMF prior performs vMF meanshift clustering of features
    Used in Eq. 1 and Section III-A, based on the mean-shift mask transformer [19]. The paper relies on this equivalence to argue that VQ codebooks produce meta-semantic clusters.
  • domain assumption Joint training of pixel reconstruction and semantic segmentation with a shared decoder benefits both tasks
    Ablation in Table II supports this empirically, but the mechanism is not derived; it is a learned assumption about optimization.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Incorporating Feature Pyramid Tokenization and Open Vocabulary Semantic Segmentation." pith.science (2026). https://pith.science/paper/TIA5MDFC

@misc{pith2026241214145,
  author       = {Pith},
  title        = {Pith review of: Incorporating Feature Pyramid Tokenization and Open Vocabulary Semantic Segmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TIA5MDFC}},
  note         = {Machine review of arXiv:2412.14145}
}
read the original abstract

The visual understanding are often approached from 3 granular levels: image, patch and pixel. Visual Tokenization, trained by self-supervised reconstructive learning, compresses visual data by codebook in patch-level with marginal information loss, but the visual tokens does not have semantic meaning. Open Vocabulary semantic segmentation benefits from the evolving Vision-Language models (VLMs) with strong image zero-shot capability, but transferring image-level to pixel-level understanding remains an imminent challenge. In this paper, we treat segmentation as tokenizing pixels and study a united perceptual and semantic token compression for all granular understanding and consequently facilitate open vocabulary semantic segmentation. Referring to the cognitive process of pretrained VLM where the low-level features are progressively composed to high-level semantics, we propose Feature Pyramid Tokenization (PAT) to cluster and represent multi-resolution feature by learnable codebooks and then decode them by joint learning pixel reconstruction and semantic segmentation. We design loosely coupled pixel and semantic learning branches. The pixel branch simulates bottom-up composition and top-down visualization of codebook tokens, while the semantic branch collectively fuse hierarchical codebooks as auxiliary segmentation guidance. Our experiments show that PAT enhances the semantic intuition of VLM feature pyramid, improves performance over the baseline segmentation model and achieves competitive performance on open vocabulary semantic segmentation benchmark. Our model is parameter-efficient for VLM integration and flexible for the independent tokenization. We hope to give inspiration not only on improving segmentation but also on semantic visual token utilization.

Figures

Figures reproduced from arXiv: 2412.14145 by the authors.

Figure 1
Figure 1. Open Vocabulary Segmentation example. After learning pyramid tok [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. PAT concept. The image-level and pixel-level understanding gap [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. PAT architecture. The global tokens from [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Mid stage PAT VQ module. The module decouples the semantic [PITH_FULL_IMAGE:figures/full_fig_p003_4.png]
Figure 5
Figure 5. Figure 5: Accumulated SAN baseline improvements using different VLM. [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 6
Figure 6. Figure 6: Feature Clustering versus Tokenization. between image- and pixel-level understanding. With decou￾pled learning of perception and semantic compression, the contradictory encoding behavior between them is alleviated. The semantic-rich token representation thus enables pa…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 25 canonical work pages

  1. [1]

    Learning transferable visual models from natural lan- guage supervision,

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al., “Learning transferable visual models from natural lan- guage supervision,” in International Conference on Machine Learning . PMLR, 2021, pp. 8748–8763

    work page 2021

  2. [2]

    Scaling open- vocabulary image segmentation with image-level labels,

    Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Yi Lin, “Scaling open- vocabulary image segmentation with image-level labels,” in European Conference on Computer Vision . Springer, 2022, pp. 540–557

    work page 2022

  3. [3]

    Maskclip: Masked self-distillation advances contrastive language- image pretraining,

    Xiaoyi Dong, Jianmin Bao, Yinglin Zheng, Ting Zhang, Dongdong Chen, Hao Yang, Ming Zeng, Weiming Zhang, Lu Yuan, Dong Chen, et al., “Maskclip: Masked self-distillation advances contrastive language- image pretraining,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 10995–11005

    work page 2023

  4. [4]

    A simple framework for text-supervised semantic seg- mentation,

    Muyang Yi, Quan Cui, Hao Wu, Cheng Yang, Osamu Yoshie, and Hongtao Lu, “A simple framework for text-supervised semantic seg- mentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 7071–7080

    work page 2023

  5. [5]

    Open-vocabulary semantic segmentation with mask-adapted clip,

    Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu, “Open-vocabulary semantic segmentation with mask-adapted clip,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 7061–7070

    work page 2023

  6. [6]

    Segment anything,

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al., “Segment anything,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 4015–4026

    2023

  7. [7]

    Open-vocabulary sam: Segment and recognize twenty-thousand classes interactively,

    Haobo Yuan, Xiangtai Li, Chong Zhou, Yining Li, Kai Chen, and Chen Change Loy, “Open-vocabulary sam: Segment and recognize twenty-thousand classes interactively,”arXiv preprint arXiv:2401.02955, 2024

    arXiv 2024

  8. [8]

    Possam: Panoptic open- vocabulary segment anything,

    Vibashan VS, Shubhankar Borse, Hyojin Park, Debasmit Das, Vishal Patel, Munawar Hayat, and Fatih Porikli, “Possam: Panoptic open- vocabulary segment anything,” arXiv preprint arXiv:2403.09620 , 2024

    arXiv 2024

Show all 47 references
  1. [9]

    Diffusion models for zero-shot open-vocabulary segmentation,

    Laurynas Karazija, Iro Laina, Andrea Vedaldi, and Christian Rupprecht, “Diffusion models for zero-shot open-vocabulary segmentation,” arXiv preprint arXiv:2306.09316, 2023

    2023 arXiv

  2. [10]

    Open-vocabulary panoptic segmentation with text-to- image diffusion models,

    Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello, “Open-vocabulary panoptic segmentation with text-to- image diffusion models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 2955–2966

    2023

  3. [11]

    Cat- seg: Cost aggregation for open-vocabulary semantic segmentation,

    Seokju Cho, Heeseong Shin, Sunghwan Hong, Seungjun An, Seungjun Lee, Anurag Arnab, Paul Hongsuck Seo, and Seungryong Kim, “Cat- seg: Cost aggregation for open-vocabulary semantic segmentation,” arXiv preprint arXiv:2303.11797 , 2023

    2023 arXiv

  4. [12]

    Clipself: Vision transformer dis- tills itself for open-vocabulary dense prediction,

    Size Wu, Wenwei Zhang, Lumin Xu, Sheng Jin, Xiangtai Li, Wen- tao Liu, and Chen Change Loy, “Clipself: Vision transformer dis- tills itself for open-vocabulary dense prediction,” arXiv preprint arXiv:2310.01403, 2023

    2023 arXiv

  5. [13]

    Clip-dinoiser: Teaching clip a few dino tricks,

    Monika Wysocza ´nska, Oriane Sim ´eoni, Micha¨el Ramamonjisoa, Andrei Bursuc, Tomasz Trzci ´nski, and Patrick P ´erez, “Clip-dinoiser: Teaching clip a few dino tricks,” arXiv preprint arXiv:2312.12359 , 2023

    2023 arXiv

  6. [14]

    What do vision transformers learn? a visual exploration,

    Amin Ghiasi, Hamid Kazemi, Eitan Borgnia, Steven Reich, Manli Shu, Micah Goldblum, Andrew Gordon Wilson, and Tom Goldstein, “What do vision transformers learn? a visual exploration,” arXiv preprint arXiv:2212.06727, 2022

    2022 arXiv

  7. [15]

    Deep inside convolutional networks: Visualising image classification models and saliency maps,

    Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman, “Deep inside convolutional networks: Visualising image classification models and saliency maps,” arXiv preprint arXiv:1312.6034 , 2013

    2013 arXiv

  8. [16]

    Visualizing and understanding con- volutional networks,

    Matthew D Zeiler and Rob Fergus, “Visualizing and understanding con- volutional networks,” in Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13. Springer, 2014, pp. 818–833

    2014

  9. [17]

    Masked-attention mask transformer for universal image segmentation,

    Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar, “Masked-attention mask transformer for universal image segmentation,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2022, pp. 1290–1299

    2022

  10. [18]

    kmax-deeplab: k-means mask transformer,

    Qihang Yu, Huiyu Wang, Siyuan Qiao, Maxwell Collins, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen, “kmax-deeplab: k-means mask transformer,” arXiv preprint arXiv:2207.04044 , 2022

    2022 arXiv

  11. [19]

    Mean shift mask transformer for unseen object instance segmentation,

    Yangxiao Lu, Yuqiao Chen, Nicholas Ruozzi, and Yu Xiang, “Mean shift mask transformer for unseen object instance segmentation,” arXiv preprint arXiv:2211.11679, 2022

    2022 arXiv

  12. [20]

    Feature pyramid networks for object detection,

    Tsung-Yi Lin, Piotr Doll ´ar, Ross Girshick, Kaiming He, Bharath Har- iharan, and Serge Belongie, “Feature pyramid networks for object detection,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2017, pp. 2117–2125

    2017

  13. [21]

    U-net: Con- volutional networks for biomedical image segmentation,

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox, “U-net: Con- volutional networks for biomedical image segmentation,” in Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, P...

    2015

  14. [22]

    Side adapter network for open-vocabulary semantic segmentation,

    Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xiang Bai, “Side adapter network for open-vocabulary semantic segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 2945–2954

    2023

  15. [23]

    Convolutions die hard: Open-vocabulary segmentation with single frozen convolutional clip,

    Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang-Chieh Chen, “Convolutions die hard: Open-vocabulary segmentation with single frozen convolutional clip,” Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  16. [24]

    Sed: A simple encoder-decoder for open-vocabulary semantic segmen- tation,

    Bin Xie, Jiale Cao, Jin Xie, Fahad Shahbaz Khan, and Yanwei Pang, “Sed: A simple encoder-decoder for open-vocabulary semantic segmen- tation,” arXiv preprint arXiv:2311.15537 , 2023

    2023 arXiv

  17. [25]

    Neural discrete representa- tion learning,

    Aaron Van Den Oord, Oriol Vinyals, et al., “Neural discrete representa- tion learning,” Advances in neural information processing systems , vol. 30, 2017

    2017

  18. [26]

    Taming transformers for high-resolution image synthesis,

    Patrick Esser, Robin Rombach, and Bjorn Ommer, “Taming transformers for high-resolution image synthesis,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021, pp. 12873–12883

    2021

  19. [27]

    Vector-quantized image modeling with improved vqgan,

    Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu, “Vector-quantized image modeling with improved vqgan,” arXiv preprint arXiv:2110.04627, 2021

    2021 arXiv

  20. [28]

    High-resolution image synthesis with latent diffu- sion models,

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer, “High-resolution image synthesis with latent diffu- sion models,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2022, pp. 10684–10695

    2022

  21. [29]

    Image as a foreign language: Beit pretraining for all vision and vision-language tasks,

    Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al., “Image as a foreign language: Beit pretraining for all vision and vision-language tasks,” arXiv preprint arXiv:2208.10442, 2022

    2022 arXiv

  22. [30]

    Coco-stuff: Thing and stuff classes in context,

    Holger Caesar, Jasper Uijlings, and Vittorio Ferrari, “Coco-stuff: Thing and stuff classes in context,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2018, pp. 1209–1218

    2018

  23. [31]

    The role of context for object detection and semantic segmentation in the wild,

    Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong- Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille, “The role of context for object detection and semantic segmentation in the wild,” in Proceedings of the IEEE conference on computer vision and pattern reco...

    2014

  24. [32]

    Scene parsing through ade20k dataset,

    Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba, “Scene parsing through ade20k dataset,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 633–641

    2017

  25. [33]

    Peco: Perceptual codebook for bert pre-training of vision transformers,

    Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu, “Peco: Perceptual codebook for bert pre-training of vision transformers,” arXiv preprint arXiv:2111.12710, 2021

    2021 arXiv

  26. [34]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi, “Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,” in International conference on machine learning. PMLR, 2023, pp. 19730–19742

    2023

  27. [35]

    Spae: Semantic pyramid autoencoder for multi- modal generation with frozen llms,

    Lijun Yu, Yong Cheng, Zhiruo Wang, Vivek Kumar, Wolfgang Macherey, Yanping Huang, David Ross, Irfan Essa, Yonatan Bisk, Ming- Hsuan Yang, et al., “Spae: Semantic pyramid autoencoder for multi- modal generation with frozen llms,” Advances in Neural Information Processing System...

    2024

  28. [36]

    A simple baseline for open-vocabulary semantic segmentation with pre-trained vision-language model,

    Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai, “A simple baseline for open-vocabulary semantic segmentation with pre-trained vision-language model,” in European Conference on Computer Vision . Springer, 2022, pp. 736–753

    2022

  29. [37]

    Denoising vision transformers,

    Jiawei Yang, Katie Z Luo, Jiefeng Li, Kilian Q Weinberger, Yonglong Tian, and Yue Wang, “Denoising vision transformers,” arXiv preprint arXiv:2401.02957, 2024

    2024 arXiv

  30. [38]

    Featup: A model-agnostic framework for features at any resolution,

    Stephanie Fu, Mark Hamilton, Laura Brandt, Axel Feldman, Zhoutong Zhang, and William T Freeman, “Featup: A model-agnostic framework for features at any resolution,” arXiv preprint arXiv:2403.10516 , 2024

    2024 arXiv

  31. [39]

    Learning to upsample by learning to sample,

    Wenze Liu, Hao Lu, Hongtao Fu, and Zhiguo Cao, “Learning to upsample by learning to sample,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 6027–6037

    2023

  32. [40]

    Unlocking pre-trained image backbones for semantic image synthesis,

    Tariq Berrada, Jakob Verbeek, Camille Couprie, and Karteek Alahari, “Unlocking pre-trained image backbones for semantic image synthesis,” arXiv preprint arXiv:2312.13314 , 2023

    2023 arXiv

  33. [41]

    Semantic image synthesis with spatially-adaptive normalization,

    Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu, “Semantic image synthesis with spatially-adaptive normalization,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 2337–2346

    2019

  34. [42]

    Perceptual losses for real-time style transfer and super-resolution,

    Justin Johnson, Alexandre Alahi, and Li Fei-Fei, “Perceptual losses for real-time style transfer and super-resolution,” in European Conference on Computer Vision . Springer, 2016, pp. 694–711

    2016

  35. [43]

    Microsoft coco captions: Data collection and evaluation server,

    Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Doll ´ar, and C Lawrence Zitnick, “Microsoft coco captions: Data collection and evaluation server,”arXiv preprint arXiv:1504.00325, 2015

    2015 arXiv

  36. [44]

    Microsoft coco: Common objects in context,

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Per- ona, Deva Ramanan, Piotr Doll ´ar, and C Lawrence Zitnick, “Microsoft coco: Common objects in context,” in European conference on computer vision. Springer, 2014, pp. 740–755

    2014

  37. [45]

    Eva- clip: Improved training techniques for clip at scale,

    Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao, “Eva- clip: Improved training techniques for clip at scale,” arXiv preprint arXiv:2303.15389, 2023

    2023 arXiv

  38. [46]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium,

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter, “Gans trained by a two time-scale update rule converge to a local nash equilibrium,” in Advances in Neural Information Processing Systems, I. Guyon, U. V on Luxburg, S. Bengio, H. Wallac...

    2017

  39. [47]

    Visual autoregressive modeling: Scalable image generation via next- scale prediction,

    Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang, “Visual autoregressive modeling: Scalable image generation via next- scale prediction,” arXiv preprint arXiv:2404.02905 , 2024

    2024 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.