Pith. sign in

REVIEW 4 major objections 6 minor 68 references

LlamaSeg: Image Segmentation via Autoregressive Mask Generation

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read By reformulating segmentation as visual generation, LlamaSeg predicts masks as visual tokens and outperforms existing generative approaches on the paper's benchmarks.

desk verdict Solid, incremental autoregressive segmentation paper with a genuinely useful dataset; the headline 'finer masks' claim is not established because the mAHD comparison is selection-biased. read the letter →

arxiv 2505.19422 v2 pith:U3XBPZDY submitted 2025-05-26 cs.CV

Jiru Deng , Tengjin Weng , Tianyu Yang , Wenhan Luo , Zhiheng Li , Wenhao Jiang This is my paper · ORCID
classification cs.CV
keywords imagesegmentationautoregressivegenerationvisualtokensVQGANopen-vocabularyreferringexpressionSA-OVRSdatasetaverageHausdorffdistance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that image segmentation can be treated as a visual generation problem instead of a classification or coordinate-prediction problem. It encodes a mask as a compact sequence of discrete visual tokens from a VQGAN tokenizer and trains a LLaMA-style transformer to generate those tokens one by one, conditioned on an image and a language instruction. The same trained model handles semantic, open-vocabulary, and referring-expression segmentation. To support large-scale training, the paper builds a 2M-mask dataset, SA-OVRS, with over 5,800 labels and textual descriptions, and proposes a contour-fidelity metric combining IoU with average Hausdorff distance. If the claims hold, autoregressive next-token prediction, the recipe behind large language models, can carry fine-grained visual output tasks without separate expert segmentation modules.

What carries the argument

The load-bearing mechanism is the mask tokenizer plus autoregressive decoder. A VQGAN with downsampling rate 16 and a codebook of discrete vectors converts a black-and-white mask into a 2D grid of code indices, which are flattened into a 1D sequence. The transformer predicts the next code index given the image tokens, text tokens, and previously generated mask tokens; at inference, greedy decoding over the codebook is followed by a binarization of the reconstructed mask. Learnable begin-of-image and begin-of-mask separators mark modality boundaries, and the from-scratch variant uses a frozen vision-language encoder whose patch size matches the tokenizer's downsampling rate so image tokens and mask tokens stay spatially aligned. The new dataset pipeline, built from a captioning model that proposes labels, a grounding detector that matches labels to masks via bounding-box IoU, and cross-verification of referring expressions, supplies the training signal the paper credits for its gains.

What would settle it

Take a random sample of, say, 1,000 SA-OVRS mask-label pairs and have human annotators judge whether the label truly describes the highlighted mask. If the mismatch rate is high, or if removing disputed pairs eliminates the reported +1.6 mIoU and +4.9 cIoU pre-training gains, the central claim that the dataset teaches accurate open-vocabulary segmentation would be undermined.

Watch

Extended reading notes

Core claim

The central claim is that masks can be serialized into visual tokens and predicted directly by a causal transformer, making segmentation a first-class generative task. LlamaSeg treats a binary mask as an RGB image, encodes it with a frozen VQGAN into code indices, and trains a LLaMA-style model to predict these indices using only mask-token cross-entropy loss. The paper reports that this approach, at 0.77B-1.5B parameters, outperforms existing generative segmentation baselines on ADE20K, COCO-Stuff, PAS-20, and RefCOCO-family benchmarks, and that its masks align with ground-truth contours substantially better as measured by the proposed mAHD metric. It also introduces SA-OVRS, a dataset of 1.93 million masks with open-vocabulary labels and referring/reasoning expressions, and shows that pre-training on it boosts downstream performance in its ablations.

Load-bearing premise

The automated SA-OVRS pipeline, with no human audit, produces mask-label pairs that are correct enough to teach open-vocabulary segmentation; if many pairs are wrong, the reported pre-training gains and open-vocabulary abilities rest on noisy supervision.

Editorial extensions

If this is right

  • Segmentation can be unified with other autoregressive multimodal tasks under one next-token prediction paradigm, removing the need for separate expert segmentation modules.
  • The SA-OVRS dataset, with 1.93M masks and over 5,800 labels, becomes a reusable resource for training and evaluating open-vocabulary and referring segmentation models.
  • The proposed mAHD metric, computed within IoU threshold groups, offers a more informative way to compare contour fidelity for generative mask models.
  • Higher input resolution and larger model scale improve fine-grained mask quality, with the reported best results coming from the 1.5B model at 384-pixel resolution.
  • Pre-training on SA-OVRS is directly beneficial in the paper's ablations: removing it lowers ADE20K mIoU by 1.6 and refCOCO cIoU by 4.9.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The use of a frozen image tokenizer trained on natural images suggests that generic visual codebooks already preserve enough contour detail for mask generation; a dedicated mask tokenizer might shift the reconstruction-quality trade-off, which could be tested directly.
  • Because the SA-OVRS pipeline is fully automated with no human audit, the reported open-vocabulary gains may stem partly from dataset scale and diversity rather than from label accuracy; a random human audit of mask-label pairs would separate these factors.
  • The LoRA-fine-tuned 1B variant points toward a single model that both understands images and segments them without pipeline components, which could be extended to other dense prediction tasks such as depth or edge estimation.
  • The mAHD-with-thresholds evaluation could generalize beyond this paper to any generative model that outputs masks, making contour fidelity a standard reporting axis rather than an auxiliary metric.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes LlamaSeg, an autoregressive image segmentation framework that encodes segmentation masks as VQGAN visual tokens and generates them via a LLaMA-style transformer conditioned on image and text tokens. It introduces SA-OVRS, a 2M-mask dataset built from SA-1B masks with labels from Qwen2-VL-72B, detections from GroundingDINO, and a two-stage text generation/verification pipeline. Experiments report gains over Unified-IO and Unified-IO2 on ADE20K, COCO-Stuff, PAS-20, and RefCOCO-series benchmarks, and propose a mean Average Hausdorff Distance (mAHD) metric for contour fidelity. Code and dataset are promised.

Significance. If the reported results hold, the paper makes a useful contribution: it demonstrates that a relatively compact autoregressive model can perform semantic and referring segmentation without external segmentation modules, and the SA-OVRS dataset could be a reusable open-vocabulary segmentation resource. The mIoU and cIoU gains in Tables 1 and 2 are internally consistent and use standard benchmark protocols. However, the distinctive claim of 'finer, more accurate masks' rests entirely on a mAHD evaluation that is currently vulnerable to selection bias, and the SA-OVRS dataset lacks validation of label correctness. With a strengthened evaluation and dataset audit, the core idea is promising.

major comments (4)
  1. [§5.2, Table 3] The mAHD protocol computes per-model, per-IoU-threshold subsets without reporting coverage counts or sample sizes. Because the qualifying prediction set is defined independently for each model, a model that rarely reaches high IoU is scored only on its easiest masks, while a model with many high-IoU predictions is measured on a larger, harder set. This makes the large margins (e.g., 26.61 vs 76.35 on ADE20K at IoU-0.5) uninterpretable as evidence of finer contours. The instability of Unified-IO-XL on RefCOCO (with values 72.58, 71.69, 71.16, 69.99 then 85.65 at IoU-0.9) strongly suggests very small and uneven subsamples. Please report the number of qualifying predictions per cell or the coverage rate, and re-evaluate on matched subsets or with a matched-IoU distribution; without this, the abstract's 'finer, more accurate segmentation masks' claim is not established.
  2. [Eq. (3) and Abstract/Introduction] The metric d_AHD in Eq. (3) is the standard mean average Hausdorff distance between two boundary point sets. It is not a 'composite metric that combines IoU and AHD' as claimed in the abstract and contribution list, because the IoU threshold is used only as a selection filter before computing AHD, not as a component of the score. Please either rename and describe the metric accurately or develop a true composite scalar that integrates IoU and AHD in a single value.
  3. [§4.1, §4.2, Table 6] The SA-OVRS annotation pipeline relies on Qwen2-VL-72B-generated labels and GroundingDINO detections with ad hoc thresholds (IoU >0.9 for multi-box matches, IoU >0.85 for single-box, confidence >0.3, a four-box cap, and nested-box IoU >0.97), and no human audit or error-rate estimate is reported for the resulting 1.93M pairs. The paper itself notes in Section 4 that pipelines of this kind produce hierarchical inconsistencies such as 'vegetables' vs 'broccoli', yet the open-vocabulary capability claims and the +1.6 mIoU / +4.9 cIoU gains attributed to SA-OVRS pre-training in Table 6 depend on these pairs being correct. Please provide a validation study, such as a human evaluation on a random subset, per-category error analysis, or agreement statistics, to establish that the dataset is sufficiently reliable for the claims made.
  4. [§5.2, Tables 1-2] The experimental comparison is restricted to Unified-IO and Unified-IO2 (plus LaSagnA in Table 1). The claim that LlamaSeg 'consistently outperforms existing generative approaches' is broader than what these two baselines support; other autoregressive or generative segmentation models (e.g., GIT, All in Tokens, Text4Seg, or more recent generalist models) are neither compared nor discussed. Please either expand the comparison to a representative set of generative segmentation models or carefully scope the claim to the evaluated baselines and acknowledge the limitation.
minor comments (6)
  1. [Eq. (3)] The notation '1/X' and '1/Y' is ambiguous; please use |X| and |Y| (or explicit cardinality) for the reciprocal of the number of boundary points.
  2. [§5.1] The sentence 'We AdamW optimizer with a learning rate...' is missing a verb; it should read 'We use the AdamW optimizer...'.
  3. [Abstract] The word 'composite metric' is used for what is currently a thresholded AHD rather than a composite score; please align the terminology with the actual definition in Eq. (3).
  4. [§4.1] Qwen2-VL-72B is described as a 'captioning model' but is also used for label generation and cross-verification; please clarify the exact role and prompt used at each stage.
  5. [Table 2] Several cells are marked with '-' (e.g., Unified-IO-Large on RefCOCO testB); please state in the caption or text whether these results were not evaluated or not available from the released checkpoints.
  6. [§5.3, Table 4] The ablation shows that further fine-tuning the VQGAN tokenizer degrades reconstruction quality, but the paper does not explain why; a brief analysis (e.g., overfitting or representation drift) would help the reader interpret the result.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: all load-bearing evaluations use external benchmarks and the dataset ablation is a genuine controlled comparison.

full rationale

I walked the claimed derivation chain and found no step in which a prediction reduces by construction to a fitted input or to a self-citation. The central claim (better masks than Unified-IO/Unified-IO2) is supported by mIoU/cIoU tables computed on external benchmark test sets after standard fine-tuning, and by the mAHD boundary-distance metric defined in Eq. 3, which is applied unchanged to all compared models. SA-OVRS pre-training is ablated in Table 6 against a no-SA-OVRS baseline, so its contribution is not assumed by construction. The mAHD evaluation conditions on per-model IoU-threshold subsets and the paper reports no coverage counts; that is a selection-bias / validity concern, not a circularity, because the conditioning rule is fixed in advance and identical across models. Likewise, calling Eq. 3 a 'composite' metric overstates what is a standard mean boundary distance used at multiple thresholds; this is an overclaim, not a circular reduction. No load-bearing self-citations appear: the reference list contains no papers by the present authors, and the external tokenizer/MLLM components (VQGAN, Janus Pro) are used as fixed pretrained building blocks rather than as justification for the paper's own predictions.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The system rests on two inherited artifacts it does not verify (the VQGAN codebook and SA-1B masks), plus hand-tuned data-pipeline thresholds that gate the entire SA-OVRS training set, plus the author-defined mAHD protocol that backs the headline contour-fidelity claim. Table 4 is the only direct validation of the tokenizer assumption. No error-rate audit exists for the annotation pipeline, and the mAHD thresholds are unexamined.

free parameters (4)
  • VQGAN mask tokenizer (LlamaGen checkpoints) = pre-trained on natural images; codebook size not stated in the paper
    Central to the claim that masks can be serialized into discrete tokens; reconstruction fidelity (Table 4, frozen VQGAN IoU 96.8) is inherited, not trained by the authors.
  • SA-OVRS matching thresholds = IoU > 0.9 (multi-box), IoU > 0.85 (single-box), confidence > 0.3, nested IoU > 0.97, max 10 labels, 4-box cap
    Hand-chosen numbers in Section 4.1 control which 1.93M mask-label pairs enter the dataset; no sensitivity analysis or human validation is given.
  • mAHD IoU thresholds = [0.5, 0.6, 0.7, 0.8, 0.9]
    Author-defined thresholds gate which predictions are scored for contour distance (Section 5.1); no justification for the set is provided.
  • Training protocol hyperparameters = LR 2e-4 / 1e-4, 4 / 10 / 20 epochs, batch size 128 for tokenizer ablation
    Standard choices but single-run; some interact with reported gains, e.g., the 384-pixel variant adds 3.9 mIoU on ADE20K.
assumptions (4)
  • domain assumption Masks are representable as special RGB images, and a VQGAN trained on natural images preserves their contours.
    Section 3.2; the entire method depends on this, partially supported by Table 4 (frozen VQGAN reconstruction IoU 96.8, mAHD 6.1).
  • domain assumption SigLIP2 patch-16 visual tokens are spatially aligned with 16x-downsampled mask tokens.
    Section 3.3; the paper asserts 'token-level alignment' and shows one attention map (Fig. 5) as evidence.
  • domain assumption Cross-entropy over mask tokens with greedy decoding is an appropriate training and inference objective for masks.
    Supported by the decoding ablation (Table 5, greedy search best); treated as a modeling choice rather than derived.
  • domain assumption Qwen2-VL-72B and GroundingDINO outputs used to build SA-OVRS are accurate enough without human quality control at scale.
    Sections 4.1-4.2; cross-verification exists for referring expressions, but no error-rate audit for labels or mask-label pairings is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LlamaSeg: Image Segmentation via Autoregressive Mask Generation." pith.science (2026). https://pith.science/paper/U3XBPZDY

@misc{pith2026250519422,
  author       = {Pith},
  title        = {Pith review of: LlamaSeg: Image Segmentation via Autoregressive Mask Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U3XBPZDY}},
  note         = {Machine review of arXiv:2505.19422}
}
abstract

We present \textbf{LlamaSeg}, a visual autoregressive framework that unifies multiple image segmentation tasks via natural language instructions. By reformulating segmentation as visual generation, LlamaSeg encodes masks as visual tokens and uses a LLaMA-style Transformer for direct next-token prediction, naturally fitting segmentation into autoregressive architectures. To support large-scale training, we introduce a data annotation pipeline and construct the \textbf{SA-OVRS} dataset, which contains \textbf{2M} segmentation masks annotated with over \textbf{5,800} open vocabulary labels or diverse textual descriptions, spanning diverse real-world scenarios. This enables our model to localize objects in images based on text prompts and to generate fine-grained masks. We further introduce the composite metric average Hausdorff Distance ($d_{\mathrm{AHD}}$) to evaluate mask contour fidelity for generative models better. Experiments show that LlamaSeg consistently outperforms existing generative approaches on multiple segmentation benchmarks and delivers finer, more accurate segmentation masks. Code and dataset are available at \href{https://github.com/GML-FMGroup/llamaseg}{https://github.com/GML-FMGroup/llamaseg}.

Figures

Figures reproduced from arXiv: 2505.19422 by the authors.

Figure 1
Figure 1. Different implementations of image segmentation in autoregressive frameworks. (a) Embedding as mask. (b) Coordinates as mask. (c) Text as mask. (d) Visual tokens as mask (ours). On the other hand, the success of autoregressive image generation [16–20] further substantiates the potential of autoregressive frameworks for vision modeling. These approaches typically guide image generation through category labels, textua… view at source ↗
Figure 2
Figure 2. Overall framework of LlamaSeg. Our model comprises two components: a mask tokenizer and an autoregressive model. GT labels are only used as supervision during training. them to produce the segmentation mask. For the autoregressive model utilizing a pretrained MLLM, the text encoder is omitted, while the remaining procedure remains unchanged. 3.2 Mask Tokenizer We use an image tokenizer to transform segmentation mask… view at source ↗
Figure 3
Figure 3. Our two-stage data generation pipeline. The first stage requires generating labels and matching them with segmentation masks, and then identifying discrete entities. The second stage generates textual data based on the labels. 4.1 Label Data For a given image, we aim to collect object labels and associate them with corresponding masks. Qwen2-VL-72B [34] is employed as the captioning model. A carefully designed promp… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Visual comparison between our model and existing visual generative models [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Attention heatmap of mask tokens in the last self-attention layer of the autoregressive model. Scores have been log-transformed to enhance visual contrast. 2D spatial relationship. To examine whether the model can learn 2D spatial relationships from 1D tokens, we selec…
Figure 6
Figure 6. Figure 6: Comparison between the segmentation masks and reconstructed masks. 9 [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

68 extracted references · 22 canonical work pages

  1. [1]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems , 35:23716–23736, 2022

    2022

  2. [2]

    InstructBLIP: Towards general-purpose vision-language models with instruction tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards general-purpose vision-language models with instruction tuning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023

    work page 2023

  3. [3]

    Emu: Generative pretraining in multimodality

    Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Emu: Generative pretraining in multimodality. arXiv preprint arXiv:2307.05222, 2023

    arXiv 2023

  4. [4]

    Visionllm: Large language model is also an open-ended decoder for vision-centric tasks

    Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. Advances in Neural Information Processing Systems , 36:61501–61513, 2023

    2023

  5. [5]

    u-llava: Unifying multi-modal tasks via large language model

    Jinjin Xu, Liwu Xu, Yuzhe Yang, Xiang Li, Fanyi Wang, Yanchun Xie, Yi-Jie Huang, and Yaqian Li. u-llava: Unifying multi-modal tasks via large language model. In ECAI 2024, pages 618–625. IOS Press, 2024

    work page 2024

  6. [6]

    Omg-llava: Bridging image-level, object-level, pixel-level reasoning and understanding

    Tao Zhang, Xiangtai Li, Hao Fei, Haobo Yuan, Shengqiong Wu, Shunping Ji, Chen Change Loy, and Shuicheng Yan. Omg-llava: Bridging image-level, object-level, pixel-level reasoning and understanding. Advances in Neural Information Processing Systems , 37:71737–71767, 2024. 10

    work page 2024

  7. [7]

    Visionllm v2: An end-to-end generalist multimodal large language model for hundreds of vision-language tasks

    Jiannan Wu, Muyan Zhong, Sen Xing, Zeqiang Lai, Zhaoyang Liu, Zhe Chen, Wenhai Wang, Xizhou Zhu, Lewei Lu, Tong Lu, et al. Visionllm v2: An end-to-end generalist multimodal large language model for hundreds of vision-language tasks. Advances in Neural Information Processing Systems, 37:69925–69975, 2024

    2024

  8. [8]

    Lisa: Reasoning segmentation via large language model

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9579–9589, 2024

    2024

Show all 68 references
  1. [9]

    Glamm: Pixel grounding large multimodal model

    Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S Khan. Glamm: Pixel grounding large multimodal model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Re...

    2024

  2. [10]

    Text4seg: Reimagining image segmentation as text generation

    Mengcheng Lan, Chaofeng Chen, Yue Zhou, Jiaxing Xu, Yiping Ke, Xinjiang Wang, Litong Feng, and Wayne Zhang. Text4seg: Reimagining image segmentation as text generation. arXiv preprint arXiv:2410.09855, 2024

    2024 arXiv

  3. [11]

    Git: Towards generalist vision transformer through universal language interface

    Haiyang Wang, Hao Tang, Li Jiang, Shaoshuai Shi, Muhammad Ferjad Naeem, Hongsheng Li, Bernt Schiele, and Liwei Wang. Git: Towards generalist vision transformer through universal language interface. In European Conference on Computer Vision, pages 55–73. Springer, 2024

    2024

  4. [12]

    Polyformer: Referring image segmentation as sequential polygon generation

    Jiang Liu, Hui Ding, Zhaowei Cai, Yuting Zhang, Ravi Kumar Satzoda, Vijay Mahadevan, and R Manmatha. Polyformer: Referring image segmentation as sequential polygon generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 18653–18...

    2023

  5. [13]

    Jack of all tasks master of many: Design- ing general-purpose coarse-to-fine vision-language model

    Shraman Pramanick, Guangxing Han, Rui Hou, Sayan Nag, Ser-Nam Lim, Nicolas Ballas, Qifan Wang, Rama Chellappa, and Amjad Almahairi. Jack of all tasks master of many: Design- ing general-purpose coarse-to-fine vision-language model. In Proceedings of the IEEE/CVF Conference on ...

    2024

  6. [14]

    Gsva: Generalized segmentation via multimodal large language models

    Zhuofan Xia, Dongchen Han, Yizeng Han, Xuran Pan, Shiji Song, and Gao Huang. Gsva: Generalized segmentation via multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3858–3869, 2024

    2024

  7. [15]

    Segment anything

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision , pages 4015–4026, 2023

    2023

  8. [16]

    Taming transformers for high-resolution image synthesis

    Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873–12883, 2021

    2021

  9. [17]

    Zero-shot text-to-image generation

    Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea V oss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International conference on machine learning, pages 8821–8831. Pmlr, 2021

    2021

  10. [18]

    Generating diverse high-fidelity images with vq-vae-2

    Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems , 32, 2019

    2019

  11. [19]

    Visual autoregressive modeling: Scalable image generation via next-scale prediction

    Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. Advances in neural information processing systems, 37:84839–84865, 2024

    2024

  12. [20]

    Autoregressive model beats diffusion: Llama for scalable image generation

    Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06525, 2024

    2024 arXiv

  13. [21]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 11

    2023 arXiv

  14. [22]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  15. [23]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , p...

    2021

  16. [24]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  17. [25]

    Improving language understanding by generative pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018

    2018

  18. [26]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877–1901, 2020

    1901

  19. [27]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020

    2020

  20. [28]

    Palm: Scaling language modeling with pathways

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1– 113, 2023

    2023

  21. [29]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730–19742. PMLR, 2023

    2023

  22. [30]

    Visual instruction tuning.Advances in neural information processing systems , 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems , 36:34892–34916, 2023

    2023

  23. [31]

    Llava-next: Improved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, 2024

    2024

  24. [32]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  25. [33]

    Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 2023

    2023 arXiv

  26. [34]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  27. [35]

    Per-pixel classification is not all you need for semantic segmentation

    Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. Advances in neural information processing systems , 34:17864– 17875, 2021

    2021

  28. [36]

    Segformer: Simple and efficient design for semantic segmentation with transformers

    Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in neural information processing systems , 34:12077–12090, 2021

    2021

  29. [37]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human langu...

    2019

  30. [38]

    Vision-language transformer and query generation for referring segmentation

    Henghui Ding, Chang Liu, Suchen Wang, and Xudong Jiang. Vision-language transformer and query generation for referring segmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 16321–16330, 2021

    2021

  31. [39]

    Cris: Clip-driven referring image segmentation

    Zhaoqing Wang, Yu Lu, Qiang Li, Xunqiang Tao, Yandong Guo, Mingming Gong, and Tongliang Liu. Cris: Clip-driven referring image segmentation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 11686–11695, 2022

    2022

  32. [40]

    Restr: Convolution- free referring image segmentation using transformers

    Namyup Kim, Dongwon Kim, Cuiling Lan, Wenjun Zeng, and Suha Kwak. Restr: Convolution- free referring image segmentation using transformers. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition , pages 18145–18154, 2022

    2022

  33. [41]

    Gres: Generalized referring expression segmen- tation

    Chang Liu, Henghui Ding, and Xudong Jiang. Gres: Generalized referring expression segmen- tation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 23592–23601, 2023

    2023

  34. [42]

    Pixellm: Pixel reasoning with large multimodal model

    Zhongwei Ren, Zhicheng Huang, Yunchao Wei, Yao Zhao, Dongmei Fu, Jiashi Feng, and Xiaojie Jin. Pixellm: Pixel reasoning with large multimodal model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26374–26383, 2024

    2024

  35. [43]

    Generative semantic segmentation

    Jiaqi Chen, Jiachen Lu, Xiatian Zhu, and Li Zhang. Generative semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 7111–7120, 2023

    2023

  36. [44]

    All in tokens: Unifying output space of visual tasks via soft token

    Jia Ning, Chen Li, Zheng Zhang, Chunyu Wang, Zigang Geng, Qi Dai, Kun He, and Han Hu. All in tokens: Unifying output space of visual tasks via soft token. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 19900–19910, 2023

    2023

  37. [45]

    Unified-io: A unified model for vision, language, and multi-modal tasks

    Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: A unified model for vision, language, and multi-modal tasks. arXiv preprint arXiv:2206.08916, 2022

    2022 arXiv

  38. [46]

    Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action

    Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action. In Proceedings of the IEEE/CVF Conference on Computer Vision an...

    2024

  39. [47]

    Scaling autoregressive models for content-rich text-to-image generation

    Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206.10789 , 2(3):5, 2022

    2022 arXiv

  40. [48]

    Chameleon: Mixed-modal early-fusion foundation models, 2024

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models, 2024. URL https://arxiv. org/abs/2405.09818, 9

    2024 arXiv

  41. [49]

    Neural discrete representation learning

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017

    2017

  42. [50]

    Vector-quantized image modeling with improved vqgan

    Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vqgan. arXiv preprint arXiv:2110.04627, 2021

    2021 arXiv

  43. [51]

    Movq: Modulating quantized vectors for high-fidelity image generation

    Chuanxia Zheng, Tung-Long Vuong, Jianfei Cai, and Dinh Phung. Movq: Modulating quantized vectors for high-fidelity image generation. Advances in Neural Information Processing Systems , 35:23412–23425, 2022

    2022

  44. [52]

    Auto-encoding variational bayes, 2013

    Diederik P Kingma, Max Welling, et al. Auto-encoding variational bayes, 2013

    2013

  45. [53]

    Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features

    Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alab- dulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, an...

    2025 arXiv

  46. [54]

    Gaussian error linear units (gelus)

    Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016

    2016 arXiv

  47. [55]

    Root mean square layer normalization

    Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32, 2019

    2019

  48. [56]

    Glu variants improve transformer

    Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020

    2002 arXiv

  49. [57]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024

    2024

  50. [58]

    Janus-pro: Unified multimodal understanding and generation with data and model scaling

    Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint arXiv:2501.17811, 2025

    2025 arXiv

  51. [59]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11975–11986, 2023

    2023

  52. [60]

    Coco-stuff: Thing and stuff classes in context

    Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1209–1218, 2018

    2018

  53. [61]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European Conference on Computer Vision, pages 38–55. Springer, 2024

    2024

  54. [62]

    Semantic understanding of scenes through the ade20k dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127:302–321, 2019

    2019

  55. [63]

    Modeling context in referring expressions

    Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14 , pages 69–85. Springer, 2016

    2016

  56. [64]

    Referitgame: Referring to objects in photographs of natural scenes

    Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) , pages 787–798, 2014

    2014

  57. [65]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  58. [66]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022

    2022

  59. [67]

    Language-based segmentation assistant for complex queries

    Cong Wei, Haoxian Tan, Yujie Zhong, Yujiu Yang, and Lin Ma Lasagna. Language-based segmentation assistant for complex queries. arXiv preprint arXiv:2404.08506, 2(3):6, 2024

    2024 arXiv

  60. [68]

    Lavt: Language-aware vision transformer for referring image segmentation

    Zhao Yang, Jiaqi Wang, Yansong Tang, Kai Chen, Hengshuang Zhao, and Philip HS Torr. Lavt: Language-aware vision transformer for referring image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 18155–18165, 2022. 14

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.