Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

HRSeg: High-Resolution Visual Perception and Enhancement for Reasoning Segmentation

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read High-resolution perception—not stronger language reasoning—is the main bottleneck in reasoning segmentation, and cropping images into encoder-sized patches plus region attention fixes it more cheaply than naive upscaling.

desk verdict Solid empirical paper on high-res reasoning segmentation, but the headline attribution to resolution is confounded with a backbone switch. read the letter →

arxiv 2507.12883 v2 pith:TSSGNKYO submitted 2025-07-17 cs.CV

Weihuang Lin , Yiwei Ma , Xiaoshuai Sun , Shuting He , Jiayi Ji , Liujuan Cao , Rongrong Ji This is my paper
classification cs.CV
keywords reasoningsegmentationmultimodallargelanguagemodelshigh-resolutionperceptionregionattentionmaskproposalselectionfine-grainedfeaturesLLaVASAM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that the main bottleneck in reasoning segmentation is perceptual: visual encoders pre-trained at low resolution (here, DINOv2 at $518\times518$) miss the fine detail that implicit user instructions often point to. It argues that simply interpolating positional embeddings to accept bigger images gives little gain and raises compute steeply, so it proposes HRSeg, which instead crops the high-resolution image into encoder-sized pieces and fuses them with the global view. The load-bearing result is that its two modules—High-Resolution Perception (HRP) and High-Resolution Enhancement (HRE)—take HRSeg from a 52.3 to a 57.4 gIoU (average per-image intersection-over-union) on the ReasonSeg validation set, and from 45.5 to 58.7 gIoU on LLM-Seg40K, while leaving the mask proposal generator untouched. If the claim is right, fine-grained visual detail, not stronger language reasoning, is what current reasoning-segmentation models lack, and that detail can be recovered with cheap local crops rather than by retraining the segmenter.

What carries the argument

The central machinery is region attention over cropped high-resolution features. In HRP, global feature tokens attend only to the local feature block that lies in their spatial region of a magnified crop, so the frozen low-resolution encoder can absorb fine detail without quadratic cost over the whole image; the attended local features are then fused with the global features by a weighted sum with weight $\gamma$ (set to 0.8). Mask pooling aggregates the resulting hybrid features through each SAM proposal to form mask features, and HRE applies $L=2$ layers of cross-attention from those mask features to the hybrid features, with a residual connection and layer normalization, so each mask can use information outside its own masked area. The named object is the region-attention module; it is what converts a frozen $518\times518$ encoder plus crops into fine-grained hybrid features at roughly linear, not quadratic, cost in resolution.

What would settle it

Compute SAM proposal recall on ReasonSeg and LLM-Seg40K by checking whether any proposal from the $32\times32$ grid overlaps a given ground-truth object. If a sizable fraction of small or elongated objects receive no overlapping proposal, HRSeg's reported accuracy is capped by proposal recall regardless of HRP/HRE; a cleaner test is to inject the ground-truth mask as an extra proposal at inference and check whether gIoU jumps to near-perfect, which would separate proposal recall from selection error.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a reasoning-segmentation model can become markedly more accurate by giving its frozen visual encoder high-resolution local views and then letting mask features borrow detail from those views—without fine-tuning the segmenter. HRSeg keeps the two-stage recipe: SAM proposes masks from a $32\times32$ grid of point prompts, and a LLaVA-based MLLM with LoRA selects among them. The High-Resolution Perception module upscales the image by a factor $N=2$, crops it into $518\times518$ patches, runs the frozen encoder over both the global view and the crops, and uses region attention so each global token can pull in corresponding fine-grained local features. The High-Resolution Enhancement module then runs $L=2$ layers of cross-attention that let each mask feature absorb those hybrid features while keeping a residual of its original content. With this design, HRSeg reaches 57.4 gIoU and 54.7 cIoU on ReasonSeg validation and 58.7 gIoU and 58.0 cIoU on LLM-Seg40K, beating the fine-tuned LLM-Seg baseline by 5.1 and 13.2 gIoU points respectively, while the naive interpolation baseline the authors construct stops improving and grows costly as resolution increases.

Load-bearing premise

The final mask must be one of the proposals that the frozen SAM model produces from a fixed $32\times32$ grid of point prompts; if the target object is absent from every proposal, no later module can create it, and the paper does not measure how often that happens.

Editorial extensions

If this is right

  • If the central claim is correct, reasoning-segmentation systems can be improved without retraining their segmenter: frozen SAM proposals plus high-resolution crops and attention-based refinement close much of the perceptual gap.
  • The fine-tuning numbers quantify how much accuracy is recoverable from resolution alone: 57.4 versus 52.3 gIoU on ReasonSeg and 58.7 versus 45.5 gIoU on LLM-Seg40K against the LLM-Seg baseline.
  • The ablation ordering—HRP alone, then HRP plus HRE—shows the two modules are complementary, with the perception module contributing most of the gain and the enhancement module adding a consistent increment.
  • The larger gains on long-query reasoning instructions suggest fine-grained visual detail matters most when the instruction depends on subtle visual evidence rather than a simple referent.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the reported gains are bounded by the recall of SAM proposals from the fixed $32\times32$ point grid, and the paper does not measure that recall, so part of the headroom may be better selection among existing proposals rather than recovery of objects the grid missed.
  • Beyond the paper: replacing the fixed grid with density-adaptive or attention-guided point prompts and re-running the same HRP/HRE pipeline would directly test whether the remaining errors are proposal-recall errors or feature-alignment errors.
  • Beyond the paper: because HRP and HRE only require mask proposals plus a frozen visual encoder, the same high-resolution recipe should transfer to other proposal-based dense prediction tasks, such as referring expression segmentation and open-vocabulary segmentation; the paper's RefCOCO/+/g results already hint at such transfer.
  • Beyond the paper: in the resolution ablation, performance still rises from $N=2$ to $N=4$ (59.3 versus 57.4 gIoU on ReasonSeg), so calling $N=2$ optimal is a compute trade-off, not a demonstrated perceptual ceiling.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes HRSeg, a reasoning segmentation model built on the LLM-Seg two-stage pipeline in which frozen SAM generates mask proposals and a multimodal LLM selects the best matches. HRSeg adds two modules. The High-Resolution Perception (HRP) module crops the image into high-resolution local views, fuses them with the global view through region attention, and uses mask pooling over DINOv2-L features to obtain mask features. The High-Resolution Enhancement (HRE) module refines these mask features by cross-attending to the hybrid high-resolution features. The model is trained with mask-selection, IoU-distribution, and text-generation losses. Experiments on ReasonSeg, LLM-Seg40K, and RefCOCO/+/g report substantial improvements over LLM-Seg and competitive or state-of-the-art results, and ablations show that adding HRP alone and then HRE further improves performance.

Significance. If the causal attribution to resolution holds, HRSeg makes a useful contribution: it shows that a proposal-selection formulation can benefit from richer visual features without fine-tuning SAM, and the reported gains over LLM-Seg on ReasonSeg and LLM-Seg40K are large. The paper ships a complete, reproducible pipeline with code availability, uses the same training recipe as the LLM-Seg baseline for controlled comparisons, and provides ablation studies for the main components, the fusion weight, image resolution, mask selection strategy, and threshold. These are concrete strengths. The main weakness is that the title-level claim that high-resolution perception is the cause of the gains is confounded by the simultaneous switch to DINOv2-L mask features, and the absence of an N=1 control leaves the central attribution unproven.

major comments (4)
  1. [Section 4.6, Table 6; Section 3.2, Eq. (10)] The ablation table used to support the high-resolution claim does not isolate resolution. The row labeled 'N=-' is the unmodified LLM-Seg baseline, whose mask features come from the baseline's own feature pipeline, whereas the HRP module in HRSeg obtains mask features by pooling DINOv2-L features (Eq. 10). Thus the comparison between the 'N=-' row and the 'N=2' row changes two variables at once: the mask-feature backbone/representation and the input resolution. The +5.1 and +13.2 gIoU gains reported in Tables 1 and 3 could in whole or in part come from the switch to DINOv2-L rather than from high-resolution crops. Please add an ablation with HRP active and N=1 (i.e., the global 518x518 image without cropped high-resolution views) to isolate the resolution effect, and state which feature backbone is used for each row.
  2. [Section 1 and Fig. 1(b)] The paper claims that HRSeg 'outperforms the naive baseline in both accuracy and efficiency' (Section 1), but no quantitative efficiency comparison is given. Fig. 1(b) plots a schematic curve but has no axis labels or numerical values, and the 'naive baseline' with interpolated positional embeddings is never tabulated in the experiments. Please provide GFLOPs and, ideally, inference time or wall-clock segmentation time for HRSeg, the naive baseline, and the LLM-Seg baseline, or clearly state the basis for the efficiency claim.
  3. [Section 3.1 and Section 4.2] The pipeline inherits the proposal recall of frozen SAM, which is prompted only with a 32x32 grid of points. HRP and HRE re-rank and refine among these proposals but cannot recover a target object that is absent from SAM's proposal set, and the paper does not measure or report the proposal recall ceiling on the evaluation datasets. Because the manuscript attributes HRSeg's accuracy to fine-grained perception, it needs to show that the performance gap to the baseline is not primarily a re-ranking artifact and to quantify the upper bound imposed by the proposal generator. Please add an analysis of SAM proposal recall on ReasonSeg and LLM-Seg40K (e.g., the best achievable gIoU/cIoU by oracle selection among the proposals).
  4. [Tables 1-8] All reported metrics are single-run point estimates without error bars, standard deviations, or significance tests. Some of the main reported advantages are small in absolute terms (e.g., +2.7 gIoU on ReasonSeg val without fine-tuning in Table 1), so it is impossible to assess whether the differences are robust to random seed and training variability. Please report results from multiple runs (at least for the key comparison and the main ablation), with mean and standard deviation, or otherwise justify that the model is deterministic in practice.
minor comments (5)
  1. [Fig. 1] The caption of Fig. 1 contains an extraneous Chinese phrase '第六版', which appears to be a residual template artifact; it should be removed.
  2. [Fig. 1(b)] The axes of Fig. 1(b) are not labeled. The text says the plot compares gIoU and GFLOPs, but the left and right panels should be explicitly labeled with units, and the legend should identify the naive baseline and HRSeg curves.
  3. [Eq. (10)] Equation (10) 'M_HRP = M · F_T^r / Σ M' is notationally unclear: the mask M appears both as a set of proposals and as a weighting vector, and the summation over M is not defined in terms of its dimensions. Please clarify the pooling operation (e.g., by defining M as a binary or continuous mask matrix and giving explicit index ranges).
  4. [Section 2.1 and References] The introductory sentence 'visual encoders are typically pre-trained at lower resolutions, such as DINOv2' cites references [4, 49, 55, 72], but only [49] is the DINOv2 paper; [4] is a diffusion backbone paper, [55] is SigLIP 2, and [72] is DINO detection. The citation clustering should be corrected to avoid implying that all four references are DINOv2.
  5. [Section 4.2] The implementation details state that the mask proposals are generated with a 32x32 grid, but the number of proposals K and the batch size, training steps, and DeepSpeed configuration are not given. Please provide the proposal count, training iterations, and hardware setup to support reproducibility.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the performance claims are measured against external benchmarks, and the HRP/HRE modules are independently specified; the only mild issues are validation-set tuning and a non-load-bearing self-citation.

full rationale

HRSeg's central claims are empirical: gIoU/cIoU numbers in Tables 1 and 3 are computed on ReasonSeg and LLM-Seg40K against external baselines, not derived from fitted constants or from a self-citation chain. The HRP and HRE modules are defined by explicit equations (Eqs. 3-12) and ablated in Table 4; the gains are measured outcomes rather than consequences of definition. The hyperparameters gamma=0.8 and the IoP threshold=0.7 are selected by searching the ReasonSeg validation split (Tables 5 and 8), which is ordinary model selection and does not make the reported results circular, although it is a mild evaluation caveat. The only self-citation is reference [45] (INF-LLaVA), cited in Related Work among cropping-based MLLM methods; it is not used as the load-bearing justification for HRP/HRE or for the performance claims, so it is not a circularity. The ablation in Table 6 does confound resolution with the switch to DINOv2-L features (the '-' row is the unmodified LLM-Seg baseline rather than HRSeg with N=1), but that is a confound/attribution weakness, not a logical circularity: the N=2, 3, 4 progression still provides independent evidence on the resolution axis. No prediction in the paper reduces by construction to a fitted input.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The architecture rests on standard attention building blocks plus three domain assumptions: pretrained encoders transfer to high-resolution crops, spatial alignment between global and cropped local tokens holds, and SAM's grid proposals cover all targets. Hyperparameters gamma, N, and the IoP threshold are tuned on validation splits rather than derived, which adds a modest circularity burden to the reported numbers.

free parameters (4)
  • gamma = 0.8
    Weight in Eq. 9 for fusing global and local hybrid features; chosen by scanning on the ReasonSeg validation split (Table 5).
  • magnification_N = 2
    Sets the high-resolution input to 1036x1036; chosen as an accuracy and efficiency trade-off on validation splits (Table 6).
  • iop_threshold = 0.7
    Threshold for multi-mask selection; tuned on the ReasonSeg validation split (Table 8).
  • hre_layers = 2
    Depth L of the HRE cross-attention stack; set without a dedicated ablation in the paper.
assumptions (4)
  • standard math Softmax attention, KL divergence, and linear projections are used as standard machine-learning building blocks.
    The method relies on conventional attention and loss formulas without formal proof, which is unproblematic for an empirical paper.
  • domain assumption Frozen DINOv2 and SAM, pretrained at 518x518, remain effective on 1036x1036 inputs through cropping and grid prompting.
    Sections 3.1 and 4.2 use DINOv2 for both global and local features and frozen SAM for proposals; if resolution transfer degrades, the whole pipeline loses value.
  • domain assumption Global tokens and high-resolution crop tokens are spatially aligned in blocks of N by N after aspect-ratio scaling and cropping, with n_l = N times n_g.
    Eqs. 7 and 8 sample the local feature grid at aligned offsets; misalignment would let region attention mix features from wrong image locations.
  • domain assumption SAM's 32x32 grid of point prompts produces proposals covering all targets.
    Section 3.1 states masks come only from grid-prompted proposals; mask selection cannot recover objects absent from this proposal set.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HRSeg: High-Resolution Visual Perception and Enhancement for Reasoning Segmentation." pith.science (2026). https://pith.science/paper/TSSGNKYO

@misc{pith2026250712883,
  author       = {Pith},
  title        = {Pith review of: HRSeg: High-Resolution Visual Perception and Enhancement for Reasoning Segmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TSSGNKYO}},
  note         = {Machine review of arXiv:2507.12883}
}
read the original abstract

The reasoning segmentation task involves segmenting objects within an image by interpreting implicit user instructions, which may encompass subtleties such as contextual cues and open-world knowledge. Despite significant advancements made by existing approaches, they remain constrained by low perceptual resolution, as visual encoders are typically pre-trained at lower resolutions. Furthermore, simply interpolating the positional embeddings of visual encoders to enhance perceptual resolution yields only marginal performance improvements while incurring substantial computational costs. To address this, we propose HRSeg, an efficient model with high-resolution fine-grained perception. It features two key innovations: High-Resolution Perception (HRP) and High-Resolution Enhancement (HRE). The HRP module processes high-resolution images through cropping, integrating local and global features for multi-granularity quality. The HRE module enhances mask features by integrating fine-grained information from high-resolution images, refining their alignment with text features for precise segmentation. Extensive ablation studies validate the effectiveness of our modules, while comprehensive experiments on multiple benchmark datasets demonstrate HRSeg's superior performance.

Figures

Figures reproduced from arXiv: 2507.12883 by the authors.

Figure 1
Figure 1. (a) Compared to LLM-Seg[56], which misses critical details in this complex scene, HRSeg outperforms it due to its fine-grained perception capabilities. (b) Comparison of performance (gIoU) and computational complexity (GFLOPs) between the two pipelines. the Vision Transformer (ViT) [13, 40, 41, 69], their computational complexity increases quadratically with resolution, resulting in significant computational overhea… view at source ↗
Figure 2
Figure 2. The overview framework of the proposed HRSeg. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Details of High-Resolution Perception module. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 3
Figure 3. Figure 3: Firstly, we reshape F𝑔 and F𝑙 to get F𝑔 ∈ R 𝑛𝑔×𝑛𝑔×𝑑 and F𝑙 ∈ R 𝑛𝑙 ×𝑛𝑙 ×𝑑 , where 𝑁𝑔 = 𝑛𝑔 × 𝑛𝑔, 𝑁𝑙 = 𝑛𝑙 × 𝑛𝑙 , and 𝑛𝑙 = 𝑁 × 𝑛𝑔. To perform region-level attention, we provide a definition to capture the region of F𝑙 : 𝑅(F𝑙 ) = F𝑙 [𝑖 : 𝑖 + 𝑁 , 𝑗 : 𝑗 + 𝑁 , :], 0 ≤ 𝑖, 𝑗 ≤ 𝑛…
Figure 4
Figure 4. Figure 4: Qualitative results comparison between HRSeg (Ours) and other methods. HRSeg showcases its fine-grained perception [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. PixDLM: A Dual-Path Multimodal Language Model for UAV Reasoning Segmentation

    cs.CV 2026-04 unverdicted novelty 7.0 of 10

    The work introduces the UAV Reasoning Segmentation task, the DRSeg benchmark dataset, and PixDLM as a baseline dual-path multimodal language model for reasoning-based segmentation in aerial imagery.

Reference graph

Works this paper leans on

87 extracted references · 27 canonical work pages · cited by 1 Pith paper

  1. [1]

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A frontier large vision- language model with versatile abilities. arXiv preprint arXiv:2308.12966 (2023)

    arXiv 2023

  2. [2]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al . 2025. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923 (2025)

    arXiv 2025

  3. [3]

    Zechen Bai, Tong He, Haiyang Mei, Pichao Wang, Ziteng Gao, Joya Chen, Zheng Zhang, and Mike Zheng Shou. 2024. One token to seg them all: Language instructed reasoning segmentation in videos. Advances in Neural Information Processing Systems 37 (2024), 6833–6859

    2024

  4. [4]

    Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu

  5. [5]

    Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. 2024. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954 (2024)

    arXiv 2024

  6. [6]

    Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. 2018. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1209–1218

    2018

  7. [7]

    Xianjie Chen, Roozbeh Mottaghi, Xiaobai Liu, Sanja Fidler, Raquel Urtasun, and Alan Yuille. 2014. Detect what you can: Detecting and representing objects using holistic models and body parts. In Proceedings of the IEEE conference on computer vision and pattern recognition . 1971–1978

    2014

  8. [8]

    Yi-Chia Chen, Wei-Hua Li, Cheng Sun, Yu-Chiang Frank Wang, and Chu-Song Chen. 2024. SAM4MLLM: Enhance Multi-Modal Large Language Model for Referring Expression Segmentation. In European Conference on Computer Vision . Springer, 323–340

    2024

Show all 87 references
  1. [9]

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al . 2024. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF conference on compu...

    2024

  2. [10]

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 20...

    2023

  3. [11]

    Henghui Ding, Chang Liu, Suchen Wang, and Xudong Jiang. 2021. Vision- language transformer and query generation for referring segmentation. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision . 16321– 16330

    2021

  4. [12]

    Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Songyang Zhang, Haodong Duan, Wenwei Zhang, Yining Li, et al. 2024. Internlm- xcomposer2-4khd: A pioneering large vision-language model handling resolu- tions from 336 pixels to 4k hd. arXiv preprint arXi...

    2024 arXiv

  5. [13]

    Alexey Dosovitskiy. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)

    2020 arXiv

  6. [14]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al . 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 (2025)

    2025 arXiv

  7. [15]

    Zonghao Guo, Ruyi Xu, Yuan Yao, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, and Gao Huang. 2025. Llava-uhd: an lmm perceiving any aspect ratio and high-resolution images. In European Conference on Computer Vision. Springer, 390–406

    2025

  8. [16]

    Agrim Gupta, Piotr Dollar, and Ross Girshick. 2019. Lvis: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 5356–5364

    2019

  9. [17]

    Ju He, Shuo Yang, Shaokang Yang, Adam Kortylewski, Xiaoding Yuan, Jie-Neng Chen, Shuai Liu, Cheng Yang, Qihang Yu, and Alan Yuille. 2022. Partimagenet: A large, high-quality dataset of parts. In European Conference on Computer Vision . Springer, 128–145

    2022

  10. [18]

    Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan, Liang Zhang, Bo Zhang, Chen Li, Ji Zhang, Qin Jin, Fei Huang, et al. 2024. mplug-docowl 1.5: Unified structure learning for ocr-free document understanding. arXiv preprint arXiv:2403.12895 (2024)

    2024 arXiv

  11. [19]

    Ronghang Hu, Marcus Rohrbach, and Trevor Darrell. 2016. Segmentation from natural language expressions. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part I

    2016

  12. [20]

    Mingxin Huang, Yuliang Liu, Dingkang Liang, Lianwen Jin, and Xiang Bai. 2024. Mini-monkey: Alleviating the semantic sawtooth effect for lightweight mllms via complementary image pyramid. arXiv preprint arXiv:2408.02034 (2024)

    2024 arXiv

  13. [21]

    Donggon Jang, Yucheol Cho, Suin Lee, Taehyeon Kim, and Dae-Shik Kim. 2025. MMR: A Large-scale Benchmark Dataset for Multi-target and Multi-granularity Reasoning Segmentation. arXiv preprint arXiv:2503.13881 (2025)

    2025 arXiv

  14. [22]

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al . 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088 (2024)

    2024 arXiv

  15. [23]

    Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. 2014. Referitgame: Referring to objects in photographs of natural scenes. InProceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). 787–798

    2014

  16. [24]

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al

  17. [25]

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia

  18. [26]

    In Proceedings of the IEEE/CVF International Conference on Computer Vision

    Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 4015–4026

  19. [27]

    Wentong Li, Yuqian Yuan, Jian Liu, Dongqi Tang, Song Wang, Jie Qin, Jianke Zhu, and Lei Zhang. 2024. Tokenpacker: Efficient visual projector for multimodal llm. arXiv preprint arXiv:2407.02392 (2024)

    2024 arXiv

  20. [28]

    Xiangtai Li, Haobo Yuan, Wei Li, Henghui Ding, Size Wu, Wenwei Zhang, Yining Li, Kai Chen, and Chen Change Loy. 2024. Omg-seg: Is one model good enough for all segmentation?. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition. 27948–27959

    2024

  21. [29]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning . PMLR, 19730–19742

    2023

  22. [30]

    Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. 2024. Monkey: Image resolution and text label are important things for large multi-modal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern R...

    2024

  23. [31]

    Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. 2023. Open-vocabulary semantic segmentation with mask-adapted clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition ...

    2023

  24. [32]

    Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Rui- hang Chu, Shaoteng Liu, and Jiaya Jia. 2024. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403.18814 (2024)

    2024 arXiv

  25. [33]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Cheng- gang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 (2024)

    2024 arXiv

  26. [34]

    Chang Liu, Henghui Ding, and Xudong Jiang. 2023. Gres: Generalized referring expression segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 23592–23601

    2023

  27. [35]

    Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, et al . 2023. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arXiv preprint arXiv:2311.07575 (2023)

    2023 arXiv

  28. [36]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual Instruc- tion Tuning. arXiv preprint arXiv:2304.08485 (2023)

    2023 arXiv

  29. [37]

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. 2025. Grounding dino: Marry- ing dino with grounded pre-training for open-set object detection. In European Conference on Computer Vision . Springer, 38–55

    2025

  30. [38]

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024. Llava-next: Improved reasoning, ocr, and world knowledge

    2024

  31. [39]

    Zuyan Liu, Yuhao Dong, Ziwei Liu, Winston Hu, Jiwen Lu, and Yongming Rao

  32. [40]

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021. Swin transformer: Hierarchical vision transformer us- ing shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision. 10012–10022

    2021

  33. [41]

    Yuqi Liu, Bohao Peng, Zhisheng Zhong, Zihao Yue, Fanbin Lu, Bei Yu, and Jiaya Jia. 2025. Seg-zero: Reasoning-chain guided segmentation via cognitive reinforcement. arXiv preprint arXiv:2503.06520 (2025)

    2025 arXiv

  34. [42]

    I Loshchilov. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)

    2017 arXiv

  35. [43]

    arXiv preprint arXiv:2409.12961 (2024)

    Oryx mllm: On-demand spatial-temporal understanding at arbitrary reso- lution. arXiv preprint arXiv:2409.12961 (2024)

    2024 arXiv

  36. [44]

    Gen Luo, Yiyi Zhou, Xiaoshuai Sun, Liujuan Cao, Chenglin Wu, Cheng Deng, and Rongrong Ji. 2020. Multi-task collaborative network for joint referring expression comprehension and segmentation. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition ...

    2020

  37. [45]

    Zhenhua Liu, Yunhe Wang, Kai Han, Wei Zhang, Siwei Ma, and Wen Gao. 2021. Post-training quantization for vision transformer.Advances in Neural Information MM ’25, October 27–31, 2025, Dublin, Ireland Weihuang Lin et al. Processing Systems 34 (2021), 28092–28103

    2021

  38. [46]

    Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. 2016. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition. 11–20

    2016

  39. [47]

    Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, et al. 2024. Deepseek-vl: towards real- world vision-language understanding. arXiv preprint arXiv:2403.05525 (2024)

    2024 arXiv

  40. [48]

    R OpenAI. 2023. Gpt-4 technical report. arxiv 2303.08774. View in Article 2, 5 (2023)

    2023 arXiv

  41. [49]

    Yiwei Ma, Zhibin Wang, Xiaoshuai Sun, Weihuang Lin, Qiang Zhou, Jiayi Ji, and Rongrong Ji. 2024. INF-LLaVA: Dual-perspective Perception for High-Resolution Multimodal Large Language Model. arXiv preprint arXiv:2407.16198 (2024)

    2024 arXiv

  42. [50]

    Vignesh Ramanathan, Anmol Kalia, Vladan Petrovic, Yi Wen, Baixue Zheng, Baishan Guo, Rui Wang, Aaron Marquez, Rama Kovvuri, Abhishek Kadian, et al

  43. [51]

    AI Meta. 2024. Introducing meta llama 3: The most capable openly available llm to date. Meta AI (2024)

    2024

  44. [52]

    Zhongwei Ren, Zhicheng Huang, Yunchao Wei, Yao Zhao, Dongmei Fu, Jiashi Feng, and Xiaojie Jin. 2024. Pixellm: Pixel reasoning with large multimodal model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 26374–26383

    2024

  45. [53]

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El- Nouby, et al. 2023. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 (2023)

    2023 arXiv

  46. [54]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. LLaMA: open and efficient foundation language models. arXiv. arXiv preprint arXiv:2302.13971 (2023)

    2023 arXiv

  47. [55]

    InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Paco: Parts and attributes of common objects. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 7141–7151

  48. [56]

    Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S Khan. 2024. Glamm: Pixel grounding large multimodal model. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pa...

    2024

  49. [57]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191 (2024)

    2024 arXiv

  50. [58]

    InternLM Team. 2023. Internlm: A multilingual language model with progres- sively enhanced capabilities

    2023

  51. [59]

    XuDong Wang, Shaolun Zhang, Shufan Li, Konstantinos Kallidromitis, Kehan Li, Yusuke Kato, Kazuki Kozuka, and Trevor Darrell. 2024. SegLLM: Multi-round Reasoning Segmentation. arXiv preprint arXiv:2410.18923 (2024)

    2024 arXiv

  52. [60]

    Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. 2025. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization...

    2025 arXiv

  53. [61]

    Junchi Wang and Lei Ke. 2024. LLM-Seg: Bridging Image Segmentation and Large Language Model Reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 1765–1774

    2024

  54. [62]

    Jiannan Wu, Muyan Zhong, Sen Xing, Zeqiang Lai, Zhaoyang Liu, Wenhai Wang, Zhe Chen, Xizhou Zhu, Lewei Lu, Tong Lu, et al. 2024. VisionLLM v2: An End- to-End Generalist Multimodal Large Language Model for Hundreds of Vision- Language Tasks. arXiv preprint arXiv:2406.08394 (2024)

    2024 arXiv

  55. [63]

    Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al . 2023. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. Advances in Neural Information Processing Systems 36 (2023), 61501–61513

    2023

  56. [64]

    Cilin Yan, Haochen Wang, Shilin Yan, Xiaolong Jiang, Yao Hu, Guoliang Kang, Weidi Xie, and Efstratios Gavves. 2024. Visa: Reasoning video object segmentation via large language models. In European Conference on Computer Vision . Springer, 98–115

    2024

  57. [65]

    Zhaoqing Wang, Yu Lu, Qiang Li, Xunqiang Tao, Yandong Guo, Mingming Gong, and Tongliang Liu. 2022. Cris: Clip-driven referring image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 11686–11695

    2022

  58. [66]

    Cong Wei, Yujie Zhong, Haoxian Tan, Yingsen Zeng, Yong Liu, Zheng Zhao, and Yujiu Yang. 2024. InstructSeg: Unifying Instructed Visual Segmentation with Multi-modal Large Language Models. arXiv preprint arXiv:2412.14006 (2024)

    2024 arXiv

  59. [67]

    Yuqi Yang, Peng-Tao Jiang, Jing Wang, Hao Zhang, Kai Zhao, Jinwei Chen, and Bo Li. 2024. Empowering Segmentation Ability to Multi-modal Large Language Models. arXiv preprint arXiv:2403.14141 (2024)

    2024 arXiv

  60. [68]

    Zhuofan Xia, Dongchen Han, Yizeng Han, Xuran Pan, Shiji Song, and Gao Huang

  61. [69]

    In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Gsva: Generalized segmentation via multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 3858–3869

  62. [70]

    Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Junfeng Tian, Qi Qian, Ji Zhang, et al . 2023. Ureader: Universal ocr-free visually-situated language understanding with multimodal large language model. arXiv preprint arXiv:2310.05126 (2023)

    2023 arXiv

  63. [71]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Cheng- peng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671 (2024)

    2024 arXiv

  64. [72]

    Senqiao Yang, Tianyuan Qu, Xin Lai, Zhuotao Tian, Bohao Peng, Shu Liu, and Jiaya Jia. 2023. LISA++: An Improved Baseline for Reasoning Segmentation with Large Language Model. arXiv preprint arXiv:2312.17240 (2023)

    2023 arXiv

  65. [73]

    Tao Zhang, Xiangtai Li, Hao Fei, Haobo Yuan, Shengqiong Wu, Shunping Ji, Chen Change Loy, and Shuicheng Yan. 2024. Omg-llava: Bridging image- level, object-level, pixel-level reasoning and understanding. arXiv preprint arXiv:2406.19389 (2024)

    2024 arXiv

  66. [74]

    Zhao Yang, Jiaqi Wang, Yansong Tang, Kai Chen, Hengshuang Zhao, and Philip HS Torr. 2022. Lavt: Language-aware vision transformer for referring image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 18155–18165

    2022

  67. [75]

    Ting Yao, Yehao Li, Yingwei Pan, Yu Wang, Xiao-Ping Zhang, and Tao Mei

  68. [76]

    IEEE transactions on pattern analysis and machine intelligence 45, 9 (2023), 10870–10882

    Dual vision transformer. IEEE transactions on pattern analysis and machine intelligence 45, 9 (2023), 10870–10882

    2023

  69. [77]

    Chenchen Zhu, Fanyi Xiao, Andrés Alvarado, Yasmine Babaei, Jiabo Hu, Hichem El-Mohri, Sean Culatana, Roshan Sumbaly, and Zhicheng Yan. 2023. Egoobjects: A large-scale egocentric dataset for fine-grained object understanding. InProceed- ings of the IEEE/CVF International Confer...

    2023

  70. [78]

    Ao Zhang, Yuan Yao, Wei Ji, Zhiyuan Liu, and Tat-Seng Chua. 2023. Next-chat: An lmm for chat, detection and segmentation. arXiv preprint arXiv:2311.04498 (2023)

    2023 arXiv

  71. [79]

    Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M Ni, and Heung-Yeung Shum. 2022. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. arXiv preprint arXiv:2203.03605 (2022)

    2022 arXiv

  72. [80]

    Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Wang, Lijuan Wang, Jianfeng Gao, and Yong Jae Lee. 2024. Segment everything everywhere all at once. Advances in Neural Information Processing Systems 36 (2024)

    2024

  73. [81]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems 36 (2023), 46595–46623

    2023

  74. [82]

    Rongkun Zheng, Lu Qi, Xi Chen, Yi Wang, Kun Wang, Yu Qiao, and Hengshuang Zhao. 2024. ViLLa: Video Reasoning Segmentation with Large Language Model. arXiv preprint arXiv:2407.14500 (2024)

    2024 arXiv

  75. [83]

    Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. 2017. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition . 633–641

    2017

  76. [85]

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large lan- guage models. arXiv preprint arXiv:2304.10592 (2023)

    2023 arXiv

  77. [86]

    Xueyan Zou, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Behl, Jianfeng Wang, Lu Yuan, et al. 2023. Generalized decoding for pixel, image, and language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 15116–15127

    2023

  78. [2023]

    In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    All are worth words: A vit backbone for diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 22669–22679

  79. [2024]

    In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Lisa: Reasoning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 9579–9589

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.