Pith. sign in

REVIEW 4 major objections 6 minor 7 cited by

Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videos

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that one lightweight model can segment a prompted region in an image or video and, in the same pass, output its category, definition, function, and caption.

desk verdict PAM has a sensible architecture and a useful dataset, but its zero-shot SOTA claims collapse because the training corpus overlaps with the evaluation benchmarks. read the letter →

arxiv 2506.05302 v1 pith:EMEKERJM submitted 2025-06-05 cs.CV

Weifeng Lin , Xinyu Wei , Ruichuan An , Tianhe Ren , Tingwei Chen , Renrui Zhang , Ziyu Guo , Wentao Zhang
show 2 more authors
Lei Zhang Hongsheng Li
This is my paper
classification cs.CV
keywords region-levelunderstandinginteractivesegmentationvision-languagemodelSAM2videocaptioningstreamingSemanticPerceiverbilingualannotations
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PAM claims that one lightweight model can do what previously required a chain of separate systems: given a click, box, or mask, it segments the prompted object in an image or video and simultaneously says what the object is, what that label means, what the object does in the scene, and what it looks like in detail. The contribution is that SAM 2's frozen segmentation features, passed through a small Semantic Perceiver into a 1.5B or 3B language model, are enough to support all of these outputs at once, with the mask and semantic decoders running in parallel. If the claim holds, region-level visual understanding stops needing an external segmenter plus a captioning model and becomes a single efficient interaction, including for videos and for streaming video captioning. The paper backs this with a 1.5M-image and 0.6M-video region-semantic dataset and reports stronger or comparable numbers than prior 7B-13B models at lower inference cost. The authors say the zero-shot evaluation results are achieved without fine-tuning on the benchmark datasets.

What carries the argument

The load-bearing component is the Semantic Perceiver, a lightweight two-layer transformer with self-attention, image-to-token cross-attention, token-to-image cross-attention, and a point-wise MLP, whose design mirrors SAM 2's feature-fusing module. It receives the enhanced mask tokens--which already carry IoU, prompt, and mask information--and the image embeddings that have interacted with them, concatenates $N_s = 16$ learnable semantic tokens, and emits $64 \times N$ visual tokens plus $N \times 16$ semantic tokens for the LLM. This bridge is what lets PAM keep SAM 2 frozen and still give the language model object-level localization and semantic context; the parallel mask and semantic decoders are what translate that shared representation into simultaneous outputs. A second carrier is the data pipeline: storyboard-style keyframes with Set-of-Mark highlighting are sent to a strong vision-language model to produce refined, temporally aware captions, and event segmentation plus previous-clip context generates the streaming video captions.

What would settle it

Scan the released training data for frames from the validation or test splits of LVIS, PACO, COCO-Text, Total-Text, Visual Genome, RefCOCOg, HC-STVG, Elysium, BensMOT, and the ActivityNet subset; if any appear, the zero-shot claim is contradicted, and retraining with those splits held out would reveal the size of the gap.

Watch

Extended reading notes

Core claim

PAM's central claim, stated on its own terms, is that SAM 2's rich intermediate features already carry enough general vision, localization, and semantic information to drive an LLM's region-level understanding, if those features are condensed properly. The Semantic Perceiver takes the enhanced mask tokens and updated image embeddings from SAM 2's feature-fusing module, adds 16 learnable semantic tokens, and produces compact visual and semantic token sets that Qwen2.5 consumes alongside the task instruction. Because the semantic decoder shares one forward pass with the mask decoder, the model returns a segmentation mask, a category label, a context-aware definition, and a caption together, for both images and videos. On the benchmarks reported, PAM-3B reaches the best or near-best score on LVIS, PACO, COCO-Text, Total-Text, Visual Genome, RefCOCOg, Ferret-Bench, Elysium, BensMOT, and HC-STVG, while running faster and using less memory than the compared 7B-13B systems.

Load-bearing premise

The load-bearing premise is that the benchmark evaluation splits were not part of the training corpus, so the reported zero-shot scores measure generalization rather than memory of training data.

Editorial extensions

If this is right

  • A single visual prompt can produce a mask plus category, definition, function, and caption at once, so applications can drop the serial pipeline of external segmenter, detector, and captioner for region-level understanding.
  • A 1.5B or 3B LLM appears sufficient for competitive region understanding when the visual backbone is a frozen SAM 2, suggesting model scale matters less than where the visual information comes from.
  • Video and streaming video captioning can reuse SAM 2's memory mechanism by carrying denser tokens from the last frame of each clip into the next clip and appending the previous description to the prompt, with no separate streaming model.
  • The released bilingual dataset of 1.5M image and 0.6M video region-semantic annotations gives other region-level models a ready-made training resource.
  • If the claimed zero-shot results reproduce with the released code, the method provides a practical default baseline for future region-level understanding work.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of where the semantic ability lives: keep the Semantic Perceiver and swap SAM 2 for another frozen segmentation backbone; if performance holds, the priors are generic segmentation features, and if it drops, they are specific to SAM 2.
  • Because the training captions were generated by a strong vision-language model and then human-checked, PAM's writing style may inherit that teacher's biases; the paper's own failure cases (describing the most salient object when the prompted object leaves the frame, and repeating the previous clip's description in streaming mode) are consistent with this.
  • The training sources in Fig. 9 overlap with the benchmark datasets in Tables 1-3, and the paper does not state that evaluation splits were held out; unless that exclusion happened, the reported 'zero-shot' numbers should be read as in-distribution performance, and the true generalization gap, if any, remains unmeasured.
  • One testable extension is to use the same Semantic Perceiver design to add region-level semantics to other interactive segmentation models, or to add an instruction-following or VQA head, since the architecture is not tied to the four trained output types.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper presents Perceive Anything Model (PAM), a region-level vision-language model built on SAM 2 and Qwen2.5, with a Semantic Perceiver that converts SAM 2 features into visual and semantic tokens for a language-model decoder. The model is trained on a large corpus of image and video region-level annotations refined with GPT-4o and Qwen2.5-VL, including a newly constructed region-level streaming video captioning dataset. The authors report state-of-the-art zero-shot results on LVIS, PACO, COCO-Text, Total-Text, VG, RefCOCOg, Ref-L4, Ferret-Bench, MDVP-Bench, Elysium, BensMOT, HC-STVG, and a new G-STDC metric for streaming captioning, together with efficiency gains of 1.2-2.4x over prior methods.

Significance. PAM is a plausible and efficient architecture: using SAM 2's frozen features with a lightweight perceptiver is a sensible design, and the reported efficiency gains are worth noting. If the zero-shot results were valid, the work would provide a strong lightweight baseline for region-level understanding in images and videos, and the proposed data pipeline and G-STDC metric could be useful community resources. However, as submitted the empirical validation is compromised by training/evaluation overlap and an unvalidated new metric, so the significance of the reported results is not established.

major comments (4)
  1. [Sec. 5.1, Fig. 9, Tables 1-3] The paper states that all evaluation experiments are conducted in a 'zero-shot test manner' (Sec. 5.1), but Fig. 9 shows that the training corpus includes LVIS, PACO, COCO-Text, Total-Text, VG, RefCOCOg, Elysium, and HC-STVG, which are also the benchmarks evaluated in Tables 1-3. No section or appendix states that the evaluation splits of these datasets were excluded from training. Consequently, the reported SOTA numbers are indistinguishable from in-distribution performance, and the central zero-shot claim is unsupported.
  2. [Sec. 5.3, Table 4, App. C] The proposed G-STDC metric is used to claim optimal streaming captioning performance, but it is a single GPT-4o evaluation with no specified prompt, temperature, number of runs, or inter-rater agreement, and no human correlation study is reported. In addition, the evaluation is run on a self-curated 400-sample ActivityNet subset with manually annotated boxes, and the evaluation code is not released. These omissions make the streaming captioning results and the G-STDC-based SOTA claim unverifiable.
  3. [Sec. 5.2, Sec. 4] Ferret-Bench and MDVP-Bench use GPT-4o as the judge (Sec. 5.2), while the training annotations for PAM were also generated with GPT-4o (Sec. 4.1 and 4.2). The model is therefore rewarded for matching the style and preferences of its teacher, and the referential and description scores in Table 2 are likely inflated. A human evaluation or an alternative judge is needed to support the reported performance on these benchmarks.
  4. [Sec. D.2] The paper admits that the GPT-assisted annotation pipeline may bias the model toward describing the most salient object rather than the prompted region, introducing label inaccuracies. This admission directly undercuts the region-specificity claim that is central to the paper, and no analysis quantifies the frequency or impact of this failure mode on the Table 3 results. The manuscript should include such an analysis or a mitigation strategy.
minor comments (6)
  1. [Sec. 2] There are typographical errors, such as 'modality-specificl', and the reference list contains duplicate entries for Artemis ([48] and [49]) and SAM 2 ([52] and [53]).
  2. [Table 1] The table header contains rendering artifacts such as 'VP-LLA V A' and 'VP-SPHINX'; these should be cleaned up for readability.
  3. [Fig. 9] The dataset composition graphic is hard to parse: the '8M', the percentages, and the bold/non-bold distinction are not explained in the caption, making it difficult to determine which datasets were used for refinement and augmentation.
  4. [Sec. 5.3] No error bars or variance statistics are reported for any of the tables; given the small size of some benchmarks (e.g., the 400-sample ActivityNet subset), at least a few seeds or a bootstrap confidence interval should be reported.
  5. [App. C] The G-STDC scoring rubric is described in one sentence; please provide the exact prompt and scoring instructions to make the metric reproducible.
  6. [Sec. D.3] The statement about 'exponential increase in computational cost' with the number of frames is imprecise; the cost grows approximately linearly with token count and should be described accordingly.

Circularity Check

1 steps flagged · score 4.0 of 10

GPT-4o serves as both the generator of PAM's training captions and the judge on Ferret-Bench, MDVP-Bench, and the new G-STDC metric, making part of the reported 'quality' self-referential; the main lexical benchmarks and the architecture remain independent, so circularity is partial.

  1. self definitional [Sec 4.2 (data generation) with Sec 5.2 and Sec 5.3/App. C (GPT-4o evaluation)]
    "Then, using the original annotations as condition, we prompt GPT-4o to generate descriptions that are both refined, detailed and temporally aware. ... Additionally, we benchmark referring descriptions via Ferret-Bench [77] and MDVP-Bench [41], where GPT-4o is employed to gauge the quality of the generated responses. ... To further assess the continuity and entity consistency of descriptions for sequential events, we propose a new metric: the GPT-4o-evaluated Spatio-Temporal Description Continuity Score (G-STDC)."

    PAM's training targets for region explanations and video/streaming captions are generated by GPT-4o (with subsequent manual QA), and its performance on Ferret-Bench, MDVP-Bench, and the newly proposed G-STDC is scored by GPT-4o. The judge and the teacher are therefore the same system: a high GPT-4o score partly measures how closely PAM reproduces the GPT-4o writing style and content priors that were used to create its supervision, rather than an independent external standard of semantic quality. This is most acute for G-STDC, which is introduced and evaluated in the same paper without human-correlation or inter-rater validation, so the reported 'optimal' streaming-continuity score is a self-referential measure.

full rationale

PAM's core derivation is an engineering/data claim, not a mathematical derivation: frozen SAM 2 features are passed through a Semantic Perceiver to an LLM, and the model is trained in three stages. No equation in the paper defines an output in terms of an input in a way that would make a result true by construction. The main benchmark tables rely on lexical metrics (Semantic IoU, METEOR, CIDEr, OCR accuracy) computed against fixed annotations, so those SOTA numbers are externally falsifiable and not circular. The self-referential element is the GPT-4o teacher/judge overlap on the referring-description and streaming metrics: the model is trained on GPT-4o-generated captions and then scored by GPT-4o. This does not force the lexical results, hence partial circularity. Separately, the reader's training/evaluation overlap concern is a validity risk rather than a demonstrated circular reduction: Fig. 9 lists LVIS, PACO, COCO-Text, VG, RefCOCOg, Elysium, and HC-STVG among training sources while Tables 1-3 evaluate on those benchmarks, and Sec. 5.1 calls the protocol zero-shot without stating that evaluation splits were excluded. If the exact test samples were used in training, the SOTA claims would collapse to in-distribution fitting, but the paper does not publish enough detail to prove that reduction; it is an external-validity/correctness concern. No load-bearing self-citation chain or imported uniqueness theorem appears; citations to the authors' prior work are as baselines and dataset sources, not as the justification for PAM's central claims.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

No new physical entities, forces, or conservation laws are introduced. The Semantic Perceiver is a new module, but it is implemented with standard transformer layers and does not constitute a postulated external entity.

free parameters (4)
  • Number of semantic tokens N_s = 16
    Set by hand; ablation in Table 5 shows 16 is best on combined LVIS/RefCOCOg/HC-STVG.
  • Number of frames per video clip N = 16
    Default in Sec 5.1 and Table 8; chosen to balance token count and information.
  • Storyboard keyframe count = 6
    Six keyframes uniformly extracted per video in Sec 4.2 for caption expansion; no sensitivity analysis.
  • Pixel shuffle factors = 2x2 or 4x4
    Image tokens use 2x2, video non-prompt frames use 4x4; chosen by hand to reduce LLM token load (Sec 3.1).
assumptions (5)
  • domain assumption SAM 2's frozen intermediate features carry general vision, localization, and semantic priors sufficient for LLM decoding.
    In Sec 3.1, Semantic Perceiver is designed to use SAM 2 features as semantic inputs; no evidence is given that these features alone carry enough semantics.
  • domain assumption GPT-4o and Qwen2.5-VL generated annotations are accurate enough to train a reliable model.
    Sec 4 and App B describe using GPT-4o and Qwen2.5-VL to create definitions and captions, with manual QA; the accuracy of the teacher signal is assumed.
  • domain assumption Manual QA and rule-based filtering remove harmful annotation errors.
    App B.1 states two-stage cleaning, but no statistics on filter rates or residual error are given.
  • domain assumption Evaluation benchmarks are disjoint from training data.
    Sec 5.1 claims zero-shot evaluation, but Fig. 9 includes evaluation datasets in the training list; no exclusion is documented.
  • domain assumption Qwen2.5 is an adequate decoder for this visual-semantic task.
    Sec 3.1 adopts Qwen2.5 as semantic decoder; no comparison with other LLM decoders is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videos." pith.science (2026). https://pith.science/paper/EMEKERJM

@misc{pith2026250605302,
  author       = {Pith},
  title        = {Pith review of: Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videos},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EMEKERJM}},
  note         = {Machine review of arXiv:2506.05302}
}
read the original abstract

We present Perceive Anything Model (PAM), a conceptually straightforward and efficient framework for comprehensive region-level visual understanding in images and videos. Our approach extends the powerful segmentation model SAM 2 by integrating Large Language Models (LLMs), enabling simultaneous object segmentation with the generation of diverse, region-specific semantic outputs, including categories, label definition, functional explanations, and detailed captions. A key component, Semantic Perceiver, is introduced to efficiently transform SAM 2's rich visual features, which inherently carry general vision, localization, and semantic priors into multi-modal tokens for LLM comprehension. To support robust multi-granularity understanding, we also develop a dedicated data refinement and augmentation pipeline, yielding a high-quality dataset of 1.5M image and 0.6M video region-semantic annotations, including novel region-level streaming video caption data. PAM is designed for lightweightness and efficiency, while also demonstrates strong performance across a diverse range of region understanding tasks. It runs 1.2-2.4x faster and consumes less GPU memory than prior approaches, offering a practical solution for real-world applications. We believe that our effective approach will serve as a strong baseline for future research in region-level visual understanding.

Figures

Figures reproduced from arXiv: 2506.05302 by the authors.

Figure 1
Figure 1. Perceive Anything Model (PAM): PAM accepts various visual prompts (such as clicks, boxes, and masks) to produce region-specific information for images and videos, including masks, category, label definition, contextual function, and detailed captions. The model also handles demanding region-level streaming video captioning. Abstract We present Perceive Anything Model (PAM), a conceptually straightforward and efficie… view at source ↗
Figure 2
Figure 2. Previous Paradigms vs. Our Paradigm (PAM). (a & b) Textual/VPE methods provide region understanding using positional embeddings but typically lack simultaneous object masks. (c) RoI/Segmentation-based methods use external segmenter for object masks, subsequently fusing image and mask embeddings. (d) In contrast to previous paradigms, our method directly treats the Seg. model as vision encoder. It effectively leverag… view at source ↗
Figure 3
Figure 3. Overall Architecture of PAM. further pioneered the use of LLMs to achieve free-form dialogue synchronized with the online video stream. However, these approaches predominantly focus on general event comprehension, leaving the continuous tracking and description of specific regions within a video stream as a significant unresolved challenge. 3 Perceive Anything Model (PAM) Given visual prompts such as points, boxes, … view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Detailed illustration of our PAM workflow. Semantic Perceiver first receives enhanced image embeddings and mask tokens from the S2-FFM and outputs enriched visual tokens and semantic tokens. These are subsequently fed into the semantic decoder for decoding. Semantic De…
Figure 5
Figure 5. Figure 5: Illustrative examples of our dataset construction pipeline. The left panel displays image annotations; the right panel details annotations for non-streaming and streaming video. annotations spanning from coarse-level (categories, definitions, contextual functionalities…
Figure 6
Figure 6. Figure 6: PAM provides various semantic granularities informantion and support bilingual outputs. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Qualitative visualization examples of PAM for region-level non-streaming and streaming [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Comparison of GPU memory usage and inference [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: All Public Dataset Collection. Datasets highlighted in [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: Qualitative comparison between PAM and prior models. [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: PAM can accurately describe specific background areas, such as roads, ground surfaces, [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]
Figure 12
Figure 12. Figure 12: Failure Cases of PAM in Images [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]
Figure 13
Figure 13. Figure 13: Failure Cases of PAM in Videos. D.3 Performance on Long Videos PAM’s performance on long videos depends on the number of video frames processed. A larger number of frames enables PAM to generate more information-rich descriptions, but this incurs an exponential increa…

Discussion (0). Sign in to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. FRFDet: Efficient UAV Small Object Detection with Symmetric Sampling and Scalable Fusion

    cs.CV 2026-07 accept novelty 6.0 of 10

    FRFDet improves UAV small-object detection with learnable symmetric Inverse Bidirectional Sampling and capacity-dependent Scale-Feature Relationship Cross-Fusion, reaching SOTA among lightweight models.

  2. Scone: Bridging Composition and Distinction in Subject-Driven Image Generation via Unified Understanding-Generation Modeling

    cs.CV 2025-12 conditional novelty 6.0 of 10

    Scone unifies subject understanding and generation in a two-stage trained model to improve both composition and distinction in multi-subject image generation, outperforming prior open-source models on new benchmarks.

  3. RefBench-PRO: Perceptual and Reasoning Oriented Benchmark for Referring Expression Comprehension

    cs.CV 2025-12 conditional novelty 6.0 of 10

    RefBench-PRO organizes REC into attribute, position, interaction, relation, commonsense, and reject tasks; no tested MLLM exceeds 72%, and Ref-R1 raises Qwen2.5-VL-7B from 57.6 to 69.4 on it.

  4. Strefer: Empowering Video LLMs with Space-Time Referring and Reasoning via Synthetic Instruction Data

    cs.CV 2025-09 conditional novelty 6.0 of 10

    Adding Strefer's synthetic space-time reference questions to video instruction tuning improves mask-referred description/QA, timestamp QA, and temporal reasoning over a video-LLM baseline.

  5. OpenWorldLib: A Unified Codebase and Definition of Advanced World Models

    cs.CV 2026-04 unverdicted novelty 4.0 of 10

    OpenWorldLib offers a standardized codebase and definition for world models that combine perception, interaction, and memory to understand and predict the world.

  6. Describe Anything Model for Visual Question Answering on Text-rich Images

    cs.CV 2025-07 conditional novelty 4.0 of 10

    DAM-QA aggregates answers from full-image and sliding-window views of the Describe Anything Model with a weighted vote, improving text-rich VQA on some benchmarks but not all.

  7. Empowering Multimodal LLMs with External Tools: A Comprehensive Survey

    cs.CV 2025-08 unverdicted novelty 2.0 of 10

    A survey paper maps how external tools are used to augment multimodal large language models across data, tasks, evaluation, and future directions.

Reference graph

Works this paper leans on

89 extracted references · 38 canonical work pages · cited by 7 Pith papers

  1. [1]

    Mc-llava: Multi-concept personalized vision-language model.arXiv preprint arXiv:2411.11706, 2024

    Ruichuan An, Sihan Yang, Ming Lu, Renrui Zhang, Kai Zeng, Yulin Luo, Jiajun Cao, Hao Liang, Ying Chen, Qi She, et al. Mc-llava: Multi-concept personalized vision-language model.arXiv preprint arXiv:2411.11706, 2024

    arXiv 2024

  2. [2]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    arXiv 2025

  3. [3]

    Meteor: An automatic metric for mt evaluation with improved correlation with human judgments

    Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. InProceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72, 2005

    2005

  4. [4]

    Abductive commonsense reasoning, 2020

    Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Scott Wen tau Yih, and Yejin Choi. Abductive commonsense reasoning, 2020

    2020

  5. [5]

    Graph cuts in vision and graphics: Theories and applications

    Yuri Boykov and Olga Veksler. Graph cuts in vision and graphics: Theories and applications. InHandbook of mathematical models in computer vision, pages 79–96. Springer, 2006

    2006

  6. [6]

    Scene-text oriented referring expression comprehension.IEEE Transactions on Multimedia, 25:7208–7221, 2023

    Yuqi Bu, Liuwu Li, Jiayuan Xie, Qiong Liu, Yi Cai, Qingbao Huang, and Qing Li. Scene-text oriented referring expression comprehension.IEEE Transactions on Multimedia, 25:7208–7221, 2023

    2023

  7. [7]

    Activitynet: A large-scale video benchmark for human activity understanding

    Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. InProceedings of the ieee conference on computer vision and pattern recognition, pages 961–970, 2015

    2015

  8. [8]

    Vip-llava: Making large multimodal models understand arbitrary visual prompts

    Mu Cai, Haotian Liu, Siva Karthik Mustikovela, Gregory P Meyer, Yuning Chai, Dennis Park, and Yong Jae Lee. Vip-llava: Making large multimodal models understand arbitrary visual prompts. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12914–12923, 2024

    2024

Show all 89 references
  1. [9]

    Active contours without edges.IEEE Transactions on image processing, 10(2):266–277, 2001

    Tony F Chan and Luminita A Vese. Active contours without edges.IEEE Transactions on image processing, 10(2):266–277, 2001

    2001

  2. [10]

    Jierun Chen, Fangyun Wei, Jinjing Zhao, Sizhe Song, Bohuai Wu, Zhuoxuan Peng, S. H. Gary Chan, and Hongyang Zhang. Revisiting referring expression comprehension evaluation in the era of large multimodal models, 2024

    2024

  3. [11]

    Videollm-online: Online video large language model for streaming video

    Joya Chen, Zhaoyang Lv, Shiwei Wu, Kevin Qinghong Lin, Chenan Song, Difei Gao, Jia-Wei Liu, Ziteng Gao, Dongxing Mao, and Mike Zheng Shou. Videollm-online: Online video large language model for streaming video. InProceedings of the IEEE/CVF Conference on Computer Vision and Pa...

    2024

  4. [12]

    Shikra: Unleashing multimodal llm’s referential dialogue magic.arXiv preprint arXiv:2306.15195, 2023

    Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal llm’s referential dialogue magic.arXiv preprint arXiv:2306.15195, 2023

    2023 arXiv

  5. [13]

    Segment and track anything.arXiv preprint arXiv:2305.06558, 2023

    Yangming Cheng, Liulei Li, Yuanyou Xu, Xiaodi Li, Zongxin Yang, Wenguan Wang, and Yi Yang. Segment and track anything.arXiv preprint arXiv:2305.06558, 2023

    2023 arXiv

  6. [14]

    Total-text: A comprehensive dataset for scene text detection and recognition, 2017

    Chee Kheng Chng and Chee Seng Chan. Total-text: A comprehensive dataset for scene text detection and recognition, 2017

    2017

  7. [15]

    V ocabulary-free image classification, 2024

    Alessandro Conti, Enrico Fini, Massimiliano Mancini, Paolo Rota, Yiming Wang, and Elisa Ricci. V ocabulary-free image classification, 2024

    2024

  8. [16]

    Online action detection

    Roeland De Geest, Efstratios Gavves, Amir Ghodrati, Zhenyang Li, Cees Snoek, and Tinne Tuytelaars. Online action detection. InComputer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14, pages 269–284. Springer, 2016

    2016

  9. [17]

    Mevis: A large-scale benchmark for video segmentation with motion expressions, 2023

    Henghui Ding, Chang Liu, Shuting He, Xudong Jiang, and Chen Change Loy. Mevis: A large-scale benchmark for video segmentation with motion expressions, 2023

    2023

  10. [18]

    Actor and action video segmentation from a sentence

    Kirill Gavrilyuk, Amir Ghodrati, Zhenyang Li, and Cees GM Snoek. Actor and action video segmentation from a sentence. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 5958–5966, 2018

    2018

  11. [19]

    Icdar2017 robust reading challenge on coco-text

    Raul Gomez, Baoguang Shi, Lluis Gomez, Lukas Numann, Andreas Veit, Jiri Matas, Serge Belongie, and Dimosthenis Karatzas. Icdar2017 robust reading challenge on coco-text. In2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), volume 01, pages 14...

    2017

  12. [20]

    Ego4d: Around the world in 3,000 hours of egocentric video

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. InProceedings of the IEEE/CVF conference on computer vision a...

    2022

  13. [21]

    Regiongpt: Towards region understanding vision language model, 2024

    Qiushan Guo, Shalini De Mello, Hongxu Yin, Wonmin Byeon, Ka Chun Cheung, Yizhou Yu, Ping Luo, and Sifei Liu. Regiongpt: Towards region understanding vision language model, 2024

    2024

  14. [22]

    Trace: Temporal grounding video llm via causal event modeling.arXiv preprint arXiv:2410.05643, 2024

    Yongxin Guo, Jingyu Liu, Mingda Li, Qingbin Liu, Xi Chen, and Xiaoying Tang. Trace: Temporal grounding video llm via causal event modeling.arXiv preprint arXiv:2410.05643, 2024

    2024 arXiv

  15. [23]

    Lvis: A dataset for large vocabulary instance segmentation, 2019

    Agrim Gupta, Piotr Dollár, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation, 2019

    2019

  16. [24]

    Synthetic data for text localisation in natural images, 2016

    Ankush Gupta, Andrea Vedaldi, and Andrew Zisserman. Synthetic data for text localisation in natural images, 2016

    2016

  17. [25]

    Omni-rgpt: Unifying image and video region-level understanding via token marks, 2025

    Miran Heo, Min-Hung Chen, De-An Huang, Sifei Liu, Subhashree Radhakrishnan, Seon Joo Kim, Yu- Chiang Frank Wang, and Ryo Hachiuma. Omni-rgpt: Unifying image and video region-level understanding via token marks, 2025

    2025

  18. [26]

    Segment and caption anything

    Xiaoke Huang, Jianfeng Wang, Yansong Tang, Zheng Zhang, Han Hu, Jiwen Lu, Lijuan Wang, and Zicheng Liu. Segment and caption anything. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13405–13417, 2024

    2024

  19. [27]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  20. [28]

    Visual prompt tuning

    Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser- Nam Lim. Visual prompt tuning. InEuropean conference on computer vision, pages 709–727. Springer, 2022

    2022

  21. [29]

    Chatrex: Taming multimodal llm for joint perception and understanding.arXiv preprint arXiv:2411.18363, 2024

    Qing Jiang, Gen Luo, Yuqin Yang, Yuda Xiong, Yihao Chen, Zhaoyang Zeng, Tianhe Ren, and Lei Zhang. Chatrex: Taming multimodal llm for joint perception and understanding.arXiv preprint arXiv:2411.18363, 2024

    2024 arXiv

  22. [30]

    Icdar 2015 competition on robust reading

    Dimosthenis Karatzas, Lluis Gomez-Bigorda, Anguelos Nicolaou, Suman Ghosh, Andrew Bagdanov, Masakazu Iwamura, Jiri Matas, Lukas Neumann, Vijay Ramaseshan Chandrasekhar, Shijian Lu, et al. Icdar 2015 competition on robust reading. In2015 13th international conference on documen...

    2015

  23. [31]

    Icdar 2013 robust reading competition

    Dimosthenis Karatzas, Faisal Shafait, Seiichi Uchida, Masakazu Iwamura, Lluis Gomez i Bigorda, Sergi Robles Mestre, Joan Mas, David Fernandez Mota, Jon Almazàn Almazàn, and Lluís Pere de las Heras. Icdar 2013 robust reading competition. In2013 12th International Conference on ...

    2013

  24. [32]

    Referitgame: Referring to objects in photographs of natural scenes

    Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. InProceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 787–798, 2014

    2014

  25. [33]

    Segment anything in high quality.Advances in Neural Information Processing Systems, 36:29914–29934, 2023

    Lei Ke, Mingqiao Ye, Martin Danelljan, Yu-Wing Tai, Chi-Keung Tang, Fisher Yu, et al. Segment anything in high quality.Advances in Neural Information Processing Systems, 36:29914–29934, 2023

    2023

  26. [34]

    Segment anything.arXiv preprint arXiv:2304.02643, 2023

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything.arXiv preprint arXiv:2304.02643, 2023

    2023 arXiv

  27. [35]

    Openimages: A public dataset for large-scale multi-label and multi-class image classification.Dataset available from https://github

    Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, Sami Abu-El-Haija, Alina Kuznetsova, Hassan Rom, Jasper Uijlings, Stefan Popov, Andreas Veit, et al. Openimages: A public dataset for large-scale multi-label and multi-class image classification.Dataset available from ht...

    2017

  28. [36]

    Shamma, Michael S

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Fei-Fei Li. Visual genome: Connecting language and vision using crowdsourced dense image annotations, 2016. 12

    2016

  29. [37]

    Beyond mot: Semantic multi-object tracking

    Yunhao Li, Qin Li, Hao Wang, Xue Ma, Jiali Yao, Shaohua Dong, Heng Fan, and Libo Zhang. Beyond mot: Semantic multi-object tracking. InEuropean Conference on Computer Vision, pages 276–293. Springer, 2024

    2024

  30. [38]

    Describe anything: Detailed localized image and video captioning.arXiv preprint arXiv:2504.16072, 2025

    Long Lian, Yifan Ding, Yunhao Ge, Sifei Liu, Hanzi Mao, Boyi Li, Marco Pavone, Ming-Yu Liu, Trevor Darrell, Adam Yala, et al. Describe anything: Detailed localized image and video captioning.arXiv preprint arXiv:2504.16072, 2025

    2025 arXiv

  31. [39]

    Rouge: A package for automatic evaluation of summaries

    Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. InText summarization branches out, pages 74–81, 2004

    2004

  32. [40]

    Lawrence Zitnick, and Piotr Dollár

    Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects in context, 2015

    2015

  33. [41]

    Draw-and-understand: Leveraging visual prompts to enable mllms to comprehend what you want.arXiv preprint arXiv:2403.20271, 2024

    Weifeng Lin, Xinyu Wei, Ruichuan An, Peng Gao, Bocheng Zou, Yulin Luo, Siyuan Huang, Shang- hang Zhang, and Hongsheng Li. Draw-and-understand: Leveraging visual prompts to enable mllms to comprehend what you want.arXiv preprint arXiv:2403.20271, 2024

    2024 arXiv

  34. [42]

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  35. [43]

    Gres: Generalized referring expression segmentation

    Chang Liu, Henghui Ding, and Xudong Jiang. Gres: Generalized referring expression segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 23592–23601, 2023

    2023

  36. [44]

    Sphinx-x: Scaling data and parameters for a family of multi-modal large language models.arXiv preprint arXiv:2402.05935, 2024

    Dongyang Liu, Renrui Zhang, Longtian Qiu, Siyuan Huang, Weifeng Lin, Shitian Zhao, Shijie Geng, Ziyi Lin, Peng Jin, Kaipeng Zhang, et al. Sphinx-x: Scaling data and parameters for a family of multi-modal large language models.arXiv preprint arXiv:2402.05935, 2024

    2024 arXiv

  37. [45]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024

    2024

  38. [46]

    Kosmos-2: Grounding multimodal large language models to the world.arXiv preprint arXiv:2306.14824, 2023

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world.arXiv preprint arXiv:2306.14824, 2023

    2023 arXiv

  39. [47]

    The 2017 davis challenge on video object segmentation, 2018

    Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine-Hornung, and Luc Van Gool. The 2017 davis challenge on video object segmentation, 2018

    2017

  40. [48]

    Artemis: Towards referential understanding in complex videos

    Jihao Qiu, Yuan Zhang, Xi Tang, Lingxi Xie, Tianren Ma, Pengyu Yan, David Doermann, Qixiang Ye, and Yunjie Tian. Artemis: Towards referential understanding in complex videos. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  41. [49]

    Artemis: Towards referential understanding in complex videos, 2024

    Jihao Qiu, Yuan Zhang, Xi Tang, Lingxi Xie, Tianren Ma, Pengyu Yan, David Doermann, Qixiang Ye, and Yunjie Tian. Artemis: Towards referential understanding in complex videos, 2024

    2024

  42. [50]

    Paco: Parts and attributes of common objects, 2023

    Vignesh Ramanathan, Anmol Kalia, Vladan Petrovic, Yi Wen, Baixue Zheng, Baishan Guo, Rui Wang, Aaron Marquez, Rama Kovvuri, Abhishek Kadian, Amir Mousavi, Yiwen Song, Abhimanyu Dubey, and Dhruv Mahajan. Paco: Parts and attributes of common objects, 2023

    2023

  43. [51]

    Anwer, Erix Xing, Ming-Hsuan Yang, and Fahad S

    Hanoona Rasheed, Muhammad Maaz, Sahal Shaji Mullappilly, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M. Anwer, Erix Xing, Ming-Hsuan Yang, and Fahad S. Khan. Glamm: Pixel grounding large multimodal model, 2024

    2024

  44. [52]

    Sam 2: Segment anything in images and videos.arXiv preprint arXiv:2408.00714, 2024

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos.arXiv preprint arXiv:2408.00714, 2024

    2024 arXiv

  45. [53]

    Sam 2: Segment anything in images and videos, 2024

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenh...

    2024

  46. [54]

    Grounded sam: Assembling open-world models for diverse visual tasks.arXiv preprint arXiv:2401.14159, 2024

    Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, et al. Grounded sam: Assembling open-world models for diverse visual tasks.arXiv preprint arXiv:2401.14159, 2024. 13

    2024 arXiv

  47. [55]

    Objects365: A large-scale, high-quality dataset for object detection

    Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019

    2019

  48. [56]

    Textocr: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text, 2021

    Amanpreet Singh, Guan Pang, Mandy Toh, Jing Huang, Wojciech Galuba, and Tal Hassner. Textocr: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text, 2021

    2021

  49. [57]

    Icdar 2019 competition on large-scale street view text with partial labeling – rrc-lsvt, 2019

    Yipeng Sun, Zihan Ni, Chee-Kheng Chng, Yuliang Liu, Canjie Luo, Chun Chet Ng, Junyu Han, Errui Ding, Jingtuo Liu, Dimosthenis Karatzas, Chee Seng Chan, and Lianwen Jin. Icdar 2019 competition on large-scale street view text with partial labeling – rrc-lsvt, 2019

    2019

  50. [58]

    Human-centric spatio-temporal video grounding with visual transformers, 2021

    Zongheng Tang, Yue Liao, Si Liu, Guanbin Li, Xiaojie Jin, Hongxu Jiang, Qian Yu, and Dong Xu. Human-centric spatio-temporal video grounding with visual transformers, 2021

    2021

  51. [59]

    Human- centric spatio-temporal video grounding with visual transformers.IEEE Transactions on Circuits and Systems for Video Technology, 32(12):8238–8249, 2021

    Zongheng Tang, Yue Liao, Si Liu, Guanbin Li, Xiaojie Jin, Hongxu Jiang, Qian Yu, and Dong Xu. Human- centric spatio-temporal video grounding with visual transformers.IEEE Transactions on Circuits and Systems for Video Technology, 32(12):8238–8249, 2021

    2021

  52. [60]

    Cider: Consensus-based image description evaluation

    Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575, 2015

    2015

  53. [61]

    Coco-text: Dataset and benchmark for text detection and recognition in natural images, 2016

    Andreas Veit, Tomas Matera, Lukas Neumann, Jiri Matas, and Serge Belongie. Coco-text: Dataset and benchmark for text detection and recognition in natural images, 2016

    2016

  54. [62]

    Elysium: Exploring object-level perception in videos via mllm, 2024

    Han Wang, Yanjie Wang, Yongjie Ye, Yuxiang Nie, and Can Huang. Elysium: Exploring object-level perception in videos via mllm, 2024

    2024

  55. [63]

    Elysium: Exploring object-level perception in videos via mllm

    Han Wang, Yongjie Ye, Yanjie Wang, Yuxiang Nie, and Can Huang. Elysium: Exploring object-level perception in videos via mllm. InEuropean Conference on Computer Vision, pages 166–185. Springer, 2024

    2024

  56. [64]

    Towards open-vocabulary video instance segmentation, 2023

    Haochen Wang, Cilin Yan, Shuai Wang, Xiaolong Jiang, XU Tang, Yao Hu, Weidi Xie, and Efstratios Gavves. Towards open-vocabulary video instance segmentation, 2023

    2023

  57. [65]

    Git: A generative image-to-text transformer for vision and language, 2022

    Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to-text transformer for vision and language, 2022

    2022

  58. [66]

    V3det: Vast vocabulary visual detection dataset, 2023

    Jiaqi Wang, Pan Zhang, Tao Chu, Yuhang Cao, Yujie Zhou, Tong Wu, Bin Wang, Conghui He, and Dahua Lin. V3det: Vast vocabulary visual detection dataset, 2023

    2023

  59. [67]

    Caption anything: Interactive image description with diverse multimodal controls.arXiv preprint arXiv:2305.02677, 2023

    Teng Wang, Jinrui Zhang, Junjie Fei, Hao Zheng, Yunlong Tang, Zhe Li, Mingqi Gao, and Shanshan Zhao. Caption anything: Interactive image description with diverse multimodal controls.arXiv preprint arXiv:2305.02677, 2023

    2023 arXiv

  60. [68]

    The all-seeing project: Towards panoptic visual recognition and understanding of the open world.arXiv preprint arXiv:2308.01907, 2023

    Weiyun Wang, Min Shi, Qingyun Li, Wenhai Wang, Zhenhang Huang, Linjie Xing, Zhe Chen, Hao Li, Xizhou Zhu, Zhiguo Cao, et al. The all-seeing project: Towards panoptic visual recognition and understanding of the open world.arXiv preprint arXiv:2308.01907, 2023

    2023 arXiv

  61. [69]

    Grit: A generative region-to-text transformer for object understanding

    Jialian Wu, Jianfeng Wang, Zhengyuan Yang, Zhe Gan, Zicheng Liu, Junsong Yuan, and Lijuan Wang. Grit: A generative region-to-text transformer for object understanding. InEuropean Conference on Computer Vision, pages 207–224. Springer, 2024

    2024

  62. [70]

    Visionllm v2: An end-to-end generalist multimodal large language model for hundreds of vision-language tasks.Advances in Neural Information Processing Systems, 37:69925–69975, 2024

    Jiannan Wu, Muyan Zhong, Sen Xing, Zeqiang Lai, Zhaoyang Liu, Zhe Chen, Wenhai Wang, Xizhou Zhu, Lewei Lu, Tong Lu, et al. Visionllm v2: An end-to-end generalist multimodal large language model for hundreds of vision-language tasks.Advances in Neural Information Processing Sys...

    2024

  63. [71]

    Youtube-vos: A large-scale video object segmentation benchmark, 2018

    Ning Xu, Linjie Yang, Yuchen Fan, Dingcheng Yue, Yuchen Liang, Jianchao Yang, and Thomas Huang. Youtube-vos: A large-scale video object segmentation benchmark, 2018

    2018

  64. [72]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  65. [73]

    Vid2seq: Large-scale pretraining of a visual language model for dense video captioning, 2023

    Antoine Yang, Arsha Nagrani, Paul Hongsuck Seo, Antoine Miech, Jordi Pont-Tuset, Ivan Laptev, Josef Sivic, and Cordelia Schmid. Vid2seq: Large-scale pretraining of a visual language model for dense video captioning, 2023. 14

    2023

  66. [74]

    Samurai: Adapting segment anything model for zero-shot visual tracking with motion-aware memory, 2024

    Cheng-Yen Yang, Hsiang-Wei Huang, Wenhao Chai, Zhongyu Jiang, and Jenq-Neng Hwang. Samurai: Adapting segment anything model for zero-shot visual tracking with motion-aware memory, 2024

    2024

  67. [75]

    Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v, 2023

    Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v, 2023

    2023

  68. [76]

    Detecting texts of arbitrary orientations in natural images

    Cong Yao, Xiang Bai, Wenyu Liu, Yi Ma, and Zhuowen Tu. Detecting texts of arbitrary orientations in natural images. In2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 1083–1090, 2012

    2012

  69. [77]

    Ferret: Refer and ground anything anywhere at any granularity.arXiv preprint arXiv:2310.07704, 2023

    Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity.arXiv preprint arXiv:2310.07704, 2023

    2023 arXiv

  70. [78]

    Merlin: Empowering multimodal llms with foresight minds

    En Yu, Liang Zhao, Yana Wei, Jinrong Yang, Dongming Wu, Lingyu Kong, Haoran Wei, Tiancai Wang, Zheng Ge, Xiangyu Zhang, et al. Merlin: Empowering multimodal llms with foresight minds. InEuropean Conference on Computer Vision, pages 425–443. Springer, 2024

    2024

  71. [79]

    Sa2va: Marrying sam2 with llava for dense grounded understanding of images and videos, 2025

    Haobo Yuan, Xiangtai Li, Tao Zhang, Zilong Huang, Shilin Xu, Shunping Ji, Yunhai Tong, Lu Qi, Jiashi Feng, and Ming-Hsuan Yang. Sa2va: Marrying sam2 with llava for dense grounded understanding of images and videos, 2025

    2025

  72. [80]

    Osprey: Pixel understanding with visual instruction tuning

    Yuqian Yuan, Wentong Li, Jian Liu, Dongqi Tang, Xinjie Luo, Chi Qin, Lei Zhang, and Jianke Zhu. Osprey: Pixel understanding with visual instruction tuning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 28202–28211, 2024

    2024

  73. [81]

    Videorefer suite: Advancing spatial-temporal object understanding with video llm.arXiv preprint arXiv:2501.00599, 2024

    Yuqian Yuan, Hang Zhang, Wentong Li, Zesen Cheng, Boqiang Zhang, Long Li, Xin Li, Deli Zhao, Wenqiao Zhang, Yueting Zhuang, et al. Videorefer suite: Advancing spatial-temporal object understanding with video llm.arXiv preprint arXiv:2501.00599, 2024

    2024 arXiv

  74. [82]

    Faster segment anything: Towards lightweight sam for mobile applications.arXiv preprint arXiv:2306.14289, 2023

    Chaoning Zhang, Dongshen Han, Yu Qiao, Jung Uk Kim, Sung-Ho Bae, Seungkyu Lee, and Choong Seon Hong. Faster segment anything: Towards lightweight sam for mobile applications.arXiv preprint arXiv:2306.14289, 2023

    2023 arXiv

  75. [83]

    Ferret-v2: An improved baseline for referring and grounding with large language models.arXiv preprint arXiv:2404.07973, 2024

    Haotian Zhang, Haoxuan You, Philipp Dufter, Bowen Zhang, Chen Chen, Hong-You Chen, Tsu-Jui Fu, William Yang Wang, Shih-Fu Chang, Zhe Gan, et al. Ferret-v2: An improved baseline for referring and grounding with large language models.arXiv preprint arXiv:2404.07973, 2024

    2024 arXiv

  76. [84]

    Gpt4roi: Instruction tuning large language model on region-of-interest, 2025

    Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Yu Liu, Kai Chen, and Ping Luo. Gpt4roi: Instruction tuning large language model on region-of-interest, 2025

    2025

  77. [85]

    Where does it exist: Spatio-temporal video grounding for multi-form sentences

    Zhu Zhang, Zhou Zhao, Yang Zhao, Qi Wang, Huasheng Liu, and Lianli Gao. Where does it exist: Spatio-temporal video grounding for multi-form sentences. InCVPR, 2020

    2020

  78. [86]

    Chatspot: Bootstrapping multimodal llms via precise referring instruction tuning

    Liang Zhao, En Yu, Zheng Ge, Jinrong Yang, Haoran Wei, Hongyu Zhou, Jianjian Sun, Yuang Peng, Runpei Dong, Chunrui Han, et al. Chatspot: Bootstrapping multimodal llms via precise referring instruction tuning. arXiv preprint arXiv:2307.09474, 2023

    2023 arXiv

  79. [87]

    Fast segment anything.arXiv preprint arXiv:2306.12156, 2023

    Xu Zhao, Wenchao Ding, Yongqi An, Yinglong Du, Tao Yu, Min Li, Ming Tang, and Jinqiao Wang. Fast segment anything.arXiv preprint arXiv:2306.12156, 2023

    2023 arXiv

  80. [88]

    Controlcap: Controllable region-level captioning

    Yuzhong Zhao, Yue Liu, Zonghao Guo, Weijia Wu, Chen Gong, Qixiang Ye, and Fang Wan. Controlcap: Controllable region-level captioning. InEuropean Conference on Computer Vision, pages 21–38. Springer, 2024

    2024

  81. [89]

    Streaming dense video captioning

    Xingyi Zhou, Anurag Arnab, Shyamal Buch, Shen Yan, Austin Myers, Xuehan Xiong, Arsha Nagrani, and Cordelia Schmid. Streaming dense video captioning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18243–18252, 2024. 15 Appendix A Conf...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.