Pith. sign in

REVIEW 4 major objections 5 minor 3 cited by

DenseWorld-1M: Towards Detailed Dense Grounded Caption in the Real World

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read DenseWorld-1M is a one-million-image dataset with detailed grounded captions for every object, and fine-tuning multimodal large language models on it improves their grounding and understanding.

desk verdict A genuinely new dense grounded caption dataset at scale, but the quality claim needs a human error audit before the community can trust the 400k SCM-written captions. read the letter →

arxiv 2506.24102 v1 pith:OHPME3WU submitted 2025-06-30 cs.CV

Xiangtai Li , Tao Zhang , Yanwei Li , Haobo Yuan , Shihao Chen , Yikang Zhou , Jiahao Meng , Yueyi Sun
show 8 more authors
Shilin Xu Lu Qi Tianheng Cheng Yi Lin Zilong Huang Wenhao Huang Jiashi Feng Guang Shi
This is my paper
classification cs.CV
keywords densegroundedcaptioningmultimodallargelanguagemodelsreferringsegmentationregionvisualgroundingautomaticdatasetlabelinghigh-resolutionimagespixel-levelunderstanding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to build DenseWorld-1M, a real-world dataset of one million high-resolution images in which each object carries a detailed caption, a segmentation mask, and a place in a dense grounded scene caption. It argues this is the first dataset at this scale to combine detailed object-level descriptions with pixel-level grounding and spatial relations. It shows that fine-tuning on this data improves existing multimodal large language models on referring segmentation, grounded conversation generation, region captioning, and several general visual question-answering benchmarks. The paper also presents two trained models, a Detailed Region Caption model and a Spatial Caption Merging model, that accelerate the labeling pipeline while producing captions close in quality to the teacher pipeline.

What carries the argument

The load-bearing mechanism is the three-stage labeling pipeline. Stage 1 merges and refines masks from SAM and APE, guided by RAM++ tags, to decompose a scene into entity-level objects. Stage 2 crops each object, sends the crop to InternVL-2.5 78B for a brief description, then prompts the same model with a visual prompt overlaid on the original image to produce a detailed object caption, with Qwen2.5-VL filtering inconsistent ones. Stage 3 splits complex scenes into sub-images, generates grounded sub-image captions, and merges them into one final dense grounded caption with object IDs tied to masks. The two trained models carry this pipeline: DRC, a 3B region caption model that combines sub-image crops with ID patch embeddings overlaid on vision features, and SCM, a fine-tuned InternVL3-8B model that merges object captions into the final dense caption in a single pass.

What would settle it

Take a random sample of DenseWorld-1M images, have human annotators mark every object and write reference captions, then measure per-object caption accuracy and mask-to-reference alignment; if SCM-labeled images show systematically higher rates of object-count errors, hallucinated attributes, or mismatched object IDs than the InternVL-labeled subset, the main assumption fails.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that detailed, dense, pixel-grounded captions can be produced automatically at scale by decomposing the labeling problem into three stages: open-world perception, per-object caption generation, and scene-level caption merging. The resulting dataset contains 1M images, 23.1M object captions, and 23.6M masks, with scene captions averaging about 458 words and object captions about 111 words. Fine-tuning on this data improves Sa2VA by roughly 0.5 to 1.0 points on referring expression segmentation, improves grounded conversation generation on AP50, mIoU, and recall, and improves LLaVA and Qwen2.5-VL on multiple standard multimodal benchmarks. The paper also claims that even current state-of-the-art closed models cannot generate captions of this density and grounding quality when prompted directly.

Load-bearing premise

The pipeline's output is clean enough to serve as training ground truth: the paper never human-verifies the 400k image captions produced by the trained SCM, which learned from InternVL-generated labels filtered only by Qwen2.5-VL, so any systematic errors in those labels are baked into the dataset.

Editorial extensions

If this is right

  • Fine-tuning Sa2VA on DenseWorld-1M yields about 0.5 to 1.0 point gains on referring segmentation across RefCOCO, RefCOCO+, and RefCOCOg.
  • Grounded conversation generation improves on mask-based and recall metrics (AP50, mIoU, Recall), indicating stronger text-to-mask alignment after training on the dataset.
  • LLaVA trained from scratch with DenseWorld-1M improves on MMBench, MME, MMStar, SEEDBench, AI2D, MMVP, HallusionBench, and MMMU.
  • Qwen2.5-VL post-trained with DenseWorld-1M mixed with instruction data improves on most evaluated multimodal benchmarks.
  • The DRC model surpasses prior region captioning models on RefCOCOg CIDEr and METEOR without dedicated pretraining stages, and SCM cuts scene-level labeling time from about 2.6 minutes to 31 seconds per image.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: because SCM labels about 40% of the dataset and was trained on InternVL-generated labels filtered only by Qwen2.5-VL, a human-audited subset would be a natural check on whether systematic hallucination or mask-mismatch errors are silently amplified in that portion.
  • Beyond the paper: the sub-image splitting and merging strategy in stage 3 suggests a scalable recipe for even denser scenes, video frames, or the paper's stated 10M-image target, with the small merging model eventually replacing the 78B teacher in the loop.
  • Beyond the paper: the dataset's grounding structure makes it plausible for chain-of-thought or step-by-step visual reasoning training, but the paper only gestures at this O3-like use, so its value there remains an untested extension.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces DenseWorld-1M, a claimed first large-scale, dense, grounded caption dataset for real-world images, containing 1M high-resolution images, 23.1M object captions, and 23.6M masks. The dataset is produced by a fully automatic three-stage pipeline: stage-1 pixel-level masks from SAM/APE/RAM++ with merging and refinement; stage-2 object-level detailed captions from InternVL-2.5-78B with Qwen2.5-VL verification; and stage-3 scene-level dense grounded captions from InternVL with a sub-image merging strategy. Two lightweight models are trained to accelerate labeling: the Detailed Region Caption (DRC) model and the Spatial Caption Merging (SCM/SPM) model, with the latter used to annotate 400k images without direct large-model inference. Experiments report improvements after fine-tuning with DenseWorld-1M on Sa2VA for referring segmentation and grounded conversation generation, and on LLaVA and Qwen2.5-VL for several MLLM benchmarks, plus a user study for SPM and efficiency measurements.

Significance. If the dataset labels are accurate and grounded, DenseWorld-1M would be a substantial community resource: it is large (1M images), detailed (average 458 words and 20.5 sentences per scene caption), and provides pixel-level masks plus object-level captions, filling a real gap between dense text-only captions (DenseFusion) and grounded but sparse region-level datasets (GLaMM, Osprey). The DRC/SCM models are also practically useful, with reported speedups (DRC ~3x and SCM ~5x over the 78B pipeline). The paper ships quantitative efficiency numbers and releases code/models, which aids reproducibility. However, the value of the resource is conditional on label quality, and that condition is not yet demonstrated at the claimed scale.

major comments (4)
  1. [Secs. 3.2, 3.3, and 4.2] The label-quality loop lacks independent verification. Stage-2 captions are filtered only by Qwen2.5-VL-72B, stage-3 captions are produced by InternVL-2.5-78B with no consistency check, and the SCM model trained on 600k of these outputs is then used to annotate 400k additional images without any human check. The only reported human evaluation is a 100-example preference study (Sec. 5.1) that asks which of two outputs is 'better', not whether the captions are factually correct, mask-consistent, or free of object-ID/text mismatches. A human error-rate audit (e.g., on ~500 images measuring object hallucination, attribute binding, and <obj i> token-to-mask consistency) is needed to support the central claim that DenseWorld-1M is an accurate grounded caption dataset, especially for the SCM-produced 40% partition.
  2. [Sec. 5.1 and Tab. 4; Appendix D] The claim of 'consistent improvements over eight different datasets' for Qwen2.5-VL is contradicted by Tab. 4, where MME drops from 2200 to 2175 after adding DenseWorld-1M. Moreover, the Qwen2.5-VL SFT mixes DenseWorld-1M with MAmmoTH-VL-Instruct-12M at a 1:2 ratio without a control run using only MAmmoTH data; therefore the observed gains cannot be attributed to DenseWorld-1M alone. Please report a MAmmoTH-only SFT baseline or an ablation over mixing ratios.
  3. [Sec. 5.1, Tabs. 2-3, Appendix D] The Sa2VA improvements are small (0.5-1.0 points on RefCOCO and variable on GCG), no error bars or significance tests are reported, and the protocol in Appendix D says models trained on DenseWorld data are 'further fine-tuned using the training sets of downstream tasks' to bridge the domain gap. Without applying the identical extra fine-tuning to the baseline or reporting standard deviations over seeds, the observed deltas are not clearly distinguishable from training noise.
  4. [Secs. 3.3, 4.2, 5.1, and Fig. 4] The merging model is inconsistently named: Sec. 3.3 and Sec. 4.2 call it SCM, while Sec. 5.1 and Fig. 4 call it SPM, and the caption of Fig. 4 says 'Spatial Caption Merging model'. This makes it difficult to determine which model is evaluated in the user study and which model produced the 400k annotations. Please standardize the name and explicitly state the model identity in the user-study paragraph.
minor comments (5)
  1. [Sec. 5.1] Typo: 'DesneWorld-1M' should be 'DenseWorld-1M' in the first paragraph.
  2. [Tab. 1 footnote] Typo in footnote 4: 'per obejcts' should be 'per objects'.
  3. [References] References [12] and [13] are the same paper ('Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling'); duplicate citations should be merged.
  4. [Sec. 3.2] The verification step with Qwen2.5-VL-72B is described only as 'filter out all inconsistent object captions'; the prompt, the notion of consistency, and any threshold (e.g., a score or a yes/no criterion) are not specified in the main text or appendix.
  5. [Sec. 4.1] The DRC evaluation in Tab. 6 uses only the Osprey-724K dataset for fair comparison, which is sensible, but the main text should clarify that DRC was not trained on DenseWorld-1M in this experiment, to avoid confusion with the dataset claims.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: DenseWorld-1M is validated on external benchmarks; the unverified DRC/SCM labeling loop is a quality risk, not a circular derivation.

full rationale

The paper's central claim is that DenseWorld-1M, as training data, improves MLLMs on referring segmentation, grounded conversation generation, region captioning, and general vision-language benchmarks (Tables 2-6). These evaluations use external benchmarks and task-specific test sets that are independent of the automatic labeling pipeline, so the claimed gains are not forced by construction from the pipeline's own outputs. Table 6 in particular evaluates DRC on RefCOCOg using Osprey-724K data without DenseWorld-1M, providing an independent check of that model. Self-citations to prior work such as Sa2VA and OMG-LLaVA are used as baselines and background, not as a load-bearing uniqueness theorem, so they do not create circularity. The only self-referential part of the pipeline is the training of DRC and SCM: Sec. 4.2 states SCM is trained by directly using stage-3 pipeline outputs as supervised targets, and Sec. 3.3 says SCM then annotates 400k additional images, with the pipeline explicitly designed to have 'no human costs in the loop' (Fig. 2 caption). This means the 400k SCM-produced captions are unverified against independent ground truth, and the only SCM check is a 100-sample user study that compares SCM with its own stage-3 teacher rather than with human fact-checked annotations. That is a genuine data-quality and verification limitation, and it should be addressed if the dataset's accuracy claims are to be trusted. However, it is not a circular derivation in the technical sense: the 400k images are new inputs, the SCM outputs are not defined in terms of the benchmark numbers, and no equation or fitted parameter is renamed as a prediction. The downstream improvements are measured on benchmarks that are not part of the labeling loop. Under the review rule requiring an exhibited reduction from a claimed result to its own input, no such reduction is present, so the appropriate finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No fundamental physics or mathematical claims are made; the experimental burden is entirely empirical. The pipeline rests on assumptions about the accuracy of the perception and captioning models used as labelers, and on the fairness of benchmark controls. The engineering thresholds listed above are hand-chosen and unevaluated, and the 40% student-annotated portion introduces a self-referential quality risk.

free parameters (4)
  • Mask merge IOU threshold = 0.5
    Chosen by hand in Sec. 3.1 to decide when a mask contained in another is merged; this controls the final object set and therefore all downstream captions.
  • Simple/complex scene object threshold = 15 objects
    Chosen by hand in Sec. 3.3 to decide when stage-3 uses sub-image splitting; this affects caption density and grounded consistency for complex scenes.
  • DenseWorld to MAmmoTH SFT mix ratio = 1:2
    Chosen by hand in Appendix D for Qwen2.5-VL post-training; no ablation is given, so the benchmark improvements cannot be cleanly attributed to DenseWorld-1M alone.
  • Fraction of data annotated by DRC/SCM = 40%
    Chosen by hand in Sec. 3.3 and Sec. 4.2; the quality of the final 400k captions depends on two student models trained on the same pipeline's outputs, and is not independently audited.
assumptions (4)
  • domain assumption APE + SAM mask union with merge and NMS provides complete and accurate entity segmentation.
    Stage-1 relies on these models; mask errors propagate to object captions and grounded captions. See Sec. 3.1 and Figure 2.
  • domain assumption InternVL2.5-78B generates accurate, unbiased object descriptions when prompted with crops, masks, and visual prompts.
    Stage-2 and stage-3 use InternVL as the primary labeler. See Sec. 3.2 and Sec. 3.3. No human audit of the 600k teacher-generated captions is reported.
  • domain assumption Automatic verification with Qwen2.5-VL removes inconsistent captions without introducing selection bias.
    Sec. 3.2 states Qwen2.5-VL 72B is used to 'filter out all inconsistent object captions', but no precision, recall, or failure analysis of this filter is given.
  • domain assumption Improvements on external benchmarks transfer from fine-tuning on DenseWorld plus additional SFT data.
    The Qwen2.5-VL experiment mixes DenseWorld with MAmmoTH at 1:2 and compares to the original model, not to a control trained on MAmmoTH alone. See Appendix D.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DenseWorld-1M: Towards Detailed Dense Grounded Caption in the Real World." pith.science (2026). https://pith.science/paper/OHPME3WU

@misc{pith2026250624102,
  author       = {Pith},
  title        = {Pith review of: DenseWorld-1M: Towards Detailed Dense Grounded Caption in the Real World},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OHPME3WU}},
  note         = {Machine review of arXiv:2506.24102}
}
read the original abstract

Multimodal Large Language Models (MLLMs) demonstrate a complex understanding of scenes, benefiting from large-scale and high-quality datasets. Most existing caption datasets lack the ground locations and relations for visual entities. Several grounded caption datasets face the problems of missing detailed descriptions, relations, and massive object descriptions on high-resolution images. To fill this gap for the community, we present DenseWorld-1M, the first massive, detailed, dense grounded caption dataset in the real world. We design a three-stage labeling pipeline, containing open-world perception, detailed object caption generation, and dense caption merging. The first stage obtains entity-level masks and labels. The second stage generates the object-level, detailed captions with the guidance of masks and labels from the first stage. The final stage merges object captions and masks into spatial and relational dense captions. To accelerate the labeling process and improve caption quality, we present two VLM models: the Detailed Region Caption model and the Spatial Caption Merging model. Extensive experiments on various settings, including vision-language understanding, visual grounding, and region caption generation, demonstrate the effectiveness of our DenseWorld-1M dataset and labeling models.

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Actor as Its Own Critic: Unifying Region Understanding and Localization via CycleGRPO

    cs.CV 2026-07 accept novelty 7.0 of 10

    A single MLLM jointly improves region captioning and localization by rewarding captions that let it reconstruct the original mask, needing only region inputs.

  2. Kwai Keye-VL 1.5 Technical Report

    cs.CV 2025-09 conditional novelty 6.0 of 10

    Keye-VL-1.5 combines similarity-based Slow-Fast video token allocation with progressive context extension and iterative RL, reporting leading video-understanding results among 8B-scale multimodal models.

  3. Kwai Keye-VL Technical Report

    cs.CV 2025-07 conditional novelty 6.0 of 10

    Kwai Keye-VL shows that a five-mode chain-of-thought cold-start plus mix-mode reinforcement learning can push an 8B multimodal model to strong short-video and general vision-language performance.

Reference graph

Works this paper leans on

101 extracted references · 27 canonical work pages · cited by 3 Pith papers

  1. [1]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    arXiv 2025

  2. [2]

    Forest-of-thought: Scaling test-time compute for enhancing llm reasoning.arXiv preprint arXiv:2412.09078, 2024

    Zhenni Bi, Kai Han, Chuanjian Liu, Yehui Tang, and Yunhe Wang. Forest-of-thought: Scaling test-time compute for enhancing llm reasoning.arXiv preprint arXiv:2412.09078, 2024

    arXiv 2024

  3. [3]

    End-to-end object detection with transformers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. InECCV, 2020

    2020

  4. [4]

    Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts

    Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. InCVPR, 2021

    2021

  5. [5]

    Allava: Harnessing gpt4v-synthesized data for lite vision-language models

    Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: Harnessing gpt4v-synthesized data for lite vision-language models. arXiv preprint arXiv:2402.11684, 2024

    arXiv 2024

  6. [6]

    Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.IEEE TPAMI, 2017

    Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.IEEE TPAMI, 2017

    2017

  7. [7]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. InECCV, 2024

    2024

  8. [8]

    Are we on the right way for evaluating large vision-language models?arXiv preprint arXiv:2403.20330, 2024

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models?arXiv preprint arXiv:2403.20330, 2024

    arXiv 2024

Show all 101 references
  1. [9]

    Mmict: Boosting multi-modal fine-tuning with in-context examples.ACM TOMM, 2024

    Tao Chen, Enwei Zhang, Yuting Gao, Ke Li, Xing Sun, Yan Zhang, Hui Li, and Rongrong Ji. Mmict: Boosting multi-modal fine-tuning with in-context examples.ACM TOMM, 2024

    2024

  2. [10]

    A generalist framework for panoptic segmentation of images and videos

    Ting Chen, Lala Li, Saurabh Saxena, Geoffrey Hinton, and David J Fleet. A generalist framework for panoptic segmentation of images and videos. InICCV, 2023

    2023

  3. [11]

    Microsoft coco captions: Data collection and evaluation server.arXiv preprint arXiv:1504.00325, 2015

    Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server.arXiv preprint arXiv:1504.00325, 2015

    2015 arXiv

  4. [13]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271, 2024

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271, 2024

    2024 arXiv

  5. [14]

    Lmdeploy: A toolkit for compressing, deploying, and serving llm.https://github

    LMDeploy Contributors. Lmdeploy: A toolkit for compressing, deploying, and serving llm.https://github. com/InternLM/lmdeploy, 2023

    2023

  6. [15]

    Sharegpt-4o: Comprehensive multimodal annotations with gpt-4o, 2024

    Erfei Cui, Yinan He, Zheng Ma, Zhe Chen, Hao Tian, Weiyun Wang, Kunchang Li, Yi Wang, Wenhai Wang, Xizhou Zhu, Lewei Lu, Tong Lu, Yali Wang, Limin Wang, Yu Qiao, and Jifeng Dai. Sharegpt-4o: Comprehensive multimodal annotations with gpt-4o, 2024. URLhttps://sharegpt4o.github.io/

    2024

  7. [16]

    Coconut-pancap: Joint panoptic segmentation and grounded captions for fine-grained understanding and generation

    Xueqing Deng, Qihang Yu, Ali Athar, Chenglin Yang, Linjie Yang, Xiaojie Jin, Xiaohui Shen, and Liang-Chieh Chen. Coconut-pancap: Joint panoptic segmentation and grounded captions for fine-grained understanding and generation. arXiv preprint arXiv:2502.02589, 2025

    2025 arXiv

  8. [17]

    Open-vocabulary universal image segmentation with maskclip

    Zheng Ding, Jieke Wang, and Zhuowen Tu. Open-vocabulary universal image segmentation with maskclip. In ICML, 2023

    2023

  9. [18]

    On path to multimodal generalist: General-level and general-bench.ICML, 2025

    Hao Fei, Yuan Zhou, Juncheng Li, Xiangtai Li, Qingshan Xu, Bobo Li, Shengqiong Wu, Yaoting Wang, Junbao Zhou, Jiahao Meng, et al. On path to multimodal generalist: General-level and general-bench.ICML, 2025

    2025

  10. [19]

    Align-kd: Distilling cross-modal alignment knowledge for mobile vision-language model

    Qianhan Feng, Wenshuo Li, Tong Lin, and Xinghao Chen. Align-kd: Distilling cross-modal alignment knowledge for mobile vision-language model. InCVPR, 2025. 23

    2025

  11. [20]

    Mme: A comprehensive evaluation benchmark for multimodal large language models.arXiv preprint arXiv:2306.13394, 2023

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models.arXiv preprint arXiv:2306.13394, 2023

    2023 arXiv

  12. [21]

    Vita-1.5: Towards gpt-4o level real-time vision and speech interaction.arXiv preprint arXiv:2501.01957, 2025

    Chaoyou Fu, Haojia Lin, Xiong Wang, Yi-Fan Zhang, Yunhang Shen, Xiaoyu Liu, Haoyu Cao, Zuwei Long, Heting Gao, Ke Li, et al. Vita-1.5: Towards gpt-4o level real-time vision and speech interaction.arXiv preprint arXiv:2501.01957, 2025

    2025 arXiv

  13. [22]

    Imageinwords: Unlocking hyper-detailed image descriptions

    Roopal Garg, Andrea Burns, Burcu Karagol Ayan, Yonatan Bitton, Ceslee Montgomery, Yasumasa Onoe, Andrew Bunner, Ranjay Krishna, Jason Baldridge, and Radu Soricut. Imageinwords: Unlocking hyper-detailed image descriptions. arXiv preprint arXiv:2405.02793, 2024

    2024 arXiv

  14. [23]

    Fast R-CNN

    Ross Girshick. Fast R-CNN. InICCV, 2015

    2015

  15. [24]

    Mammoth-vl: Eliciting multimodal reasoning with instruction tuning at scale.arXiv preprint arXiv:2412.05237, 2024

    Jarvis Guo, Tuney Zheng, Yuelin Bai, Bo Li, Yubo Wang, King Zhu, Yizhi Li, Graham Neubig, Wenhu Chen, and Xiang Yue. Mammoth-vl: Eliciting multimodal reasoning with instruction tuning at scale.arXiv preprint arXiv:2412.05237, 2024

    2024 arXiv

  16. [25]

    Free video-llm: Prompt-guided visual perception for efficient training-free video llms.arXiv preprint arXiv:2410.10441, 2024

    Kai Han, Jianyuan Guo, Yehui Tang, Wei He, Enhua Wu, and Yunhe Wang. Free video-llm: Prompt-guided visual perception for efficient training-free video llms.arXiv preprint arXiv:2410.10441, 2024

    2024 arXiv

  17. [26]

    Mask R-CNN

    Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. InICCV, 2017

    2017

  18. [27]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. InICLR, 2022

    2022

  19. [28]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  20. [29]

    A diagram is worth a dozen images

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. InECCV, 2016

    2016

  21. [30]

    Panoptic segmentation

    Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár. Panoptic segmentation. In CVPR, 2019

    2019

  22. [31]

    Segment anything

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. InICCV, 2023

    2023

  23. [32]

    Dense-captioning events in videos

    Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. In ICCV, 2017

    2017

  24. [33]

    Visual genome: Connecting language and vision using crowdsourced dense image annotations.IJCV, 2017

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations.IJCV, 2017

    2017

  25. [34]

    Lisa: Reasoning segmentation via large language model

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. InCVPR, 2024

    2024

  26. [35]

    Seed-bench: Benchmarking multimodal llms with generative comprehension.arXiv preprint arXiv:2307.16125, 2023

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Benchmarking multimodal llms with generative comprehension.arXiv preprint arXiv:2307.16125, 2023

    2023 arXiv

  27. [36]

    Semantic flow for fast and accurate scene parsing

    Xiangtai Li, Ansheng You, Zhen Zhu, Houlong Zhao, Maoke Yang, Kuiyuan Yang, Shaohua Tan, and Yunhai Tong. Semantic flow for fast and accurate scene parsing. InECCV, 2020

    2020

  28. [37]

    Tube-link: A flexible cross tube baseline for universal video segmentation

    Xiangtai Li, Haobo Yuan, Wenwei Zhang, Guangliang Cheng, Jiangmiao Pang, and Chen Change Loy. Tube-link: A flexible cross tube baseline for universal video segmentation. InICCV, 2023

    2023

  29. [38]

    Transformer-based visual segmentation: A survey.IEEE TPAMI, 2024

    Xiangtai Li, Henghui Ding, Haobo Yuan, Wenwei Zhang, Jiangmiao Pang, Guangliang Cheng, Kai Chen, Ziwei Liu, and Chen Change Loy. Transformer-based visual segmentation: A survey.IEEE TPAMI, 2024

    2024

  30. [39]

    Panopticpartformer++: A unified and decoupled view for panoptic part segmentation

    Xiangtai Li, Shilin Xu, Yibo Yang, Haobo Yuan, Guangliang Cheng, Yunhai Tong, Zhouchen Lin, Ming-Hsuan Yang, and Dacheng Tao. Panopticpartformer++: A unified and decoupled view for panoptic part segmentation. IEEE TPAMI, 2024

    2024

  31. [40]

    Omg-seg: Is one model good enough for all segmentation? InCVPR, 2024

    Xiangtai Li, Haobo Yuan, Wei Li, Henghui Ding, Size Wu, Wenwei Zhang, Yining Li, Kai Chen, and Chen Change Loy. Omg-seg: Is one model good enough for all segmentation? InCVPR, 2024. 24

    2024

  32. [41]

    Densefusion-1m: Merging vision experts for comprehensive multimodal perception

    Xiaotong Li, Fan Zhang, Haiwen Diao, Yueze Wang, Xinlong Wang, and Lingyu Duan. Densefusion-1m: Merging vision experts for comprehensive multimodal perception. InNeurIPS, 2024

    2024

  33. [42]

    Mini-gemini: Mining the potential of multi-modality vision language models.arXiv preprint arXiv:2403.18814, 2024

    Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models.arXiv preprint arXiv:2403.18814, 2024

    2024 arXiv

  34. [43]

    Rain: Your language models can align themselves without finetuning.arXiv preprint arXiv:2309.07124, 2023

    Yuhui Li, Fangyun Wei, Jinjing Zhao, Chao Zhang, and Hongyang Zhang. Rain: Your language models can align themselves without finetuning.arXiv preprint arXiv:2309.07124, 2023

    2023 arXiv

  35. [44]

    Video-llava: Learning united visual representation by alignment before projection.arXiv preprint arXiv:2311.10122, 2023

    Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection.arXiv preprint arXiv:2311.10122, 2023

    2023 arXiv

  36. [45]

    Draw-and-understand: Leveraging visual prompts to enable mllms to comprehend what you want

    Weifeng Lin, Xinyu Wei, Ruichuan An, Peng Gao, Bocheng Zou, Yulin Luo, Siyuan Huang, Shanghang Zhang, and Hongsheng Li. Draw-and-understand: Leveraging visual prompts to enable mllms to comprehend what you want. arXiv preprint arXiv:2403.20271, 2024

    2024 arXiv

  37. [46]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. InNeurIPS, 2023

    2023

  38. [47]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR, 2024

    2024

  39. [48]

    Llava-next: Improved reasoning, ocr, and world knowledge, January 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024. URL https://llava-vl.github.io/blog/ 2024-01-30-llava-next/

    2024

  40. [49]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In ECCV, 2024

    2024

  41. [50]

    Mmbench: Is your multi-modal model an all-around player? InECCV, 2024

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? InECCV, 2024

    2024

  42. [51]

    Cheap and quick: Efficient vision-language instruction tuning for large language models.NeurIPS, 2023

    Gen Luo, Yiyi Zhou, Tianhe Ren, Shengxin Chen, Xiaoshuai Sun, and Rongrong Ji. Cheap and quick: Efficient vision-language instruction tuning for large language models.NeurIPS, 2023

    2023

  43. [52]

    Feast your eyes: Mixture-of- resolution adaptation for multimodal large language models.arXiv preprint arXiv:2403.03003, 2024

    Gen Luo, Yiyi Zhou, Yuxin Zhang, Xiawu Zheng, Xiaoshuai Sun, and Rongrong Ji. Feast your eyes: Mixture-of- resolution adaptation for multimodal large language models.arXiv preprint arXiv:2403.03003, 2024

    2024 arXiv

  44. [53]

    Video-rag: Visually-aligned retrieval-augmented long video comprehension.arXiv preprint arXiv:2411.13093, 2024

    Yongdong Luo, Xiawu Zheng, Xiao Yang, Guilin Li, Haojia Lin, Jinfa Huang, Jiayi Ji, Fei Chao, Jiebo Luo, and Rongrong Ji. Video-rag: Visually-aligned retrieval-augmented long video comprehension.arXiv preprint arXiv:2411.13093, 2024

    2024

  45. [54]

    Mllm-selector: Necessity and diversity-driven high-value data selection for enhanced visual instruction tuning.arXiv preprint arXiv:2503.20502, 2025

    Yiwei Ma, Guohai Xu, Xiaoshuai Sun, Jiayi Ji, Jie Lou, Debing Zhang, and Rongrong Ji. Mllm-selector: Necessity and diversity-driven high-value data selection for enhanced visual instruction tuning.arXiv preprint arXiv:2503.20502, 2025

    2025 arXiv

  46. [55]

    Generation and comprehension of unambiguous object descriptions

    Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. InCVPR, 2016

    2016

  47. [56]

    Docvqa: A dataset for vqa on document images

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In WACV, 2021

    2021

  48. [57]

    Docci: Descriptions of connected and contrasting images

    Yasumasa Onoe, Sunayana Rane, Zachary Berger, Yonatan Bitton, Jaemin Cho, Roopal Garg, Alexander Ku, Zarana Parekh, Jordi Pont-Tuset, Garrett Tanzer, et al. Docci: Descriptions of connected and contrasting images. URL https://arxiv.org/abs/2404.19753

    arXiv

  49. [58]

    Open world entity segmentation.TPAMI, 2022

    Lu Qi, Jason Kuen, Yi Wang, Jiuxiang Gu, Hengshuang Zhao, Philip Torr, Zhe Lin, and Jiaya Jia. Open world entity segmentation.TPAMI, 2022

    2022

  50. [59]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021

    2021

  51. [60]

    Eve: Efficient multimodal vision language models with elastic visual experts.arXiv preprint arXiv:2501.04322, 2025

    Miao Rang, Zhenni Bi, Chuanjian Liu, Yehui Tang, Kai Han, and Yunhe Wang. Eve: Efficient multimodal vision language models with elastic visual experts.arXiv preprint arXiv:2501.04322, 2025. 25

    2025 arXiv

  52. [61]

    Glamm: Pixel grounding large multimodal model

    Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S Khan. Glamm: Pixel grounding large multimodal model. In CVPR, 2024

    2024

  53. [62]

    Pixellm: Pixel reasoning with large multimodal model

    Zhongwei Ren, Zhicheng Huang, Yunchao Wei, Yao Zhao, Dongmei Fu, Jiashi Feng, and Xiaojie Jin. Pixellm: Pixel reasoning with large multimodal model. InCVPR, 2024

    2024

  54. [63]

    Laion-5b: An open large-scale dataset for training next generation image-text models.NeurIPS, 2022

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models.NeurIPS, 2022

    2022

  55. [64]

    Objects365: A large-scale, high-quality dataset for object detection

    Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. InICCV, 2019

    2019

  56. [65]

    Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning

    Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. InACL, 2018

    2018

  57. [66]

    Aligning and prompting everything all at once for universal visual perception

    Yunhang Shen, Chaoyou Fu, Peixian Chen, Mengdan Zhang, Ke Li, Xing Sun, Yunsheng Wu, Shaohui Lin, and Rongrong Ji. Aligning and prompting everything all at once for universal visual perception. InCVPR, 2024

    2024

  58. [67]

    Long-vita: Scaling large multi-modal models to 1 million tokens with leading short-context accuray.arXiv preprint arXiv:2502.05177, 2025

    Yunhang Shen, Chaoyou Fu, Shaoqi Dong, Xiong Wang, Peixian Chen, Mengdan Zhang, Haoyu Cao, Ke Li, Xiawu Zheng, Yan Zhang, et al. Long-vita: Scaling large multi-modal models to 1 million tokens with leading short-context accuray.arXiv preprint arXiv:2502.05177, 2025

    2025

  59. [68]

    Seed1.5-vl technical report.arXiv preprint arXiv:2505.07062, 2025

    ByteDance Seed Team. Seed1.5-vl technical report.arXiv preprint arXiv:2505.07062, 2025

    2025 arXiv

  60. [69]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  61. [70]

    Yfcc100m: The new data in multimedia research.Communications of the ACM, 59(2):64–73, 2016

    Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfcc100m: The new data in multimedia research.Communications of the ACM, 59(2):64–73, 2016

    2016

  62. [71]

    Cambrian-1: A fully open, vision-centric exploration of multimodal llms.NeurIPS, 2024

    Peter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Jairam Vedagiri IYER, Sai Charitha Akula, Shusheng Yang, Jihan Yang, Manoj Middepogu, Ziteng Wang, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms.NeurIPS, 2024

    2024

  63. [72]

    Eyes wide shut? exploring the visual shortcomings of multimodal llms

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. InCVPR, 2024

    2024

  64. [73]

    World to code: Multi-modal data generation via self-instructed compositional captioning and filtering.arXiv preprint arXiv:2409.20424, 2024

    Jiacong Wang, Bohong Wu, Haiyong Jiang, Xun Zhou, Xin Xiao, Haoyuan Guo, and Jun Xiao. World to code: Multi-modal data generation via self-instructed compositional captioning and filtering.arXiv preprint arXiv:2409.20424, 2024

    2024 arXiv

  65. [74]

    Vgr: Visual grounded reasoning.arXiv preprint arXiv:2506.11991, 2025

    Jiacong Wang, Zijiang Kang, Haochen Wang, Haiyong Jiang, Jiawen Li, Bohong Wu, Ya Wang, Jiao Ran, Xiao Liang, Chao Feng, et al. Vgr: Visual grounded reasoning.arXiv preprint arXiv:2506.11991, 2025

    2025 arXiv

  66. [75]

    V3det: Vast vocabulary visual detection dataset

    Jiaqi Wang, Pan Zhang, Tao Chu, Yuhang Cao, Yujie Zhou, Tong Wu, Bin Wang, Conghui He, and Dahua Lin. V3det: Vast vocabulary visual detection dataset. InICCV, 2023

    2023

  67. [76]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  68. [77]

    The all-seeing project: Towards panoptic visual recognition and understanding of the open world

    Weiyun Wang, Min Shi, Qingyun Li, Wenhai Wang, Zhenhang Huang, Linjie Xing, Zhe Chen, Hao Li, Xizhou Zhu, Zhiguo Cao, et al. The all-seeing project: Towards panoptic visual recognition and understanding of the open world. arXiv preprint arXiv:2308.01907, 2023

    2023 arXiv

  69. [78]

    The all-seeing project v2: Towards general relation comprehension of the open world

    Weiyun Wang, Yiming Ren, Haowen Luo, Tiantong Li, Chenxiang Yan, Zhe Chen, Wenhai Wang, Qingyun Li, Lewei Lu, Xizhou Zhu, et al. The all-seeing project v2: Towards general relation comprehension of the open world. In ECCV, 2024

    2024

  70. [79]

    Images speak in images: A generalist painter for in-context visual learning

    Xinlong Wang, Wen Wang, Yue Cao, Chunhua Shen, and Tiejun Huang. Images speak in images: A generalist painter for in-context visual learning. InCVPR, 2023. 26

    2023

  71. [80]

    Controlmllm: Training-free visual prompt learning for multimodal large language models

    Mingrui Wu, Xinyue Cai, Jiayi Ji, Jiale Li, Oucheng Huang, Gen Luo, Hao Fei, Guannan Jiang, Xiaoshuai Sun, and Rongrong Ji. Controlmllm: Training-free visual prompt learning for multimodal large language models. NeurIPS, 2024

    2024

  72. [81]

    Clipself: Vision transformer distills itself for open-vocabulary dense prediction

    Size Wu, Wenwei Zhang, Lumin Xu, Sheng Jin, Xiangtai Li, Wentao Liu, and Chen Change Loy. Clipself: Vision transformer distills itself for open-vocabulary dense prediction. InICLR, 2024

    2024

  73. [82]

    Rap-sam:towards real-time all-purpose segment anything

    Shilin Xu, Haobo Yuan, Qingyu Shi, Lu Qi, Jingbo Wang, Yibo Yang, Yining Li, Kai Chen, Yunhai Tong, Bernard Ghanem, Xiangtai Li, and Ming-Hsuan Yang. Rap-sam:towards real-time all-purpose segment anything. In ICLR, 2025

    2025

  74. [83]

    Visa: Reasoning video object segmentation via large language models

    Cilin Yan, Haochen Wang, Shilin Yan, Xiaolong Jiang, Yao Hu, Guoliang Kang, Weidi Xie, and Efstratios Gavves. Visa: Reasoning video object segmentation via large language models. InECCV, 2024

    2024

  75. [84]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    2025 arXiv

  76. [85]

    Pix2cap-coco: Advancing visual comprehension via pixel-level captioning.arXiv preprint arXiv:2501.13893, 2025

    Zuyao You, Junke Wang, Lingyu Kong, Bo He, and Zuxuan Wu. Pix2cap-coco: Advancing visual comprehension via pixel-level captioning.arXiv preprint arXiv:2501.13893, 2025

    2025 arXiv

  77. [86]

    Modeling context in referring expressions

    Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In ECCV, 2016

    2016

  78. [87]

    Polyphonicformer: Unified query learning for depth-aware video panoptic segmentation

    Haobo Yuan, Xiangtai Li, Yibo Yang, Guangliang Cheng, Jing Zhang, Yunhai Tong, Lefei Zhang, and Dacheng Tao. Polyphonicformer: Unified query learning for depth-aware video panoptic segmentation. InECCV, 2022

    2022

  79. [88]

    Open-vocabulary sam: Segment and recognize twenty-thousand classes interactively

    Haobo Yuan, Xiangtai Li, Chong Zhou, Yining Li, Kai Chen, and Chen Change Loy. Open-vocabulary sam: Segment and recognize twenty-thousand classes interactively. InECCV, 2024

    2024

  80. [89]

    Sa2va: Marrying sam2 with llava for dense grounded understanding of images and videos

    Haobo Yuan, Xiangtai Li, Tao Zhang, Zilong Huang, Shilin Xu, Shunping Ji, Yunhai Tong, Lu Qi, Jiashi Feng, and Ming-Hsuan Yang. Sa2va: Marrying sam2 with llava for dense grounded understanding of images and videos. arXiv preprint arXiv:2501.04001, 2025

    2025 arXiv

  81. [90]

    Instruction-guided multi-granularity segmentation and captioning with large multimodal model

    Xu Yuan, Li Zhou, Zenghui Sun, Zikun Zhou, and Jinsong Lan. Instruction-guided multi-granularity segmentation and captioning with large multimodal model. InAAAI, 2025

    2025

  82. [91]

    Osprey: Pixel understanding with visual instruction tuning

    Yuqian Yuan, Wentong Li, Jian Liu, Dongqi Tang, Xinjie Luo, Chi Qin, Lei Zhang, and Jianke Zhu. Osprey: Pixel understanding with visual instruction tuning. InCVPR, 2024

    2024

  83. [92]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A m...

    2024

  84. [93]

    Omg-llava: Bridging image-level, object-level, pixel-level reasoning and understanding

    Tao Zhang, Xiangtai Li, Hao Fei, Haobo Yuan, Shengqiong Wu, Shunping Ji, Chen Change Loy, and Shuicheng Yan. Omg-llava: Bridging image-level, object-level, pixel-level reasoning and understanding. InNeurIPS, 2024

    2024

  85. [94]

    Pixel-sail: Single transformer for pixel-grounded understanding.arXiv preprint arXiv:2504.10465, 2025

    Tao Zhang, Xiangtai Li, Zilong Huang, Yanwei Li, Weixian Lei, Xueqing Deng, Shihao Chen, Shunping Ji, and Jiashi Feng. Pixel-sail: Single transformer for pixel-grounded understanding.arXiv preprint arXiv:2504.10465, 2025

    2025 arXiv

  86. [95]

    Seeing clearly by layer two: Enhancing attention heads to alleviate hallucination in lvlms.arXiv preprint arXiv:2411.09968, 2024

    Xiaofeng Zhang, Yihao Quan, Chaochen Gu, Chen Shen, Xiaosong Yuan, Shaotian Yan, Hao Cheng, Kaijie Wu, and Jieping Ye. Seeing clearly by layer two: Enhancing attention heads to alleviate hallucination in lvlms.arXiv preprint arXiv:2411.09968, 2024

    2024 arXiv

  87. [96]

    Enhancing multimodal large language models complex reason via similarity computation

    Xiaofeng Zhang, Fanshuo Zeng, Yihao Quan, Zheng Hui, and Jiawei Yao. Enhancing multimodal large language models complex reason via similarity computation. InAAAI, 2025

    2025

  88. [97]

    Mme-realworld: Could your multimodal llm challenge high-resolution real-world scenarios that are difficult for humans?arXiv preprint arXiv:2408.13257, 2024

    Yi-Fan Zhang, Huanyu Zhang, Haochen Tian, Chaoyou Fu, Shuangqing Zhang, Junfei Wu, Feng Li, Kun Wang, Qingsong Wen, Zhang Zhang, et al. Mme-realworld: Could your multimodal llm challenge high-resolution real-world scenarios that are difficult for humans?arXiv preprint arXiv:24...

    2024 arXiv

  89. [98]

    Llava-uhd v2: an mllm integrating high-resolution feature pyramid via hierarchical window transformer.arXiv preprint arXiv:2412.13871, 2024

    Yipeng Zhang, Yifan Liu, Zonghao Guo, Yidan Zhang, Xuesong Yang, Chi Chen, Jun Song, Bo Zheng, Yuan Yao, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. Llava-uhd v2: an mllm integrating high-resolution feature pyramid via hierarchical window transformer.arXiv preprint arXiv:2412...

    2024 arXiv

  90. [99]

    Recognize anything: A strong image tagging model

    Youcai Zhang, Xinyu Huang, Jinyu Ma, Zhaoyang Li, Zhaochuan Luo, Yanchun Xie, Yuzhuo Qin, Tong Luo, Yaqian Li, Shilong Liu, et al. Recognize anything: A strong image tagging model. InCVPR, 2024

    2024

  91. [100]

    Mlvu: A comprehensive benchmark for multi-task long video understanding.arXiv preprint arXiv:2406.04264, 2024

    Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding.arXiv preprint arXiv:2406.04264, 2024

    2024 arXiv

  92. [101]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479, 2025

    2025 arXiv

  93. [102]

    Genimage: A million-scale benchmark for detecting ai-generated image.NeurIPS, 2023

    Mingjian Zhu, Hanting Chen, Qiangyu Yan, Xudong Huang, Guanyu Lin, Wei Li, Zhijun Tu, Hailin Hu, Jie Hu, and Yunhe Wang. Genimage: A million-scale benchmark for detecting ai-generated image.NeurIPS, 2023. 28

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.