Pith. sign in

REVIEW 6 major objections 5 minor 1 cited by

Argus: Vision-Centric Reasoning with Grounded Chain-of-Thought

T0 review · 6 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Argus shows that explicitly re-reading the image region a question asks about improves MLLM reasoning and grounding.

desk verdict A solid, incremental architecture paper with a genuinely useful re-sampling-vs-re-encoding comparison, but the SOTA claim is softened by training/eval overlap and design choices tuned on the reported benchmarks. read the letter →

arxiv 2505.23766 v1 pith:SWOKGWPV submitted 2025-05-29 cs.CV

Yunze Man , De-An Huang , Guilin Liu , Shiwei Sheng , Shilong Liu , Liang-Yan Gui , Jan Kautz , Yu-Xiong Wang
show 1 more author
Zhiding Yu
This is my paper
classification cs.CV
keywords multimodallargelanguagemodelsvisualchain-of-thoughtgroundedreasoninggroundingregion-of-interestre-engagementgoal-directedattentionmixture-of-vision-expertsreferringexpression
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Argus is an attempt to show that multimodal large language models reason better when they can deliberately re-focus on the image region a question points to, not just passively consume a global image embedding. The authors add a bounding-box prediction step to the answer stream, then feed the predicted region back as visual chain-of-thought context before producing the final answer. Across vision-centric, text-centric, and general reasoning benchmarks, Argus outperforms public MLLMs of comparable parameter count and training scale, and it is competitive on referring-expression grounding. If the result holds, explicit, language-guided visual attention is a useful architectural ingredient for multimodal reasoning rather than a task-specific add-on.

What carries the argument

The load-bearing component is the directed visual context re-engagement module, which turns the model's predicted bounding box into goal-directed visual chain-of-thought tokens. The region is encoded either by re-encoding, which treats the padded crop as a new image and passes it through the mixture-of-vision-experts encoders, or by re-sampling, which retrieves the precomputed patch embeddings that overlap the box from a token cache. These extra visual tokens are inserted into the multimodal sequence before the final answer, so the LLM attends to both the global image and the language-selected region. The rest of the architecture—CLIP, ConvNeXt, and EVA-02 vision experts feeding a Llama-3-8B decoder—follows the standard MLLM design and is not the claimed source of the improvement.

What would settle it

Run Argus on a reasoning benchmark with three box conditions: the model's own predicted boxes, ground-truth boxes from the VCoT annotations, and random boxes of the same size. If random boxes perform as well as predicted boxes, or if ground-truth boxes do not outperform predicted boxes, then the improved reasoning cannot be attributed to accurate grounding and the re-engagement mechanism would be acting as regularization rather than as a visual search process.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that grounding-driven, goal-directed visual attention improves both multimodal reasoning and referring-grounding performance in an 8-billion-parameter MLLM. The model is trained to first output a normalized bounding box for the question-relevant region, then receive intermediate visual-context tokens sampled from that region, and only then answer. The authors compare four ways to engage the predicted region—implicit self-attention, implicit box guidance, explicit RoI re-encoding, and explicit RoI re-sampling—and find that both explicit strategies beat the implicit ones, with re-sampling generally strongest and re-encoding preferred when small objects matter, as on the V-Star benchmark. They further show that adding grounding data on top of CoT training gives additional gains, supporting the claim that grounding and reasoning reinforce each other through the same mechanism.

Load-bearing premise

At inference, the model's predicted bounding boxes are accurate enough that the regions they select carry the visual information needed to answer; if a box misses its target, the re-engaged tokens inject noise and can hurt the final answer.

Editorial extensions

If this is right

  • If Argus's results are correct, explicit RoI re-engagement yields state-of-the-art performance among public MLLMs of comparable parameter count and training scale on visual reasoning benchmarks.
  • Grounded visual CoT improves both vision-centric and text-centric reasoning over implicit self-attention, and adding referring-grounding data to the CoT training mixture adds further gains.
  • Re-sampling is the cheaper explicit strategy, using roughly half the GMACs, 26 versus 1024 extra visual tokens, and faster inference, while re-encoding is more robust when the task hinges on small objects or fine detail.
  • RoI context expansion helps re-encoding up to about 40 percent before hurting, whereas re-sampling already gets context from overlapping patches and degrades with expansion.
  • Extending the single-RoI framework to multiple RoIs improves vision-centric benchmark scores, suggesting the mechanism scales to multi-step visual search.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's gains stake a clear, testable claim about inference-time grounding: the predicted box must add task-relevant context, and an oracle-box comparison would separate the value of accurate grounding from the value of training with grounding data.
  • If the mechanism generalizes, explicit goal-directed re-engagement could become a standard module for MLLMs handling high-resolution or cluttered scenes, since it is orthogonal to encoder scaling and data mixing.
  • A natural stress test is to degrade box accuracy at inference; the framework predicts graceful degradation for re-encoding, which re-reads pixels, but sharper degradation for re-sampling, which is stuck with the initial token cache.
  • The distinction between stimulus-driven and goal-directed attention suggests that future models could chain multiple re-engagement steps for sequential visual search, which the paper's multi-RoI experiment begins to support.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 5 minor

Summary. This paper introduces Argus, an 8B-parameter multimodal LLM that augments standard end-to-end vision-language modeling with a grounding-driven visual attention re-engagement mechanism. At inference, the model first outputs text-format bounding boxes for regions relevant to the user question; then, for each predicted box, it retrieves the overlapping patch tokens from the initial visual encoder cache (re-sampling) or re-encodes the cropped region (re-encoding), and finally generates the answer with these additional visual context tokens. The authors report state-of-the-art results among public MLLMs of comparable scale on several vision-centric and text-centric benchmarks, and strong referring-expression grounding accuracy. They further analyze design choices including the engagement strategy, expansion ratio, encoder capacity, and a multi-RoI extension, and provide efficiency measurements for the two strategies.

Significance. If the gains are attributable to the proposed mechanism, the work would make a valuable contribution to vision-centric reasoning by showing that explicit, language-guided RoI re-engagement is an effective and efficient architectural addition. The paper is also useful for its systematic comparison of four engagement strategies and its detailed efficiency analysis. However, the significance is currently limited by the following concerns: (i) training data overlap with several evaluation benchmarks, (ii) a baseline inconsistency between Tables 1 and 5, (iii) an ablation that cannot separate the effect of additional grounding data from the re-engagement mechanism, and (iv) a main model whose multi-RoI inference extension is only described in the supplementary material. These issues must be resolved before the scientific claims can be accepted.

major comments (6)
  1. [§4.1, §4.4, Table 1] The SFT data include VCoT (Section 4.1), which explicitly lists TextVQA, DocVQA, and GQA among its constituent datasets, and Table 1 reports evaluation on exactly these benchmarks. In addition, the grounding training mixture (Shikra, as described in Section 4.1) includes RefCOCO-family data on which Table 2 reports results. This train/evaluation overlap raises the possibility that the reported gains derive at least in part from memorized associations rather than from a general re-engagement mechanism. The paper should provide evidence that the overlap is not responsible for the gains, for example by re-evaluating on subsets of these benchmarks that are not present in the training data, or by reporting results on held-out benchmarks not used in training at all.
  2. [Table 5 vs Table 1] The 'Baseline (Eagle-X3)' row reports V-Star=55.3, CVB-2D=64.9, TextVQA=66.3, ChartQA=63.0, while the same model in Table 1 has V-Star=60.7, CV-Bench-2D=66.4, TextVQA=70.9, ChartQA=70.4. The paper does not explain this discrepancy. If the Table 5 baseline is a re-training under the 'accelerated and unified training schedule' described in Section 4.5, that must be stated in the caption and the exact schedule and data composition provided; otherwise the claimed +CoT and ++Grounding increments are not interpretable.
  3. [§4.5.2, Table 5] The incremental ablation from 'Baseline' to '+ CoT signals' to '++ Grounding' jointly changes both the training data (adding VCoT and then GRIT/Shikra) and the inference-time mechanism (adding box prediction and re-sampling). This cannot attribute the observed gains to the re-engagement mechanism as opposed to the additional training data. A controlled condition that keeps the training data identical to the '+ CoT' or '++ Grounding' setting but disables the visual token re-engagement at inference (e.g., generating only the box text without retrieving visual tokens) is needed to support the paper's central claim.
  4. [§3.2, Table C, Table 1] The main model described in Section 3 and Figure 3 performs a single RoI prediction per question, yet Table 1 reports the multi-RoI version of Argus (V-Star 78.5, CV-Bench-3D 69.6) whose inference procedure is described only in the supplementary material. The main paper should either present the multi-RoI extension as part of the core method or report the single-RoI version in the main table. Without this, the reader cannot tell which architecture the headline results belong to.
  5. [§3.2, Table A] The effectiveness of re-sampling depends on the accuracy of the predicted bounding boxes at inference, since tokens are selected by intersection with the predicted boxes. Supplementary Table A shows that deliberate context expansion by 80% degrades re-sampling V-Star accuracy by 6.3 points (67.0 to 60.7), indicating high sensitivity to box quality. However, the paper never measures grounding accuracy of the predicted boxes on the reasoning benchmarks. Please report box IoU or Acc@0.5 on these benchmarks (or on a representative subset) to verify that the boxes are accurate enough for the mechanism to help. If grounding accuracy is low, the mechanism may add noise rather than relevant context.
  6. [§4.5.1, Table 3, Table C] The design choices of re-sampling over re-encoding, and the multi-RoI extension, were selected based on performance on V-Star, CV-Bench, TextVQA, and ChartQA (Tables 3 and C) – the same benchmarks used for the headline results in Table 1. With a single run per configuration and no held-out validation set, the reported improvements are likely to be optimistic. The authors should at least disclose the full space of explored choices and the selection procedure, or report results on a validation split that is disjoint from the final evaluation benchmarks.
minor comments (5)
  1. [Table C caption] The caption ends with the fragment 'where .' and should be completed.
  2. [§1] Typo: 're-engagment' should be 're-engagement'.
  3. [§4.1] The sentence 'To ensure ensures strong general-purpose multimodal understanding capabilities' has a duplicated verb; it should read 'To ensure robust ...'.
  4. [Table 1] The label 'ref :' before the proprietary/reference models is not defined; please clarify that these are reference models not directly comparable in scale or training data.
  5. [§4.4] In 'Argus achieve state-of-the-art performance', 'achieve' should be 'achieves'.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the central mechanism is isolated by internal ablations, and self-citations to Eagle are architectural inheritance rather than a logical reduction.

full rationale

Argus's claim that explicit RoI re-engagement improves reasoning is supported by controlled ablations (Tables 3 and 4) that vary the re-engagement mechanism under a unified training schedule, and by the extension experiments in the supplement. The paper does inherit its MoVE encoder design, pre-alignment, and initialization from the authors' prior Eagle work, but this is disclosed as a baseline and does not force the reported differences; Table 3 compares Implicit Att., Box Guidance, RoI Re-encoding, and RoI Re-sampling directly. The VCoT training data is sourced from some benchmark families that also appear in evaluation (e.g., TextVQA, GQA, ChartQA), which is a data-composition and generalization concern rather than a by-construction equivalence: the paper reports held-out benchmark scores after standard supervised training, and no equation or parameter fit makes the evaluation quantity identical to a training target. The selection of re-sampling over re-encoding was partially influenced by the same benchmarks used for reporting, but this is model selection on validation-style ablations, not circular definition of the predicted quantity. Overall, no circular step meeting the quote-and-reduce standard was found.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper is an empirical systems contribution; it introduces no analytic derivations. Its central claims rest on the accuracy of predicted boxes, the quality of VCoT annotations, and the legitimacy of evaluating on benchmarks that also appear in training. The main hand-tuned design variables are the re-engagement strategy, the RoI expansion ratio, and the number of vision experts.

free parameters (3)
  • RoI context expansion ratio = 0% for re-sampling (20-40% for re-encoding)
    Selected via ablation in Table A/Figure A of the supplementary; directly affects re-engagement performance and is tuned on the same benchmarks used for evaluation.
  • Number of vision experts (MoVE) = 3 (CLIP-ViT-L/14, ConvNeXt-XXL, EVA-02-L/16)
    Copied from Eagle-X3; not varied in this paper, but central to the model's tokenization.
  • Visual token count = 1024 (32x32 grid)
    Fixed design choice; affects how much visual information is compressed.
assumptions (4)
  • domain assumption The cognitive distinction between stimulus-driven and goal-directed attention transfers to MLLM design.
    Motivates the explicit RoI re-engagement module in Section 1 and Figure 2.
  • domain assumption Predicted bounding boxes from the LLM are sufficiently accurate for the re-engagement module to help reasoning.
    Core to Section 3.2; if boxes are noisy, the added visual tokens inject irrelevant context.
  • domain assumption The VCoT dataset's RoI annotations are of sufficient quality to teach grounding for reasoning.
    VCoT is the main source of visual CoT training data in Section 4.1; its reliability is assumed.
  • standard math Standard transformer training and evaluation pipelines behave as expected.
    Background assumption for all deep learning experiments; no formal proof is given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Argus: Vision-Centric Reasoning with Grounded Chain-of-Thought." pith.science (2026). https://pith.science/paper/SWOKGWPV

@misc{pith2026250523766,
  author       = {Pith},
  title        = {Pith review of: Argus: Vision-Centric Reasoning with Grounded Chain-of-Thought},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SWOKGWPV}},
  note         = {Machine review of arXiv:2505.23766}
}
read the original abstract

Recent advances in multimodal large language models (MLLMs) have demonstrated remarkable capabilities in vision-language tasks, yet they often struggle with vision-centric scenarios where precise visual focus is needed for accurate reasoning. In this paper, we introduce Argus to address these limitations with a new visual attention grounding mechanism. Our approach employs object-centric grounding as visual chain-of-thought signals, enabling more effective goal-conditioned visual attention during multimodal reasoning tasks. Evaluations on diverse benchmarks demonstrate that Argus excels in both multimodal reasoning tasks and referring object grounding tasks. Extensive analysis further validates various design choices of Argus, and reveals the effectiveness of explicit language-guided visual region-of-interest engagement in MLLMs, highlighting the importance of advancing multimodal intelligence from a visual-centric perspective. Project page: https://yunzeman.github.io/argus/

Figures

Figures reproduced from arXiv: 2505.23766 by the authors.

Figure 1
Figure 1. Visual question answering, grounding, and chain-of-thought reasoning with Argus. “ctx-token” is short for context token. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Illustration of two visual attention mechanisms. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Illustration of Argus architecture. In addition to standard unconditioned visual tokenization, our method incorporates an addi [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Illustration of two visual CoT mechanisms. Re-encoding [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Qualitative evaluation of Argus. We achieve superior performance in challenging multimodal reasoning and perception tasks. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. OPLD: On-Policy Latent Distillation for Multimodal Reasoning

    cs.CV 2026-07 conditional novelty 6.0 of 10

    On-policy distillation of a CoT-privileged teacher’s token preferences and latent trajectories yields a CoT-free student that beats prior visual-latent methods on several multimodal benchmarks.

Reference graph

Works this paper leans on

115 extracted references · 41 canonical work pages · cited by 1 Pith paper

  1. [1]

    Qwen-VL: A versatile vision-language model for understanding, localization, text reading, and beyond.arXiv preprint arXiv:2308.12966, 2023

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-VL: A versatile vision-language model for understanding, localization, text reading, and beyond.arXiv preprint arXiv:2308.12966, 2023. 2, 5, 6, 7

    arXiv 2023

  2. [2]

    Qwen2.5-VL technical report

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhao- hai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Jun- yang Lin. Qwen2.5-VL technical repor...

    arXiv 2025

  3. [3]

    Graph of thoughts: Solving elab- orate problems with large language models

    Maciej Besta, Nils Blach, Ales Kubicek, Robert Ger- stenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Ny- czyk, and Torsten Hoefler. Graph of thoughts: Solving elab- orate problems with large language models. In AAAI, 2024. 3

    2024

  4. [4]

    COYO-700M: Image-text pair dataset

    Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon Kim. COYO-700M: Image-text pair dataset. https : / / github . com / kakaobrain/coyo-dataset, 2022. 3

    2022

  5. [5]

    Image- level or object-level? A tale of two resampling strategies for long-tailed detection

    Nadine Chang, Zhiding Yu, Yu-Xiong Wang, Animashree Anandkumar, Sanja Fidler, and Jose M Alvarez. Image- level or object-level? A tale of two resampling strategies for long-tailed detection. In ICML, 2021. 4

    2021

  6. [6]

    Contrastive lo- calized language-image pre-training

    Hong-You Chen, Zhengfeng Lai, Haotian Zhang, Xinze Wang, Marcin Eichner, Keen You, Meng Cao, Bowen Zhang, Yinfei Yang, and Zhe Gan. Contrastive lo- calized language-image pre-training. arXiv preprint arXiv:2410.02746, 2024. 2

    arXiv 2024

  7. [7]

    MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning

    Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.09478, 2023. 6, 7

    arXiv

  8. [8]

    Shikra: Unleashing multi- modal LLM’s referential dialogue magic

    Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multi- modal LLM’s referential dialogue magic. arXiv preprint arXiv:2306.15195, 2023. 4, 5, 6, 7, 3

    arXiv 2023

Show all 115 references
  1. [9]

    ShareGPT4v: Improving large multi-modal models with better captions

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. ShareGPT4v: Improving large multi-modal models with better captions. In ECCV, 2024. 5, 3

    2024

  2. [10]

    Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. TMLR, 2023. 3

    2023

  3. [11]

    UNITER: Universal image-text representation learning

    Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. UNITER: Universal image-text representation learning. In ECCV, 2020. 6, 7

    2020

  4. [12]

    How far are we to GPT- 4V? closing the gap to commercial multimodal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhang- wei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, Ji Ma, Jiaqi Wang, Xiaoyi Dong, Hang Yan, Hewei Guo, Conghui He, Botian Shi, Zhenjiang Jin, Chao Xu, Bin Wang, Xingjian Wei, Wei Li, Wenjian Zhang, Bo Zhan...

    arXiv

  5. [13]

    InternVL: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. InternVL: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In CVPR,

  6. [14]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhang- hao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing GPT-4 with 90%* chatgpt quality. https://vicuna. lmsys. org, 2023. 3, 2

    2023

  7. [15]

    Control of goal- directed and stimulus-driven attention in the brain

    Maurizio Corbetta and Gordon L Shulman. Control of goal- directed and stimulus-driven attention in the brain. Nature reviews neuroscience, 2002. 1

    2002

  8. [16]

    TransVG: End-to-end visual grounding with transformers

    Jiajun Deng, Zhengyuan Yang, Tianlang Chen, Wengang Zhou, and Houqiang Li. TransVG: End-to-end visual grounding with transformers. In ICCV, 2021. 6, 7

    2021

  9. [17]

    An im- age is worth 16x16 words: Transformers for image recog- nition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An im- age is worth 16x16 words: Transformers for image recog- nitio...

    2021

  10. [18]

    EV A: Exploring the limits of masked visual represen- tation learning at scale

    Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. EV A: Exploring the limits of masked visual represen- tation learning at scale. In CVPR, 2023. 3, 2

    2023

  11. [19]

    EV A-02: A visual representa- tion for neon genesis

    Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. EV A-02: A visual representa- tion for neon genesis. Image and Vision Computing, 2024. 3, 5, 2

    2024

  12. [20]

    Large-scale adversarial training for vision-and-language representation learning

    Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu. Large-scale adversarial training for vision-and-language representation learning. In NeurIPS,

  13. [21]

    G-LLaV A: Solving ge- ometric problem with multi-modal large language model

    Jiahui Gao, Renjie Pi, Jipeng Zhang, Jiacheng Ye, Wanjun Zhong, Yufei Wang, Lanqing Hong, Jianhua Han, Hang Xu, Zhenguo Li, and Lingpeng Kong. G-LLaV A: Solving ge- ometric problem with multi-modal large language model. arXiv preprint arXiv:2312.11370, 2023. 5, 3 9

    2023 arXiv

  14. [22]

    Mini-InternVL: a flexible-transfer pocket multi-modal model with 5% parameters and 90% perfor- mance

    Zhangwei Gao, Zhe Chen, Erfei Cui, Yiming Ren, Weiyun Wang, Jinguo Zhu, Hao Tian, Shenglong Ye, Junjun He, Xizhou Zhu, Lewei Lu, Tong Lu, Yu Qiao, Jifeng Dai, and Wenhai Wang. Mini-InternVL: a flexible-transfer pocket multi-modal model with 5% parameters and 90% perfor- mance....

    2024

  15. [23]

    Chain of thought prompt tuning in vision language models

    Jiaxin Ge, Hongyin Luo, Siyuan Qian, Yulu Gan, Jie Fu, and Shanghang Zhang. Chain of thought prompt tuning in vision language models. arXiv preprint arXiv:2304.07919,

    arXiv

  16. [24]

    ICDAR2019 competition on scanned receipt ocr and information extrac- tion

    Zheng Huang, Kai Chen, Jianhua He, Xiang Bai, Dimos- thenis Karatzas, Shijian Lu, and CV Jawahar. ICDAR2019 competition on scanned receipt ocr and information extrac- tion. In ICDAR, 2019. 5, 3

    2019

  17. [25]

    GQA: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. GQA: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019. 2, 5, 6, 3

    2019

  18. [26]

    Psychology, briefer course

    William James. Psychology, briefer course. Harvard Uni- versity Press, 1984. 1

    1984

  19. [27]

    DVQA: Understanding data visualizations via ques- tion answering

    Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. DVQA: Understanding data visualizations via ques- tion answering. In CVPR, 2018. 5, 3

    2018

  20. [28]

    MDETR- modulated detection for end-to-end multi-modal under- standing

    Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. MDETR- modulated detection for end-to-end multi-modal under- standing. In ICCV, 2021. 6, 7

    2021

  21. [29]

    De- coupling representation and classifier for long-tailed recog- nition

    Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. De- coupling representation and classifier for long-tailed recog- nition. arXiv preprint arXiv:1910.09217, 2019. 4

    1910 arXiv

  22. [30]

    Directed attention as a common resource for executive functioning and self- regulation

    Stephen Kaplan and Marc G Berman. Directed attention as a common resource for executive functioning and self- regulation. Perspectives on psychological science, 2010. 1

    2010

  23. [31]

    Referitgame: Referring to objects in pho- tographs of natural scenes

    Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in pho- tographs of natural scenes. In EMNLP, 2014. 2, 3

    2014

  24. [32]

    A diagram is worth a dozen images

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In ECCV, 2016. 5, 3

    2016

  25. [33]

    Ocr- free document understanding transformer

    Geewook Kim, Teakgyu Hong, Moonbin Yim, JeongYeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. Ocr- free document understanding transformer. In ECCV, 2022. 5, 6, 3

    2022

  26. [34]

    Berg, Wan-Yen Lo, Piotr Doll´ar, and Ross Girshick

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Doll´ar, and Ross Girshick. Segment anything. arXiv:2304.02643,

    arXiv

  27. [35]

    Berg, Wan-Yen Lo, Piotr Doll´ar, and Ross Girshick

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Doll´ar, and Ross Girshick. Segment anything. In ICCV, 2023. 2

    2023

  28. [36]

    Large language models are zero-shot reasoners

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In NeurIPS, 2022. 3

    2022

  29. [37]

    Shamma, Michael S

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalan- tidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Fei-Fei Li. Visual genome: Connecting language and vision using crowdsourced dense image annotations. ...

    2017

  30. [38]

    The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale

    Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Ui- jlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Ste- fan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. The open images dataset v4: Unified image classification, object detection, and ...

    1956

  31. [39]

    LLaV A-OneVision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. LLaV A-OneVision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 1, 2, 3

    2024 arXiv

  32. [40]

    Blip: Bootstrapping language-image pre-training for uni- fied vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for uni- fied vision-language understanding and generation. In ICML, 2022. 3

    2022

  33. [41]

    Mini-Gemini: Mining the potential of multi-modality vision language models

    Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-Gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403.18814,

    arXiv

  34. [42]

    V oCoT: Unleashing visually grounded multi-step reasoning in large multi-modal models

    Zejun Li, Ruipu Luo, Jiwen Zhang, Minghui Qiu, and Zhongyu Wei. V oCoT: Unleashing visually grounded multi-step reasoning in large multi-modal models. arXiv preprint arXiv:2405.16919, 2024. 3, 4

    2024 arXiv

  35. [43]

    VILA: On pre-training for visual language models

    Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Moham- mad Shoeybi, and Song Han. VILA: On pre-training for visual language models. In CVPR, 2024. 1, 2, 3, 5

    2024

  36. [44]

    Microsoft COCO: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll ´ar, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014. 2

    2014

  37. [45]

    SPHINX: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models

    Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, Jiaming Han, Siyuan Huang, Yichi Zhang, Xum- ing He, Hongsheng Li, and Yu Qiao. SPHINX: The joint mixing of weights, tasks, and visual embeddings for multi-modal lar...

    2023 arXiv

  38. [46]

    Visual spa- tial reasoning

    Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spa- tial reasoning. Transactions of the Association for Compu- tational Linguistics, 11:635–651, 2023. 5, 3

    2023

  39. [47]

    Aligning large multi-modal model with robust instruction tuning

    Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Aligning large multi-modal model with robust instruction tuning. arXiv preprint arXiv:2306.14565, 2023. 5, 3

    2023 arXiv

  40. [48]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023. 2, 3, 4, 5

    2023

  41. [49]

    LLaV A-NeXT: Im- proved reasoning, OCR, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. LLaV A-NeXT: Im- proved reasoning, OCR, and world knowledge, 2024. 1, 2, 3, 4, 5, 6 10

    2024

  42. [50]

    LLaV A-Plus: Learning to use tools for creating multimodal agents

    Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang, Hang Su, Jun Zhu, Lei Zhang, Jianfeng Gao, and Chunyuan Li. LLaV A-Plus: Learning to use tools for creating multimodal agents. arXiv preprint arXiv:2311.05437, 2023. 3

    2023 arXiv

  43. [51]

    Grounding DINO: Mar- rying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang. Grounding DINO: Mar- rying dino with grounded pre-training for open-set object detection. In ECCV, 2024. 3, 4, 6, 7

    2024

  44. [52]

    A convnet for the 2020s

    Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feicht- enhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In CVPR, 2022. 3, 5, 2

    2022

  45. [53]

    Decoupled weight decay regularization

    I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 5

    2017 arXiv

  46. [54]

    Learn to explain: Multimodal reason- ing via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai- Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reason- ing via thought chains for science question answering. In NeurIPS, 2022. 3

    2022

  47. [55]

    ChartQA: A benchmark for question answering about charts with visual and logical reasoning

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In ACL, 2022. 2, 5, 6, 1, 3, 4

    2022

  48. [56]

    Docvqa: A dataset for vqa on document images

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In WACV,

  49. [57]

    Infographicvqa

    Minesh Mathew, Viraj Bagal, Rub `en Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographicvqa. In WACV, 2022. 5, 3

    2022

  50. [58]

    MM1: Methods, analysis & insights from multi- modal LLM pre-training

    Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, Anton Belyi, Hao- tian Zhang, Karanjeet Singh, Doug Kang, Ankur Jain, Hongyu H `e, Max Schwarzer, Tom Gunter, Xiang Kong, Aonan Zhan...

    arXiv

  51. [59]

    Architecture of connectivity within a cingulo- fronto-parietal neurocognitive network for directed atten- tion

    Robert J Morecraft, Changiz Geula, and M-Marsel Mesu- lam. Architecture of connectivity within a cingulo- fronto-parietal neurocognitive network for directed atten- tion. Archives of neurology, 1993. 1

    1993

  52. [60]

    Brain mechanisms for directed at- tention

    Vernon B Mountcastle. Brain mechanisms for directed at- tention. Journal of the Royal Society of Medicine , 1978. 1

    1978

  53. [61]

    Visual-O1: Understanding ambiguous instructions via multi-modal multi-turn chain-of-thoughts reasoning

    Minheng Ni, Yutao Fan, Lei Zhang, and Wangmeng Zuo. Visual-O1: Understanding ambiguous instructions via multi-modal multi-turn chain-of-thoughts reasoning. arXiv preprint arXiv:2410.03321, 2024. 3

    2024 arXiv

  54. [62]

    Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V . V o, Marc Szafraniec, Vasil Khalidov, Pierre Fer- nandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El- Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Mich...

    2024

  55. [63]

    Kosmos-2: Ground- ing multimodal large language models to the world

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Ground- ing multimodal large language models to the world. arXiv preprint arXiv:2306.14824, 2023. 5, 3

    2023 arXiv

  56. [64]

    Flickr30k entities: Collecting region-to-phrase corre- spondences for richer image-to-sentence models

    Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazeb- nik. Flickr30k entities: Collecting region-to-phrase corre- spondences for richer image-to-sentence models. In ICCV,

  57. [65]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021. 2, 3, 5

    2021

  58. [66]

    GLaMM: Pixel grounding large multimodal model

    Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdel- rahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S Khan. GLaMM: Pixel grounding large multimodal model. In CVPR, 2024. 3, 6

    2024

  59. [67]

    Grounding DINO 1.5: Advance the edge of open-set object detection

    Tianhe Ren, Qing Jiang, Shilong Liu, Zhaoyang Zeng, Wenlong Liu, Han Gao, Hongjie Huang, Zhengyu Ma, Xi- aoke Jiang, Yihao Chen, Yuda Xiong, Hao Zhang, Feng Li, Peijun Tang, Kent Yu, and Lei Zhang. Grounding DINO 1.5: Advance the edge of open-set object detection. arXiv prepri...

    2024 arXiv

  60. [68]

    Grounded sam: Assembling open-world models for diverse visual tasks,

    Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kun- chang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, Zhaoyang Zeng, Hao Zhang, Feng Li, Jie Yang, Hongyang Li, Qing Jiang, and Lei Zhang. Grounded sam: Assembling open-world models for diverse visual tasks,

  61. [69]

    Visual chain of thought: bridging logical gaps with multimodal infill- ings

    Daniel Rose, Vaishnavi Himakunthala, Andy Ouyang, Ryan He, Alex Mei, Yujie Lu, Michael Saxon, Chinmay Sonar, Diba Mirza, and William Yang Wang. Visual chain of thought: bridging logical gaps with multimodal infill- ings. arXiv preprint arXiv:2305.02317, 2023. 3

    2023 arXiv

  62. [70]

    Vi- sual CoT: advancing multi-modal language models with a comprehensive dataset and benchmark for chain-of-thought reasoning

    Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuo- fan Zong, Letian Wang, Yu Liu, and Hongsheng Li. Vi- sual CoT: advancing multi-modal language models with a comprehensive dataset and benchmark for chain-of-thought reasoning. In NeurIPS, 2024. 3, 4, 5, 6

    2024

  63. [71]

    Objects365: A large-scale, high-quality dataset for object detection

    Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In ICCV, 2019. 2

    2019

  64. [72]

    Eagle: Explor- ing the design space for multimodal llms with mixture of encoders

    Min Shi, Fuxiao Liu, Shihao Wang, Shijia Liao, Subhashree Radhakrishnan, De-An Huang, Hongxu Yin, Karan Sapra, Yaser Yacoob, Humphrey Shi, Bryan Catanzaro, Andrew Tao, Jan Kautz, Zhiding Yu, and Guilin Liu. Eagle: Explor- ing the design space for multimodal llms with mixture o...

    2025

  65. [73]

    Textcaps: a dataset for image caption- ing with reading comprehension

    Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and 11 Amanpreet Singh. Textcaps: a dataset for image caption- ing with reading comprehension. In ECCV, 2020. 5, 3

    2020

  66. [74]

    Towards VQA models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards VQA models that can read. In CVPR,

  67. [75]

    Introducing the next generation of claude,

    Anthropic Team. Introducing the next generation of claude,

  68. [76]

    Chameleon: Mixed-modal early-fusion foundation models

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818 ,

    arXiv

  69. [77]

    Emu3: Next-token prediction is all you need

    Emu3 Team. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409.18869, 2024. 3

    2024 arXiv

  70. [78]

    Gemini: A family of highly capable multi- modal models

    Gemini Team. Gemini: A family of highly capable multi- modal models. arXiv preprint arXiv:2312.11805, 2023. 1, 2

    2023 arXiv

  71. [79]

    Gemini 1.5: Unlocking multimodal un- derstanding across millions of tokens of context

    Gemini Team. Gemini 1.5: Unlocking multimodal un- derstanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024. 2

    2024 arXiv

  72. [80]

    Laion-gpt4v dataset, 2023

    LAION Team. Laion-gpt4v dataset, 2023. 5, 3

    2023

  73. [81]

    The Llama 3 herd of models

    Meta Team. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. 3, 5, 2, 4

    2024 arXiv

  74. [82]

    GPT-4 technical report

    OpenAI Team. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023. 1, 2, 3, 6

    2023 arXiv

  75. [83]

    InternVL2: Better than the best—expanding performance boundaries of open-source multimodal models with the progressive scaling strategy,

    OpenGVLab Team. InternVL2: Better than the best—expanding performance boundaries of open-source multimodal models with the progressive scaling strategy,

  76. [84]

    Openhermes 2.5: An open dataset of synthetic data for generalist llm assistants., 2023

    Teknium. Openhermes 2.5: An open dataset of synthetic data for generalist llm assistants., 2023. 5, 3

    2023

  77. [85]

    Cambrian-1: A fully open, vision-centric exploration of multimodal llms

    Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, Austin Wang, Rob Fergus, Yann LeCun, and Saining Xie. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. arXiv...

    2024 arXiv

  78. [86]

    Eyes wide shut? exploring the visual shortcomings of multimodal LLMs

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal LLMs. In CVPR, 2024. 1, 2, 6

    2024

  79. [87]

    Document understanding dataset and evalu- ation (dude)

    Jordy Van Landeghem, Rub `en Tito, Łukasz Borchmann, Michał Pietruszka, Pawel Joziak, Rafal Powalski, Dawid Jurkiewicz, Mickael Coustaty, Bertrand Anckaert, Ernest Valveny, Matthew Blaschko, Sien Moens, and Tomasz Stanislawek. Document understanding dataset and evalu- ation (d...

    2023

  80. [88]

    C. Wah, S. Branson, P. Welinder, P. Perona, and S. Be- longie. The caltech-ucsd birds-200-2011 dataset. Technical Report CNS-TR-2011-001, California Institute of Technol- ogy, 2011. 5, 3

    2011

  81. [89]

    To see is to believe: Prompting GPT-4v for better visual instruction tuning

    Junke Wang, Lingchen Meng, Zejia Weng, Bo He, Zuxuan Wu, and Yu-Gang Jiang. To see is to believe: Prompting GPT-4v for better visual instruction tuning. arXiv preprint arXiv:2311.07574, 2023. 5, 3

    2023 arXiv

  82. [90]

    OFA: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework

    Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. OFA: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In ICML, 2022. 6, 7

    2022

  83. [91]

    Qwen2-VL: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-VL: Enhancing vision-language model’s pe...

    2024 arXiv

  84. [92]

    CogVLM: Visual expert for pretrained language models

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. CogVLM: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03079, 2023. 3

    2023 arXiv

  85. [93]

    Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In ICLR, 2023. 3

    2023

  86. [94]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS, 2022. 3

    2022

  87. [95]

    Visual ChatGPT: Talking, drawing and editing with visual foundation mod- els

    Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual ChatGPT: Talking, drawing and editing with visual foundation mod- els. arXiv preprint arXiv:2303.04671, 2023. 3

    2023 arXiv

  88. [96]

    V*: Guided visual search as a core mechanism in multimodal LLMs

    Penghao Wu and Saining Xie. V*: Guided visual search as a core mechanism in multimodal LLMs. In CVPR, 2024. 2, 3, 6, 7, 1, 4

    2024

  89. [97]

    Grok, 2024

    xAI Team. Grok, 2024. 1, 2, 6

    2024

  90. [98]

    Florence-2: Advancing a unified representation for a va- riety of vision tasks

    Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan. Florence-2: Advancing a unified representation for a va- riety of vision tasks. In CVPR, 2024. 3

    2024

  91. [99]

    Denoising vision trans- formers

    Jiawei Yang, Katie Z Luo, Jiefeng Li, Congyue Deng, Leonidas Guibas, Dilip Krishnan, Kilian Q Weinberger, Yonglong Tian, and Yue Wang. Denoising vision trans- formers. In ECCV, 2024. 4

    2024

  92. [100]

    UniTAB: Unifying text and box outputs for grounded vision-language modeling

    Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Faisal Ahmed, Zicheng Liu, Yumao Lu, and Lijuan Wang. UniTAB: Unifying text and box outputs for grounded vision-language modeling. In ECCV, 2022. 6, 7

    2022

  93. [101]

    Tree of thoughts: Deliberate problem solving with large language models

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In NeurIPS, 2024. 3

    2024

  94. [102]

    Ferret: Refer and ground any- thing anywhere at any granularity

    Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground any- thing anywhere at any granularity. In ICLR, 2023. 2, 3, 4, 5, 6, 7

    2023

  95. [103]

    Modeling context in referring expres- sions

    Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expres- sions. In ECCV, 2016. 2, 5, 6, 3

    2016

  96. [104]

    Mattnet: Modular at- 12 tention network for referring expression comprehension

    Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L Berg. Mattnet: Modular at- 12 tention network for referring expression comprehension. In CVPR, 2018. 6, 7

    2018

  97. [105]

    Mmmu: A massive multi-discipline multi- modal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A m...

    2024

  98. [106]

    Llava-grounding: Grounded visual chat with large multimodal models

    Hao Zhang, Hongyang Li, Feng Li, Tianhe Ren, Xueyan Zou, Shilong Liu, Shijia Huang, Jianfeng Gao, Lei Zhang, Chunyuan Li, and Jianwei Yang. Llava-grounding: Grounded visual chat with large multimodal models. In ECCV, 2024. 4

    2024

  99. [107]

    Ferret-v2: An improved baseline for referring and grounding with large language models

    Haotian Zhang, Haoxuan You, Philipp Dufter, Bowen Zhang, Chen Chen, Hong-You Chen, Tsu-Jui Fu, William Yang Wang, Shih-Fu Chang, Zhe Gan, and Yinfei Yang. Ferret-v2: An improved baseline for referring and grounding with large language models. In COLM, 2024. 2, 4

    2024

  100. [108]

    Improve vision language model chain- of-thought reasoning

    Ruohong Zhang, Bowen Zhang, Yanghao Li, Haotian Zhang, Zhiqing Sun, Zhe Gan, Yinfei Yang, Ruoming Pang, and Yiming Yang. Improve vision language model chain- of-thought reasoning. arXiv preprint arXiv:2410.16198 ,

    arXiv

  101. [109]

    LLaV AR: En- hanced visual instruction tuning for text-rich image under- standing

    Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. LLaV AR: En- hanced visual instruction tuning for text-rich image under- standing. arXiv preprint arXiv:2306.17107, 2023. 5, 3

    2023 arXiv

  102. [110]

    GroundHog: Grounding large language models to holistic segmentation

    Yichi Zhang, Ziqiao Ma, Xiaofeng Gao, Suhaila Shakiah, Qiaozi Gao, and Joyce Chai. GroundHog: Grounding large language models to holistic segmentation. In CVPR, 2024. 3, 4

    2024

  103. [111]

    Automatic chain of thought prompting in large language models

    Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. In ICLR, 2023. 3

    2023

  104. [112]

    Multimodal chain-of- thought reasoning in language models

    Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal chain-of- thought reasoning in language models. arXiv preprint arXiv:2302.00923, 2023. 3

    2023 arXiv

  105. [113]

    PSALM: Pixelwise segmentation with large multi-modal model

    Zheng Zhang, Yeyao Ma, Enming Zhang, and Xiang Bai. PSALM: Pixelwise segmentation with large multi-modal model. In ECCV, 2024. 3, 4

    2024

  106. [114]

    DdCoT: Duty-distinct chain-of-thought prompting for multimodal reasoning in language models

    Ge Zheng, Bin Yang, Jiajin Tang, Hong-Yu Zhou, and Sibei Yang. DdCoT: Duty-distinct chain-of-thought prompting for multimodal reasoning in language models. In NeurIPS,

  107. [115]

    Visual7w: Grounded question answering in images

    Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei- Fei. Visual7w: Grounded question answering in images. In CVPR, 2016. 5, 3 13 ARGUS : Vision-Centric Reasoning with Grounded Chain-of-Thought Supplementary Material Re-encoding Re-sampling Method ChartQA V-Star ChartQA V-St...

    2016

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.