Pith. sign in

REVIEW 2 major objections 4 minor 62 references

Describe Anything Model for Visual Question Answering on Text-rich Images

T0 review · 2 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper shows that the Describe Anything Model, designed to describe arbitrary image regions, can do zero-shot text-rich VQA by answering on the full image and on overlapping sliding-window patches and merging exact-match answers with…

desk verdict A clean, reproducible sliding-window voting recipe for DAM that shows a real but partly selected DocVQA gain; the abstract overstates consistency. read the letter →

arxiv 2507.12441 v2 pith:NV5K3BJR submitted 2025-07-16 cs.CV cs.LG

Yen-Linh Vu , Dinh-Thang Duong , Truong-Binh Duong , Anh-Khoi Nguyen , Thanh-Huy Nguyen , Le Thien Phuc Nguyen , Jianhua Xing , Xingjian Li
show 3 more authors
Tianyang Wang Ulas Bagci Min Xu
This is my paper
classification cs.CVcs.LG
keywords VisualQuestionAnsweringtext-richimagesDescribeAnythingModelregion-awarevision-languagemodelsslidingwindowweightedvotingzero-shotVQADoc
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's aim is to show that a region-aware vision-language model trained to describe arbitrary image regions can be turned into a competitive zero-shot VQA system on text-rich images without retraining. The proposed DAM-QA protocol feeds the Describe Anything Model the whole image and a set of overlapping 512×512 crops, asks the same question on each view with a prompt that permits the answer 'unanswerable', and merges the answers by weighted voting. On the paper's evaluations the protocol beats plain DAM on four of six benchmarks, with the largest gain on DocVQA (ANLS 42.34 vs 35.22), and surpasses other region-aware models despite using only 3B parameters; on ChartQAPro and VQAv2 the reported scores are slightly below the DAM baseline. If the effect is real, it matters because it offers a cheap, training-free route to better reading of dense text in documents, charts, and natural scenes, narrowing the gap between small region-aware models and much larger generalist VLMs.

What carries the argument

The load-bearing object is the Describe Anything Model (DAM), a region-aware vision-language model that takes an image plus a point, box, or mask prompt and writes a detailed description of the indicated area; DAM-QA reuses it unchanged. The new mechanism is the sliding-window voting protocol: after resizing, crops are taken at 512×512 pixels with a 256-pixel stride, each crop and the full image are queried with a four-part VQA prompt that asks for a short answer and allows 'unanswerable', and votes are weighted so the full image counts as 1, each answering patch counts as its pixel area divided by the resized image area, and 'unanswerable' patches count as 0. This mechanism carries the argument because the gains are attributed to small text becoming visible in at least one local view and to abstaining patches no longer diluting the majority answer.

What would settle it

Run DAM and DAM-QA with the paper's fixed settings on a text-rich VQA benchmark not used in the ablations, or on a held-out split of DocVQA; if the aggregated voting answers no longer beat DAM's single full-image answers under the official metric, the central claim is unsupported.

Watch

Extended reading notes

Core claim

The central claim is that fine-grained region description, which DAM was built for, transfers to text-rich VQA when the model is given multiple overlapping views of the image at inference time. The recipe is: resize the image to a 1024-pixel longest side, extract 512×512 patches with a 256-pixel stride, run DAM on the full-image mask and on each patch with the same question, count exact-match votes, give the full-image answer weight 1 and each patch answer weight proportional to patch area, and ignore all patch answers that say 'unanswerable'. On the paper's six benchmarks this raises the official score over plain DAM on DocVQA, InfographicVQA, TextVQA, and ChartQA, and the DocVQA ANLS improves from 35.22 to 42.34. The claimed implication is that a descriptive region-aware model can serve as a strong zero-shot VQA baseline with no OCR pipeline and no fine-tuning, while remaining at 3B parameters; the paper's own tables show small dips on ChartQAPro and VQAv2 that the text summarizes as an overall consistent improvement.

Load-bearing premise

The reported gains rest on choices — the 512-pixel patch size, the 256-pixel stride, and the zero weight for 'unanswerable' votes — that the authors selected by ablating on the same benchmarks whose final scores are reported, so the improvement may not generalize to unseen text-rich data.

Editorial extensions

If this is right

  • If the central claim holds, any region-aware model that accepts image-plus-mask inputs can be upgraded to text-rich VQA by the same training-free voting wrapper, not just DAM.
  • The 7+ point DocVQA gain shows that inference-time multi-view aggregation can close a substantial part of the gap between a 3B region-aware model and 7B generalist VLMs on document-style questions.
  • The ablation results imply that patch granularity and the treatment of 'unanswerable' votes are the main controls on the gain: too fine a patch size (256×128) or any positive weight on abstentions drops official scores sharply.
  • Because the prompt's abstention rule is what prevents unsupported guesses, the protocol is most useful on benchmarks where answers are extractable and hallucination is penalized, such as DocVQA and TextVQA.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • In our reading, the same area-weighted voting recipe is a natural test for generalist VLMs too: if their failures on dense text are partly a resolution problem, multi-crop voting should improve them even without the region-aware mask prompts.
  • The exact-match vote counter cannot merge different but correct wordings, so an LLM-based answer-alignment step before voting would likely raise the reported gains; that extension is ours, not the paper's.
  • The paper tunes the zero weight for 'unanswerable' on document and chart benchmarks; a fair held-out application would likely need to re-derive that weight per domain, since ChartQAPro behaves differently from DocVQA.
  • A concrete next test the paper leaves implicit is whether patch answers are most useful when they contain a different text span from the full image; cropping strategies targeted at detected text regions might outperform the uniform grid.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper introduces DAM-QA, a zero-shot VQA approach that applies the region-aware Describe Anything Model (DAM) to a full image and to overlapping sliding-window patches, then aggregates exact-match answers by area-weighted voting, with patch-level "unanswerable" predictions assigned zero weight. The authors evaluate on six VQA benchmarks and claim consistent improvement over the DAM baseline, including a 7+ point ANLS gain on DocVQA, and report the best overall performance among region-aware models. The code is publicly available.

Significance. If the reported gains are genuine, the paper offers a simple, training-free recipe for improving text-rich VQA with region-aware models, and the release of code is a practical strength. The method is straightforward and the ablations are transparent. However, the central empirical claim is weakened by two issues: the hyperparameters are selected using the same benchmarks that are then used for the final scoring, and the abstract's "consistently outperforms" statement is contradicted by the paper's own Table 2 on two datasets. These issues are addressable in revision, so the work has potential but currently requires stronger validation.

major comments (2)
  1. [Section 4.5.2, Section 4.5.3, Tables 4 and 5, and Table 2] The sliding-window parameters (patch size 512, stride 256) and the abstention weight (0.0) are selected by ablating on DocVQA, TextVQA, ChartQA, and VQAv2, which are the same benchmarks whose final scores are reported in Table 2 as evidence of improvement. Because no held-out validation set is used, the reported gains may be inflated by selection bias. For example, Table 4 shows DocVQA ANLS varying from 36.02 (256/128) to 43.22 (512/128) across configurations, and Table 5 shows DocVQA ANLS varying from 17.82 to 42.34 across the abstention weight; the chosen configuration corresponds to the best or near-best value on these test sets. I request a held-out validation split (or a separate dataset) for configuration selection, and a report of all configurations evaluated so the reader can quantify the selection effect.
  2. [Abstract and Section 5 (Conclusion), with Table 2] The claim that DAM-QA "consistently outperforms the baseline DAM" is contradicted by Table 2: on ChartQAPro, DAM-QA scores 14.88 RAcc versus 18.90 for DAM, and on VQAv2, DAM-QA scores 79.20 VQAS versus 79.25 for DAM. The LLM scores show a similar pattern on VQAv2 (82.51 vs 82.57). The claim should be revised to something like "outperforms the baseline on most benchmarks (DocVQA, InfographicVQA, TextVQA, ChartQA), with a 7+ point gain on DocVQA," and the paper should include confidence intervals or significance tests to establish that the reported differences are not due to noise.
minor comments (4)
  1. [Tables 2, 3, and 5] The DAM baseline for DocVQA ANLS is reported as 35.22 in Table 2 but 34.84 in both Table 3 and Table 5. Since all rows use the same prompt and model, please reconcile this discrepancy and state whether the ablations were run on a different subset or with different generation settings.
  2. [Section 4.5.2, Table 4] The text states that the 512x256 configuration "strikes the optimal balance," but Table 4 shows that 512/128 gives a higher DocVQA ANLS (43.22 vs 42.34) and a higher LLM score (35.79 vs 32.18); please clarify whether the selection is based on the average over all three benchmarks or on a predefined criterion.
  3. [Section 3.3.2] When every patch is "unanswerable" and the full-image output is also "unanswerable," the final answer is "unanswerable." Please clarify how these cases are treated by the benchmark-specific metrics, particularly ChartQAPro, where unanswerable questions have a specific evaluation rule.
  4. [Section 4.4 and Table 2] The claim of "best overall performance among region-aware models" should be qualified, since ViP-LLaVA achieves a higher ANLS on InfographicVQA (22.31 vs 20.25 for DAM-QA); the "overall" statement presumably refers to an average across benchmarks, which should be made explicit.

Circularity Check

1 steps flagged · score 4.0 of 10

Main reported gains are partly selection-optimized: the patch size, stride, and abstention weight are tuned on the same benchmarks whose final scores are then presented as evidence, though the voting mechanism itself is not definitionally circular.

  1. fitted input called prediction [Section 4.5.2 (Table 4), Section 4.5.3 (Table 5), and Section 4.4 (Table 2), together with the Abstract.]
    "Section 3.3 introduces our sliding-window protocol, and Table 4 evaluates its key parameters on DocVQA, TextVQA and ChartQA. ... These results show that 512 × 256 strikes the optimal balance between fine detail and global context. ... we vary the vote weight (from 0.0 to 1.5) assigned to 'unanswerable' patch predictions in our ablation study on three benchmarks: DocVQA, TextVQA, and VQAv2. ... Performance consistently peaks at weight 0.0. ... Experiments on six VQA benchmarks show that our approach consistently outperforms the baseline DAM, with a notable 7+ point gain on DocVQA."

    The hyperparameters of the proposed recipe (patch size 512, stride 256, and zero weight for 'unanswerable' patch votes) are selected by running ablations on DocVQA, TextVQA, ChartQA, and VQAv2 and then reporting the best-performing configuration as DAM-QA in Table 2 on those same benchmarks. Thus the headline 7+ point DocVQA gain (ANLS 42.34 vs 35.22) is not a parameter-free prediction: it is the outcome of a configuration evaluated on the same data that were used to make the choice.

full rationale

The paper contains no self-citation chain, no imported uniqueness theorem, and no ansatz smuggled in via citation. The core method (sliding-window multi-view inference with exact-match, area-weighted voting) is defined cleanly in Equations 1-7, and the final answer is not definitionally tied to any ground-truth label. The only substantive circularity-adjacent issue is the evaluation protocol: Section 4.5.2 and Section 4.5.3 select the patch size, stride, and abstention vote weight using the same benchmarks (DocVQA, TextVQA, ChartQA, VQAv2) that later appear as evidence in Table 2 and the Abstract's 7+ point claim. Because the choice is made from a small grid, the expected best configuration will look better than its true average performance; this is a generalization and selection-bias problem rather than a formal equivalence. I therefore score it 4: one fitted-input-called-prediction pattern of moderate force, while acknowledging that the voting mechanism has independent content and that the authors disclose the ablations rather than hide them. No other circular steps were found.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The method introduces no new model components or entities. Its free parameters are the sliding-window geometry and the abstention vote weight, all selected by ablation on the evaluation benchmarks. The main domain assumptions concern DAM's descriptive reliability and the validity of exact-match voting.

free parameters (3)
  • patch_size = 512x512
    Selected via ablation on DocVQA, TextVQA, and ChartQA (Table 4); no held-out validation is used.
  • stride = 256
    Selected via ablation on the same benchmarks (Table 4).
  • unanswerable_vote_weight = 0.0
    Selected via ablation on DocVQA, TextVQA, and VQAv2 (Table 5); the paper's own data show large drops for nonzero weights.
assumptions (3)
  • domain assumption DAM's region-level descriptions are informative enough to serve as VQA answers for text-heavy images.
    Central premise of the method; the paper provides no independent analysis of DAM's reliability on dense text beyond the reported benchmarks.
  • domain assumption Exact string matching between predicted and reference answers is a valid basis for vote aggregation.
    The voting scheme treats only identical strings as equivalent, which may undercount semantically equivalent correct answers.
  • ad hoc to paper The 'unanswerable' output from DAM is a reliable abstention signal across patches.
    Justified only through Table 5, which is tuned on the same benchmarks, so the reliability of the abstention signal is not independently established.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Describe Anything Model for Visual Question Answering on Text-rich Images." pith.science (2026). https://pith.science/paper/NV5K3BJR

@misc{pith2026250712441,
  author       = {Pith},
  title        = {Pith review of: Describe Anything Model for Visual Question Answering on Text-rich Images},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NV5K3BJR}},
  note         = {Machine review of arXiv:2507.12441}
}
read the original abstract

Recent progress has been made in region-aware vision-language modeling, particularly with the emergence of the Describe Anything Model (DAM). DAM is capable of generating detailed descriptions of any specific image areas or objects without the need for additional localized image-text alignment supervision. We hypothesize that such region-level descriptive capability is beneficial for the task of Visual Question Answering (VQA), especially in challenging scenarios involving images with dense text. In such settings, the fine-grained extraction of textual information is crucial to producing correct answers. Motivated by this, we introduce DAM-QA, a framework with a tailored evaluation protocol, developed to investigate and harness the region-aware capabilities from DAM for the text-rich VQA problem that requires reasoning over text-based information within images. DAM-QA incorporates a mechanism that aggregates answers from multiple regional views of image content, enabling more effective identification of evidence that may be tied to text-related elements. Experiments on six VQA benchmarks show that our approach consistently outperforms the baseline DAM, with a notable 7+ point gain on DocVQA. DAM-QA also achieves the best overall performance among region-aware models with fewer parameters, significantly narrowing the gap with strong generalist VLMs. These results highlight the potential of DAM-like models for text-rich and broader VQA tasks when paired with efficient usage and integration strategies. Our code is publicly available at https://github.com/Linvyl/DAM-QA.git.

Figures

Figures reproduced from arXiv: 2507.12441 by the authors.

Figure 1
Figure 1. DAM-QA handles text-rich VQA by applying DAM [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of the proposed DAM-QA framework (a) Patch Generation (3.3.1): An input image is processed with a sliding window to generate a full-image view alongside multiple overlapping local patches. (b) Weighted Voting Scheme (3.3.2): All views are fed into the DAM model with a unified VQA prompt. The resulting answers are aggregated via a weighted voting system, where “unanswerable” predictions are assigned a weight… view at source ↗
Figure 3
Figure 3. An illustration of the prompt construction used for eval [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: LLM judge prompt template with placeholders (in red) [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Qualitative results on an InfographicVQA [ [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

62 extracted references · 31 canonical work pages

  1. [1]

    Phi-4-mini technical report: Compact yet powerful multi- modal language models via mixture-of-loras

    Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Ben- haim, Martin Cai, Vishrav Chaudhary, Congcong Chen, et al. Phi-4-mini technical report: Compact yet powerful multi- modal language models via mixture-of-loras. arXiv preprint arXiv:2503.01743, 2025. 2, 7

    arXiv 2025

  2. [2]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Men- sch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736,

  3. [3]

    Lawrence Zitnick, and Devi Parikh

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In 2015 IEEE In- ternational Conference on Computer Vision (ICCV) , pages 2425–2433, 2015. 4

    work page 2015

  4. [4]

    Manmatha

    Srikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie, and R. Manmatha. Docformer: End-to-end transformer for document understanding. In2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 973–983, 2021. 1

    work page 2021

  5. [5]

    Manmatha

    Srikar Appalaraju, Peng Tang, Qi Dong, Nishant Sankaran, Yichu Zhou, and R. Manmatha. Docformerv2: Local fea- tures for document understanding. Proceedings of the AAAI Conference on Artificial Intelligence , 38(2):709–718, 2024. 2

    work page 2024

  6. [6]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025. 2, 7

    arXiv 2025

  7. [7]

    Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluation

    Jannis Bulian, Christian Buck, Wojciech Gajewski, Ben- jamin Boerschinger, and Tal Schuster. Tomayto, tomahto. beyond token-level answer equivalence for question answer- ing evaluation. arXiv preprint arXiv:2202.07654, 2022. 6

    work page Pith review arXiv 2022

  8. [8]

    Meyer, Yuning Chai, Dennis Park, and Yong Jae Lee

    Mu Cai, Haotian Liu, Siva Karthik Mustikovela, Gregory P. Meyer, Yuning Chai, Dennis Park, and Yong Jae Lee. Vip- llava: Making large multimodal models understand arbitrary visual prompts. In 2024 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 12914– 12923, 2024. 2, 7

    work page 2024

Show all 62 references
  1. [9]

    Shikra: Unleashing multi- modal llm’s referential dialogue magic

    Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multi- modal llm’s referential dialogue magic. arXiv preprint arXiv:2306.15195, 2023. 2, 7

    2023 arXiv

  2. [10]

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Albert Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: To- wards general-purpose vision-language models with instruc- tion tuning. ArXiv, abs/2305.06500, 2023. 2

    2023 arXiv

  3. [11]

    Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models

    Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tri- pathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models. arXiv preprint arXiv:2409.17146, 2024. 7

    2024 arXiv

  4. [12]

    Scalable vision language model training via high quality data curation

    Hongyuan Dong, Zijian Kang, Weijie Yin, Xiao Liang, Chao Feng, and Jiao Ran. Scalable vision language model training via high quality data curation. arXiv preprint arXiv:2501.05952, 2025. 2

    2025 arXiv

  5. [13]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Ba- tra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing. In 2017 IEEE Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 6325–6334, 201...

    2017

  6. [14]

    A survey on llm-as-a-judge

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xue- hao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594, 2024. 6

    2024 arXiv

  7. [15]

    Regiongpt: Towards region understanding vision lan- guage model

    Qiushan Guo, Shalini De Mello, Hongxu Yin, Wonmin Byeon, Ka Chun Cheung, Yizhou Yu, Ping Luo, and Sifei Liu. Regiongpt: Towards region understanding vision lan- guage model. In 2024 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 13796– 13806, 2024. 2

    2024

  8. [16]

    Hires-llava: Restoring fragmen- tation input in high-resolution large vision-language models

    Runhui Huang, Xinpeng Ding, Chunwei Wang, Jianhua Han, Yulong Liu, Hengshuang Zhao, Hang Xu, Lu Hou, Wei Zhang, and Xiaodan Liang. Hires-llava: Restoring fragmen- tation input in high-resolution large vision-language models. In Proceedings of the Computer Vision and Pattern Re...

    2025

  9. [17]

    Seeing out of the box: End- to-end pre-training for vision-language representation learn- ing

    Zhicheng Huang, Zhaoyang Zeng, Yupan Huang, Bei Liu, Dongmei Fu, and Jianlong Fu. Seeing out of the box: End- to-end pre-training for vision-language representation learn- ing. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12971–12980, 2021. 2

    2021

  10. [18]

    Multi-agent vqa: Exploring multi-agent foundation models in zero-shot visual question answering

    Bowen Jiang, Zhijun Zhuang, Shreyas S Shivakumar, Dan Roth, and Camillo J Taylor. Multi-agent vqa: Exploring multi-agent foundation models in zero-shot visual question answering. arXiv preprint arXiv:2403.14783, 2024. 4

    2024 arXiv

  11. [19]

    Spa- tially aware multimodal transformers for textvqa

    Yash Kant, Dhruv Batra, Peter Anderson, Alexander Schwing, Devi Parikh, Jiasen Lu, and Harsh Agrawal. Spa- tially aware multimodal transformers for textvqa. In ECCV,

  12. [20]

    Binary codes capable of cor- recting deletions, insertions, and reversals

    Vladimir I Levenshtein et al. Binary codes capable of cor- recting deletions, insertions, and reversals. In Soviet physics doklady, pages 707–710. Soviet Union, 1966. 6

    1966

  13. [21]

    From gener- ation to judgment: Opportunities and challenges of llm-as-a- judge

    Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, et al. From gener- ation to judgment: Opportunities and challenges of llm-as-a- judge. arXiv preprint arXiv:2411.16594, 2024. 6

    2024

  14. [22]

    Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models

    Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07895, 2024. 7

    2024 arXiv

  15. [23]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In Interna- tional conference on machine learning, pages 12888–12900. PMLR, 2022. 2

    2022

  16. [24]

    Blip-2: Bootstrapping language-image pre-training with 9 frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with 9 frozen image encoders and large language models. In In- ternational conference on machine learning , pages 19730– 19742. PMLR, 2023. 2

    2023

  17. [25]

    Visualbert: A simple and perfor- mant baseline for vision and language

    Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and perfor- mant baseline for vision and language. arXiv preprint arXiv:1908.03557, 2019. 2

    1908 arXiv

  18. [26]

    Unimo: Towards unified-modal understanding and generation via cross-modal contrastive learning

    Wei Li, Can Gao, Guocheng Niu, Xinyan Xiao, Hao Liu, Jiachen Liu, Hua Wu, and Haifeng Wang. Unimo: Towards unified-modal understanding and generation via cross-modal contrastive learning. arXiv preprint arXiv:2012.15409 ,

    2012 arXiv

  19. [27]

    Oscar: Object-semantics aligned pre-training for vision-language tasks

    Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28,...

    2020

  20. [28]

    Enhancing visual document understanding with contrastive learning in large visual-language models

    Xin Li, Yunfei Wu, Xinghua Jiang, Zhihao Guo, Ming- ming Gong, Haoyu Cao, Yinsong Liu, Deqiang Jiang, and Xing Sun. Enhancing visual document understanding with contrastive learning in large visual-language models. In 2024 IEEE/CVF Conference on Computer Vision and Pat- tern R...

    2024

  21. [29]

    Mon- key: Image resolution and text label are important things for large multi-modal models

    Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Mon- key: Image resolution and text label are important things for large multi-modal models. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),...

    2024

  22. [30]

    Describe anything: Detailed localized image and video captioning

    Long Lian, Yifan Ding, Yunhao Ge, Sifei Liu, Hanzi Mao, Boyi Li, Marco Pavone, Ming-Yu Liu, Trevor Darrell, Adam Yala, et al. Describe anything: Detailed localized image and video captioning. arXiv preprint arXiv:2504.16072, 2025. 2, 3, 7, 8

    2025 arXiv

  23. [31]

    Perceive anything: Recognize, explain, caption, and segment anything in images and videos

    Weifeng Lin, Xinyu Wei, Ruichuan An, Tianhe Ren, Tingwei Chen, Renrui Zhang, Ziyu Guo, Wentao Zhang, Lei Zhang, and Hongsheng Li. Perceive anything: Recognize, explain, caption, and segment anything in images and videos. arXiv preprint arXiv:2506.05302, 2025. 2, 7

    2025 arXiv

  24. [32]

    Revive: regional visual representa- tion matters in knowledge-based visual question answering

    Yuanze Lin, Yujia Xie, Dongdong Chen, Yichong Xu, Chen- guang Zhu, and Lu Yuan. Revive: regional visual representa- tion matters in knowledge-based visual question answering. In Proceedings of the 36th International Conference on Neu- ral Information Processing Systems , Red H...

  25. [33]

    Textmonkey: An ocr-free large multimodal model for understanding document

    Yuliang Liu, Biao Yang, Qiang Liu, Zhang Li, Zhiyin Ma, Shuo Zhang, and Xiang Bai. Textmonkey: An ocr-free large multimodal model for understanding document. arXiv preprint arXiv:2403.04473, 2024. 2

    2024 arXiv

  26. [34]

    Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks

    Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. Advances in neural information processing systems, 32, 2019. 2

    2019

  27. [35]

    Ovis: Structural em- bedding alignment for multimodal large language model

    Shiyin Lu, Yang Li, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Han-Jia Ye. Ovis: Structural em- bedding alignment for multimodal large language model. arXiv:2405.20797, 2024. 7

    2024 arXiv

  28. [36]

    Chartqa: A benchmark for question an- swering about charts with visual and logical reasoning

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question an- swering about charts with visual and logical reasoning. ACL Findings, pages 2263–2279, 2022. 1, 5

    2022

  29. [37]

    Chartqapro: A more di- verse and challenging benchmark for chart question answer- ing

    Ahmed Masry, Mohammed Saidul Islam, Mahir Ahmed, Aayush Bajaj, Firoz Kabir, Aaryaman Kartha, Md Tah- mid Rahman Laskar, Mizanur Rahman, Shadikur Rahman, Mehrad Shahmohammadi, et al. Chartqapro: A more di- verse and challenging benchmark for chart question answer- ing. arXiv pr...

    2025 arXiv

  30. [38]

    Docvqa: A dataset for vqa on document images

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In WACV, pages 2200–2209, 2021. 1, 2, 5

    2021

  31. [39]

    Infographicvqa

    Minesh Mathew, Viraj Bagal, Rub `en Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographicvqa. In WACV, pages 1697–1706, 2022. 1, 5, 8

    2022

  32. [40]

    Im- proving automatic vqa evaluation using large language mod- els

    Oscar Ma ˜nas, Benno Krojer, and Aishwarya Agrawal. Im- proving automatic vqa evaluation using large language mod- els. Proceedings of the AAAI Conference on Artificial Intel- ligence, 38(5):4171–4179, 2024. 6

    2024

  33. [41]

    Dual dynamic consis- tency regularization for semi-supervised domain adaptation

    Ba Hung Ngo, Ba Thinh Lam, Thanh Huy Nguyen, Quang Vinh Dinh, and Tae Jong Choi. Dual dynamic consis- tency regularization for semi-supervised domain adaptation. IEEE Access, 2024. 2

    2024

  34. [42]

    Enhancing vietnamese vqa through curriculum learn- ing on raw and augmented text representations

    Khoi Anh Nguyen, Linh Yen Vu, Thang Dinh Duong, Thuan Nguyen Duong, Huy Thanh Nguyen, and Vinh Quang Dinh. Enhancing vietnamese vqa through curriculum learn- ing on raw and augmented text representations. arXiv preprint arXiv:2503.03285, 2025. 1

    2025 arXiv

  35. [43]

    Kosmos-2: Ground- ing multimodal large language models to the world

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Ground- ing multimodal large language models to the world. arXiv preprint arXiv:2306.14824, 2023. 2

    2023 arXiv

  36. [44]

    Going full-tilt boogie on document understanding with text-image-layout transformer

    Rafał Powalski, Łukasz Borchmann, Dawid Jurkiewicz, Tomasz Dwojak, Michał Pietruszka, and Gabriela Pałka. Going full-tilt boogie on document understanding with text-image-layout transformer. In Document Analysis and Recognition–ICDAR 2021: 16th International Conference, Lausan...

    2021

  37. [45]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8309–8318, 2019. 4, 5

    2019

  38. [46]

    Gemini 2.5: Pushing the frontier with ad- vanced reasoning, multimodality, long context, and next gen- eration agentic capabilities

    Gemini Team. Gemini 2.5: Pushing the frontier with ad- vanced reasoning, multimodality, long context, and next gen- eration agentic capabilities. Technical report, Google, 2025. Technical Report. 2, 7

    2025

  39. [47]

    Igl-dt: Itera- tive global-local feature learning with dual-teacher semantic segmentation framework under limited annotation scheme

    Quan Tran, Hoang-Thien Nguyen, Thanh-Huy Nguyen, Gia- Van To, Tien-Huy Nguyen, and Quan Nguyen. Igl-dt: Itera- tive global-local feature learning with dual-teacher semantic segmentation framework under limited annotation scheme. In Proceedings of the Computer Vision and Patter...

    2025

  40. [48]

    Mlg2net: Molecular global graph network for drug response prediction in lung cancer cell lines

    Thi-Oanh Tran, Thanh-Huy Nguyen, Tuan Tung Nguyen, and Nguyen Quoc Khanh Le. Mlg2net: Molecular global graph network for drug response prediction in lung cancer cell lines. Journal of Medical Systems, 49(1):1–7, 2025. 2

    2025

  41. [49]

    Describe anything in medical images

    Xi Xiao, Yunbei Zhang, Thanh-Huy Nguyen, Ba-Thinh Lam, Janet Wang, Lin Zhao, Jihun Hamm, Tianyang Wang, Xingjian Li, Xiao Wang, et al. Describe anything in medical images. arXiv preprint arXiv:2505.05804, 2025. 2

    2025 arXiv

  42. [50]

    Layoutlm: Pre-training of text and layout for document image understanding

    Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. Layoutlm: Pre-training of text and layout for document image understanding. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, page 1192–1200, New York, NY...

    2020

  43. [51]

    LayoutLMv2: Multi-modal pre-training for visually-rich document under- standing

    Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, Min Zhang, and Lidong Zhou. LayoutLMv2: Multi-modal pre-training for visually-rich document under- standing. In Proceedings of the 59th Annual Meeting of the A...

  44. [52]

    Qwen3 technical report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. 6

    2025 arXiv

  45. [53]

    Minicpm-v: A gpt-4v level mllm on your phone

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800, 2024. 7

    2024 arXiv

  46. [54]

    Ferret: Refer and ground anything anywhere at any granularity

    Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. arXiv preprint arXiv:2310.07704, 2023. 2, 7

    2023 arXiv

  47. [55]

    Mm-vet: Evaluating large multimodal models for integrated capabilities

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490, 2023. 6

    2023 arXiv

  48. [56]

    Osprey: Pixel un- derstanding with visual instruction tuning

    Yuqian Yuan, Wentong Li, Jian Liu, Dongqi Tang, Xinjie Luo, Chi Qin, Lei Zhang, and Jianke Zhu. Osprey: Pixel un- derstanding with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 28202–28211, 2024. 2, 7

    2024

  49. [57]

    Videollama 3: Frontier multi- modal foundation models for image and video understand- ing

    Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, et al. Videollama 3: Frontier multi- modal foundation models for image and video understand- ing. arXiv preprint arXiv:2501.13106, 2025. 7

    2025 arXiv

  50. [58]

    Gpt4roi: Instruction tuning large language model on region- of-interest

    Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Yu Liu, Kai Chen, and Ping Luo. Gpt4roi: Instruction tuning large language model on region- of-interest. In European Conference on Computer Vision , pages 52–70. Springer, 2025. 2, 7

    2025

  51. [59]

    Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in Neural Information Processing Systems, 36:46595–46623, 2023

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in Neural Information Processing Systems, 36:46595–46623, 2023. 6

    2023

  52. [60]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shen- glong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479, 2025. 2, 7 11

    2025 arXiv

  53. [2021]

    Association for Computational Linguistics. 1

  54. [2022]

    Curran Associates Inc. 2

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.