Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

FiVL: A Framework for Improved Vision-Language Alignment through the Lens of Training, Evaluation and Explainability

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Adding a vision-modeling loss to LLaVA-1.5 training improves most benchmarks and gives the model a built-in segmentation capability.

desk verdict Useful dataset-and-evaluation framework with a novel VM loss, but the main training claim is confounded by the lack of a same-recipe control and an SGD/AdamW discrepancy. read the letter →

arxiv 2412.14672 v2 pith:QJ27CRKD submitted 2024-12-19 cs.CV cs.AI

Estelle Aflalo , Gabriela Ben Melech Stan , Tiep Le , Man Luo , Shachar Rosenman , Sayak Paul , Shao-Yen Tseng , Vasudev Lal This is my paper · ORCID
classification cs.CVcs.AI
keywords vision-languagealignmentvisualgroundingsegmentationmaskshallucinationreliancescoreinstructiontuningvision-modelinglossattentionexplainability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces FiVL, a pipeline that turns ordinary image-text instruction data into visually grounded data by extracting key expressions from question-answer text, localizing those expressions with segmentation masks, and then training the model so that every image patch inside a mask predicts the word naming it. The central claim is that this vision-modeling loss, added to the usual language-modeling loss, makes a fine-tuned LLaVA-1.5-7b outperform its baseline on a majority of ten downstream benchmarks while also giving the model a built-in ability to segment images. The same masks power a Visual Reliance Score that measures how much a model depends on the image by comparing accuracy on original versus masked images, and the masks let the authors rank attention heads by how strongly they align vision and language. If the claim is right, hallucinations in large vision-language models can be reduced simply by teaching image patches their own names, and evaluation can directly measure whether a model is actually looking at the image.

What carries the argument

The load-bearing object is the vision-modeling (VM) loss in equation (1), $L = \lambda \cdot CE_{VM} + (1-\lambda) \cdot CE_{LM}$ with $\lambda = 0.1$, applied during LLaVA's fine-tuning stage. For each image patch inside a selected segmentation mask, the visual logit—the last linear layer's output over the text vocabulary—is trained to predict the noun key expression assigned to that patch; patches outside masks are ignored, and the LM loss still trains the textual logits. This turns image tokens into vocabulary predictions, explicitly teaching the model what each image region is called, while the same segmentation masks serve evaluation (masked-image accuracy drop equals the Visual Reliance Score) and explainability (Spearman correlation between mask and vision-to-language attention identifies heads (10,6) and (14,11)).

What would settle it

Train the identical LLaVA-1.5-7b setup with random same-size masks in place of the genuine key-expression masks; if the reported benchmark gains persist, then the content of the masks is not what drives the improvement. Alternatively, train only on the 58% of masks that human annotators judged relevant and check whether gains grow, shrink, or vanish.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that pixel-level grounding can be generated automatically and then injected into an LVLM through a simple auxiliary loss. Specifically, GPT-4o reads only the question-answer text and names the key expressions, GroundedSAM turns those expressions into segmentation masks, and each image patch inside a mask is trained to predict the corresponding key expression's text token from the model's vocabulary. The resulting model outperforms LLaVA-1.5-7b on a majority of the ten benchmarks tested, produces argmax token maps that serve as coarse segmentation maps (about 10 tokens per image versus about 73 for the baseline, with roughly three times higher IoU against GroundedSAM), and enables a Visual Reliance Score that compares models and benchmarks by the accuracy drop when key regions are masked.

Load-bearing premise

The load-bearing premise is that the automatically generated masks, derived from phrases extracted from text alone, are accurate enough to serve as training targets and evaluation perturbations, even though human judges found only 58% of masks relevant to their phrase.

Editorial extensions

If this is right

  • The FiVL-trained model outperforms LLaVA-1.5-7b on most of the ten benchmarks reported: OK-VQA, MME, POPE, ScienceQA, MMBench, LLaVA-Bench-COCO, LLaVA-in-the-wild, Text-VQA, VizWiz, and GQA.
  • The vision logits can double as segmentation maps: FiVL produces about 10 max-v tokens per image versus about 73 for the baseline, and IoU against GroundedSAM roughly triples at every threshold tested.
  • The Visual Reliance Score distinguishes benchmarks: FiVL-VQAv2 demands the most image reliance, and the average VRS across benchmarks tracks overall model ranking on a broad leaderboard.
  • The Spearman-correlation head summary isolates attention heads (10,6) and (14,11) as the strongest vision-language aligners, giving a concrete target for studying visual hallucinations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same VM-loss recipe should transfer to other LVLMs whose last-layer logits are accessible, but the paper only demonstrates it on LLaVA-1.5-7b, so that transfer is an extrapolation.
  • Because only 58% of masks passed human relevance, the training signal is partly noisy; a mask-quality filter or a relevance-weighted loss could plausibly raise the reported gains further, and the paper's own size-based selection is an early version of that idea.
  • Visual Reliance Score could be repurposed as a data-quality filter: samples whose key-expression masking barely changes accuracy are likely answerable from language priors, so they could be downweighted or removed from instruction data.
  • The cross-model correlation between VRS and leaderboard rank suggests that visual reliance is a proxy for general model capability, but verifying that requires comparing models with identical architecture and different training data.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes FiVL, a framework that uses GPT-4o to extract key expressions from question-answer pairs and GroundedSAM to produce segmentation masks, yielding FiVL-Instruct (an augmentation of LLaVA-1.5-mix-665K) and FiVL-POPE, FiVL-VQAv2, and FiVL-GQA evaluation sets. The authors fine-tune LLaVA-1.5-7b with a combined language-modeling and vision-modeling loss (Eq. 1) and report performance improvements over a baseline on several benchmarks. They also introduce a Visual Reliance Score (Eq. 2) based on masking key expressions, and they use Spearman correlations between segmentation masks and attention to identify vision-language aligned attention heads. The paper further claims that the trained model can produce approximate segmentation maps from vision logits.

Significance. If the training improvement were established, the FiVL framework would be a useful, low-cost way to create grounded instruction data for LVLMs and to evaluate how much a model relies on visual evidence. The paper includes human and automated quality assessments of the generated data, an open-source release, a simple perturbation-based evaluation metric applicable to any dataset, and a complementary explainability analysis. These are valuable additions to the literature. However, the central training claim is not yet supported because the experiments compare against a released checkpoint rather than a same-recipe control model; several smaller methodological gaps further weaken the interpretation.

major comments (4)
  1. [Section 5.1 / Figure 3 / Table 6 / Appendix F] The central claim that the Vision Modeling loss improves performance over the baseline is not supported by the current experimental design. The model is compared against the released LLaVA-1.5-7B checkpoint, not against a model fine-tuned on the original un-augmented LLaVA-1.5-mix-665K under identical conditions. This matters because Table 6 lists the optimizer as SGD while the standard LLaVA-1.5 recipe uses AdamW; without a lambda=0 control trained with the same data order, epochs, learning rate, and optimizer, the gains in Figure 3 could be due to recipe differences rather than to the proposed loss. I request a same-recipe control on the original data, ideally with multiple seeds and error bars, before the improvement claim can be accepted.
  2. [Section 4.1 / Section 4.2 / Section 5.1] The training labels for the vision-modeling loss come from segmentation masks whose quality is low: human evaluation found only 58% of masks relevant to their key expression, and the Seg1 judge found 46%. The paper states that masks are filtered by size (Figure 6) and that only nouns are used, but it never specifies the size threshold, the fraction of data removed, or the annotation/judge agreement after filtering. Since every image patch in a retained mask is assigned a vocabulary token as its visual label in Eq. (1), systematic mask errors become training-label noise. Please report the filtering criterion, the post-filter mask relevance, and ideally perform a sensitivity analysis of the training result to mask quality.
  3. [Section 5.2 / Table 3] The claim that Visual Reliance Score correlates with overall model performance is not quantified. Table 3 reports VRS for seven models, and the text compares leaderboard ranks qualitatively, but no correlation coefficient, confidence interval, or statistical test is provided, and the external leaderboard source is not described in enough detail to reproduce the rank mapping. Please provide a quantified correlation with uncertainty, or soften the claim to a qualitative observation.
  4. [Appendix G / Table 7] The segmentation-capability evaluation is partly circular: Grounded-SAM is used as ground truth for IoU, but Grounded-SAM is also the system that produced the training masks. A model trained to mimic Grounded-SAM masks will score higher by construction. In addition, the baseline and FiVL model produce very different numbers of max-v tokens, which makes the per-token IoU comparison difficult to interpret. Please evaluate with human-annotated masks or an independent segmenter, and align the comparison across a common set of tokens or use a detection-style metric.
minor comments (5)
  1. [Section 3.3 / Table 3] The text says 'we evaluated five models' but Table 3 lists seven models (Qwen2-VL, LLaVA-7B, LLaVA-13B, GPT4o, BLIP-2, Pixtral, Phi3). Please correct the wording.
  2. [Eq. (2) / Table 2] Random perturbation sometimes yields negative Visual Reliance Scores (e.g., LLaVA-13B on VQA-v2: -0.05), which means the model accuracy improves after perturbation rather than drops. Please clarify the interpretation of negative values or use a signed metric.
  3. [Table 6] The text says 'We maintained the original training setup' but Table 6 reports Optimizer SGD, while the standard LLaVA-1.5 fine-tuning setup uses AdamW. Please clarify whether the released baseline was retrained with SGD or whether this is a deliberate deviation, and discuss the impact on the comparison.
  4. [Throughout] There are several typos and small errors, including 'Additionnaly', 'lenght', 'refer as to the baseline', and the reference to 'Appendix 13' in Section 3.1, which should be 'Figure 13'.
  5. [Appendix A / Table 5] The format of Table 5 is not aligned: the first row lists three numbers without column headers, making it hard to read. Please align the table or add explicit column labels.

Circularity Check

1 steps flagged · score 6.0 of 10

The downstream-benchmark training claim is external and not circular, but the Appendix G evaluation of the model's 'inherent' segmentation reuses the same GroundedSAM masks that generated the VM training labels as ground truth, making that capability claim circular by construction.

  1. self definitional [Appendix G (IoU evaluation, Table 7); Section 5.1 (Vision Modeling loss, Eq. 1); Section 3.1 (GroundedSAM pipeline)]
    "as ground truth to compare against, we employed Grounded-SAM to produce segmentation maps given each max-v token. Grounded-SAM was implemented using the IDEA-Research/grounding-Dino-Tiny model with thresholds set at 0.2, 0.4, and 0.6, followed by facebook/sam-vit-huge with a threshold of 0.0. The Intersection over Union (IoU) score was computed between the FiVL-generated segmentation masks and the corresponding Grounded-SAM masks to quantitatively assess alignment."

    The FiVL-Instruct training labels are produced by the GroundedSAM pipeline (Section 3.1), and the Vision Modeling loss in Eq. (1) trains each image-patch vision logit inside a GroundedSAM mask to predict the key-expression noun. Appendix G then evaluates the model's 'inherent' segmentation by asking Grounded-SAM to segment the model's max-v token and computing IoU against those Grounded-SAM masks. The evaluation ground truth is therefore the same generator that produced the training targets. The reported 3x IoU improvement and 7x reduction in tokens per sample are direct consequences of training the vision logits to reproduce GroundedSAM-derived annotations, not evidence of independent segmentation ability.

full rationale

The paper's central benchmark claim is an external comparison: a LLaVA-1.5-7b model fine-tuned on FiVL-Instruct with the VM loss is evaluated against a baseline on OK-VQA, MME, POPE, ScienceQA, MMBench, and other public benchmarks, so that result is not a fitted derivation and is not circular. The Visual Reliance Score evaluation applies FiVL masks to models not trained on FiVL and includes a random-mask control, so it is also not circular by construction. The one clear circular step is the Appendix G segmentation evaluation: the VM loss (Eq. 1) supervises vision logits with GroundedSAM masks, and the same GroundedSAM pipeline is then reused as the IoU ground truth for the model's predicted masks. This makes the 'inherent segmentation' capability a self-referential measure, though it is a secondary contribution rather than the main benchmark-improvement claim. The missing no-VM control and the use of SGD in Table 6 are experimental-design confounds that weaken attribution of the benchmark gains, but they are correctness risks, not circularity, and are not scored as circular steps.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claims rest on external tools (GPT-4o, GroundedSAM, LLaVA-1.5-mix-665K) whose outputs are treated as ground truth. No new physical or conceptual entities are introduced; the Visual Reliance Score is a derived metric, not an entity. The free parameters are the fitted VM-loss weight lambda and an unspecified mask-size filtering threshold used during training-data construction.

free parameters (2)
  • lambda (vision-modeling loss weight) = 0.1
    Controls the trade-off between CE_VM and CE_LM in Eq. 1; chosen via ablation on downstream benchmarks (Appendix F, Figure 11b), so benchmark gains are tuned, not free of fitted constants.
  • segmentation mask size threshold = not reported
    In Section 4.1 masks are selected by size because smaller masks were judged more relevant (Figure 6), but the exact threshold is not given, making the filtering step under-specified.
assumptions (4)
  • domain assumption Key expressions extracted by GPT-4o from text-only question-answer pairs correspond to content that actually requires the image and is present in it.
    This premise underlies all FiVL datasets; human evaluation found 75% of key expressions pertinent and GPT-4o importance ratio 76%, so the premise is approximately but not perfectly satisfied (Sections 3.1, 4.1, 4.2.1).
  • domain assumption GroundedSAM segmentation masks correctly localize each key expression in the image.
    All training targets and VRS perturbations are masks from this pipeline; human annotators rated only 58% of masks relevant and GPT-4o Seg1 only 46%, so this premise is partially violated and the paper relies on size-based filtering to compensate (Sections 3.1, 4.1, 4.2.2).
  • domain assumption A model fine-tuned on LLaVA-1.5-mix-665K with the new VM loss and augmented data can be compared against the original released LLaVA-1.5-7b to attribute improvement to FiVL.
    No control that fine-tunes on un-augmented data with the same pipeline is reported, so the comparison conflates the data augmentation, the new loss, and retuning (Section 5.1).
  • domain assumption Masking the segmented key-expression regions measures visual reliance rather than generic occlusion or artifact.
    Partially supported by the random-mask control in Table 2, which shows small drops for random masks, but the masks may still remove more informative pixels and the metric assumes the key-expression set is sufficient to answer correctly (Section 5.2, Eq. 2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of FiVL: A Framework for Improved Vision-Language Alignment through the Lens of Training, Evaluation and Explainability." pith.science (2026). https://pith.science/paper/QJ27CRKD

@misc{pith2026241214672,
  author       = {Pith},
  title        = {Pith review of: FiVL: A Framework for Improved Vision-Language Alignment through the Lens of Training, Evaluation and Explainability},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QJ27CRKD}},
  note         = {Machine review of arXiv:2412.14672}
}
read the original abstract

Large Vision Language Models (LVLMs) have achieved significant progress in integrating visual and textual inputs for multimodal reasoning. However, a recurring challenge is ensuring these models utilize visual information as effectively as linguistic content when both modalities are necessary to formulate an accurate answer. We hypothesize that hallucinations arise due to the lack of effective visual grounding in current LVLMs. Furthermore, current vision-language benchmarks are not specifically measuring the degree to which the answer require the visual input. This limitation makes it challenging to confirm that the image is truly necessary, particularly in tasks like visual question answering. In this work, we introduce FiVL, a novel method for constructing datasets designed to train LVLMs for enhanced visual grounding and also evaluate their effectiveness in achieving it. We demonstrate the value of our datasets through three approaches. First, we introduce a novel training task based on our augmented training dataset, resulting in better performance than the baseline. Second, we present benchmarks to assess the model's ability to use image as substantive evidence, rather than relying solely on linguistic priors. Finally, we identify attention heads with the strongest vision-language alignment, enabling explainability on visual-driven hallucinations. The code is available at https://github.com/IntelLabs/fivl.

Figures

Figures reproduced from arXiv: 2412.14672 by the authors.

Figure 1
Figure 1. Dataset Collection Overview. First, GPT4-o processes the question and answer to produce "key expres [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Overview of Vision Modeling pretraining task. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Our model trained on FiVL-Instruct evaluated [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Predicted token from vision logits ( "Flo", for [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Attention heatmaps overlaid on the original [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Impact of the size of the segmentation mask. [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Segmentation Verification Prompt for GPT-4o. [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 9
Figure 9. Figure 9: Head summary for VL alignment via Spear [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Attention heatmaps overlaid on the original [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]
Figure 11
Figure 11. Figure 11: Ablations for the training method [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]
Figure 12
Figure 12. Figure 12: Segmentations produced inherently by our model. Each figure corresponds to the max-v token specified [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]
Figure 13
Figure 13. Figure 13: FiVL-Instruct system prompt used for train [PITH_FULL_IMAGE:figures/full_fig_p017_13.png]
Figure 16
Figure 16. Figure 16: Examples for GQA and POPE prompts [PITH_FULL_IMAGE:figures/full_fig_p018_16.png]
Figure 17
Figure 17. Figure 17: Web user interface for our dataset evaluation [PITH_FULL_IMAGE:figures/full_fig_p019_17.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. From Visuals to Vocabulary: Establishing Equivalence Between Image and Text Token Through Autoregressive Pre-training in MLLMs

    cs.CV 2025-02 conditional novelty 4.0 of 10

    Adding an L2 loss that pushes the language model's image hidden states back toward the input image embeddings improves LLaVA-style models on several VQA benchmarks, with some benchmarks unaffected or slightly worse.

Reference graph

Works this paper leans on

46 extracted references · 11 canonical work pages · cited by 1 Pith paper

  1. [1]

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Weizhu Chen, Yen-Chun Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, Matt...

    arXiv 2024

  2. [2]

    Estelle Aflalo, Meng Du, Shao-Yen Tseng, Yongfei Liu, Chenfei Wu, Nan Duan, and Vasudev Lal. 2022. https://arxiv.org/abs/2203.17247 Vl-interpret: An interactive visualization tool for interpreting vision-language transformers . Preprint, arXiv:2203.17247

    arXiv 2022

  3. [3]

    Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. 2018. https://arxiv.org/abs/1712.00377 Don't just assume; look and answer: Overcoming priors for visual question answering . Preprint, arXiv:1712.00377

    arXiv 2018

  4. [4]

    Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, Soham Ghosh, Amélie Héliou, Paul Jacob, Albert Q. Jiang, Kartik Khandelwal, Timothée Lacroix, Guillaume Lample, Diego Las Casas, Thibaut Lavril, Teven Le Scao, Andy Lo, William Marshall,...

    arXiv 2024

  5. [5]

    Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. 2023. Shikra: Unleashing multimodal llm's referential dialogue magic. arXiv preprint arXiv:2306.15195

    arXiv 2023

  6. [6]

    Long Chen, Xin Yan, Jun Xiao, Hanwang Zhang, Shiliang Pu, and Yueting Zhuang. 2020. https://arxiv.org/abs/2003.06576 Counterfactual samples synthesizing for robust visual question answering . Preprint, arXiv:2003.06576

    work page Pith review arXiv 2020

  7. [7]

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. 2024. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821

    arXiv 2024

  8. [8]

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023. https://openreview.net/forum?id=vvoWPYqZJA Instruct BLIP : Towards general-purpose vision-language models with instruction tuning . In Thirty-seventh Conference on Neural Information Processing Systems

    2023

Show all 46 references
  1. [9]

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. 2024. https://arxiv.org/abs/2306.13394 Mme: A comprehensive evaluation benchmark for multimodal large language models . Preprint, ...

    2024 arXiv

  2. [10]

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. https://arxiv.org/abs/1612.00837 Making the v in vqa matter: Elevating the role of image understanding in visual question answering . Preprint, arXiv:1612.00837

    2017 arXiv

  3. [11]

    Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P

    Danna Gurari, Qing Li, Abigale J. Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P. Bigham. 2018. https://arxiv.org/abs/1802.08218 Vizwiz grand challenge: Answering visual questions from blind people . Preprint, arXiv:1802.08218

    2018 arXiv

  4. [12]

    Hudson and Christopher D

    Drew A. Hudson and Christopher D. Manning. 2019. https://arxiv.org/abs/1902.09506 Gqa: A new dataset for real-world visual reasoning and compositional question answering . Preprint, arXiv:1902.09506

    2019 arXiv

  5. [13]

    Jimenez, Olga Russakovsky, and Karthik Narasimhan

    Carlos E. Jimenez, Olga Russakovsky, and Karthik Narasimhan. 2022. https://arxiv.org/abs/2203.07613 Carets: A consistency and robustness evaluative test suite for vqa . Preprint, arXiv:2203.07613

    2022 arXiv

  6. [14]

    Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. 2017. CLEVR : A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

    2017

  7. [15]

    Berg, Wan-Yen Lo, Piotr Doll \'a r, and Ross Girshick

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Doll \'a r, and Ross Girshick. 2023. Segment anything. arXiv:2304.02643

    2023 arXiv

  8. [16]

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of com...

    2017

  9. [17]

    Baiqi Li, Zhiqiu Lin, Wenxuan Peng, Jean de Dieu Nyandwi, Daniel Jiang, Zixian Ma, Simran Khanuja, Ranjay Krishna, Graham Neubig, and Deva Ramanan. 2024. https://arxiv.org/abs/2410.14669 Naturalbench: Evaluating vision-language models on natural adversarial samples . Preprint,...

    2024 arXiv

  10. [18]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023 a . https://arxiv.org/abs/2301.12597 Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models . Preprint, arXiv:2301.12597

    2023 arXiv

  11. [19]

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. 2023 b . https://arxiv.org/abs/2305.10355 Evaluating object hallucination in large vision-language models . Preprint, arXiv:2305.10355

    2023 arXiv

  12. [20]

    Lawrence Zitnick

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll \'a r, and C. Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer Vision -- ECCV 2014, pages 740--755, Cham. Springer International Publishing

    2014

  13. [21]

    Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. 2024 a . A survey on hallucination in large vision-language models. arXiv preprint arXiv:2402.00253

    2024 arXiv

  14. [22]

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024 b . https://arxiv.org/abs/2310.03744 Improved baselines with visual instruction tuning . Preprint, arXiv:2310.03744

    2024 arXiv

  15. [23]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. https://arxiv.org/abs/2304.08485 Visual instruction tuning . Preprint, arXiv:2304.08485

    2023 arXiv

  16. [24]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024 c . Visual instruction tuning. Advances in neural information processing systems, 36

    2024

  17. [25]

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang. 2024 d . https://arxiv.org/abs/2303.05499 Grounding dino: Marrying dino with grounded pre-training for open-set object detection . Prep...

    2024 arXiv

  18. [26]

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. 2024 e . https://arxiv.org/abs/2307.06281 Mmbench: Is your multi-modal model an all-around player? Preprint, arXiv:2307.06281

    2024 arXiv

  19. [27]

    Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. https://arxiv.org/abs/2209.09513 Learn to explain: Multimodal reasoning via thought chains for science question answering . Preprint, arXiv:2209.09513

    2022 arXiv

  20. [28]

    Chuofan Ma, Yi Jiang, Jiannan Wu, Zehuan Yuan, and Xiaojuan Qi. 2024. Groma: Localized visual tokenization for grounding multimodal large language models. In European Conference on Computer Vision, pages 417--435. Springer

    2024

  21. [29]

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019. https://arxiv.org/abs/1906.00067 Ok-vqa: A visual question answering benchmark requiring external knowledge . Preprint, arXiv:1906.00067

    2019 arXiv

  22. [30]

    OpenAI. 2024. https://arxiv.org/abs/2410.21276 Gpt-4o system card . Preprint, arXiv:2410.21276

    2024 arXiv

  23. [31]

    OpenCompass . 2025. Open vlm leaderboard. https://huggingface.co/spaces/opencompass/open_vlm_leaderboard. Accessed: 12/02/2025

    2025

  24. [32]

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. 2023. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306.14824

    2023 arXiv

  25. [33]

    Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S

    Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M. Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S. Khan. 2024. GLaMM : Pixel grounding large multimodal model. In Proceedings of the IEEE/CVF Conference on Computer Vision and P...

    2024

  26. [34]

    Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, Zhaoyang Zeng, Hao Zhang, Feng Li, Jie Yang, Hongyang Li, Qing Jiang, and Lei Zhang. 2024. https://arxiv.org/abs/2401.14159 Grounded sam: Assembling open-world ...

    2024 arXiv

  27. [35]

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019. https://arxiv.org/abs/1904.08920 Towards vqa models that can read . Preprint, arXiv:1904.08920

    2019 arXiv

  28. [36]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024. https://arxiv.org/abs/2409.12191 Qwen2-v...

    2024 arXiv

  29. [37]

    Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. 2024. https://openreview.net/forum?id=2msbbX3ydD Ferret: Refer and ground anything anywhere at any granularity . In The Twelfth International Conference on...

    2024

  30. [38]

    Hao Zhang, Hongyang Li, Feng Li, Tianhe Ren, Xueyan Zou, Shilong Liu, Shijia Huang, Jianfeng Gao, Chunyuan Li, Jainwei Yang, et al. 2024 a . Llava- G rounding: Grounded visual chat with large multimodal models. In European Conference on Computer Vision, pages 19--35. Springer

    2024

  31. [39]

    Yichi Zhang, Ziqiao Ma, Xiaofeng Gao, Suhaila Shakiah, Qiaozi Gao, and Joyce Chai. 2024 b . Groundhog: Grounding large language models to holistic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14227--14238

    2024

  32. [40]

    Yang Zhao, Zhijie Lin, Daquan Zhou, Zilong Huang, Jiashi Feng, and Bingyi Kang. 2023. Bubogpt: Enabling visual grounding in multi-modal llms. arXiv preprint arXiv:2307.08581

    2023 arXiv

  33. [41]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023 a . Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595--46623

    2023

  34. [42]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023 b . https://arxiv.org/abs/2306.05685 Judging llm-as-a-judge with mt-bench and chatbot arena ...

    2023 arXiv

  35. [43]

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2024. https://openreview.net/forum?id=1tZbq88f27 Mini GPT -4: Enhancing vision-language understanding with advanced large language models . In The Twelfth International Conference on Learning Representations

    2024

  36. [44]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  37. [45]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  38. [46]

    write newline

    " write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence '...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.