Analyzing the Sensitivity of Vision Language Models in Visual Question Answering

Deepak Venugopal; Monika Shah; Sanorita Dey; Somdeb Sarkhel; Sudarshan Balaji

REVIEW 5 major objections 5 minor 26 references

Analyzing the Sensitivity of Vision Language Models in Visual Question Answering

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Adding answer-preserving modifiers to visual questions consistently reduces accuracy in three current vision-language models.

desk verdict Small, honest empirical study: adding answer-preserving modifiers to VQA questions reliably drops a few points of accuracy on three closed VLMs; the central claim is plausible but the AMT verification and GPT-4o self-judging need scrutiny. read the letter →

arxiv 2507.21335 v1 pith:NCW23S7J submitted 2025-07-28 cs.CV

Monika Shah , Sudarshan Balaji , Somdeb Sarkhel , Sanorita Dey , Deepak Venugopal This is my paper · ORCID

classification cs.CV

keywords visualquestionansweringvisionlanguagemodelsGrice'smaximsmodifiersVQAv2.0GPT-4oGemini-1.5-FlashClaude-3.5-Sonnet

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper treats visual question answering as a one-turn conversation and asks whether vision-language models cope with the same conversational friction humans handle easily. The authors add visual and relational modifiers to human-crafted questions from VQA v2.0, instruct the generating model to keep the original answer unchanged, and keep only questions that crowdworkers verify as answer-preserving. Across 1,000 test questions, GPT-4o, Gemini-1.5-Flash, and Claude-3.5-Sonnet answer modified questions less accurately in almost every model/modifier combination. The authors read this consistent drop as evidence that the models are sensitive to violations of the conversational maxims, while their limitations section notes that open-ended scoring is approximate and self-modified questions may carry prompt-history bias.

What carries the argument

The load-bearing machinery is Grice's cooperative principle and its four maxims (quality, quantity, relation, manner), used to define why answer-preserving modifiers should be harmless to a robust listener. Operationally, the study rests on a three-step pipeline: a vision-language model generates modified questions; crowdworkers verify that the modification preserves the original answer; and paired significance tests—McNemar's exact test for dichotomous yes/no and numeric outcomes, Wilcoxon signed-rank for open-ended outcomes—compare each model's accuracy before and after modification. The verification step picks out the subset of modified questions for which the only plausible cause of an accuracy drop is the model's handling of the modifier itself.

What would settle it

Run a second human panel on the same verified answer-preserving modified questions and compare human accuracy on original versus modified questions; if the human drop is as large as the VLM drop, the effect is linguistic ambiguity rather than a VLM-specific limitation. Alternatively, a large valid sample of one modifier type that produces no accuracy drop in any of the three models would falsify the paper's claim of consistent degradation.

Watch

Extended reading notes

Core claim

The central claim is stated in the abstract and in Section 3.2.1: there is a consistent performance degradation in the presence of modifiers. When an answer-preserving visual or relational modifier is added to a VQA v2.0 question, the three tested vision-language models give worse answers than they give to the original question, for both yes/no and numeric questions evaluated by exact match and open-ended questions scored by an LLM similarity evaluator. Modifiers generated by Gemini-1.5-Flash produce the largest average drops for all three models, while modifiers from Claude-3.5-Sonnet produce the smallest; GPT-4o's yes/no accuracy does not change significantly on questions it modified itself. The paper interprets the pattern as evidence that vision-language models do not share humans' robustness to extra conversational detail, while stopping short of identifying why the drop occurs.

Load-bearing premise

Everything rests on the crowdworker verification that a modified question's correct answer really is unchanged; if that verification is wrong or ambiguous, the reported accuracy drop partly reflects changed ground truth rather than vision-language model sensitivity.

Editorial extensions

If this is right

If the central claim holds, standard VQA accuracy numbers likely overstate how well vision-language models handle verbose, elaborately phrased questions.
Modifier generation with human verification is a reusable stress test for comparing model robustness across models and question types.
Because GPT-4o is stable on its own modifications but degrades on other models' modifications, any single-model evaluation of self-generated perturbations may understate sensitivity.
The remaining open question—which linguistic properties of modifiers drive the drop—can be addressed by grouping modifiers by attribute type, syntactic position, and length.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A cleaner replication would score open-ended answers with a model other than GPT-4o, since GPT-4o is also one of the evaluated systems and the paper's limitations flag this area as approximate.
One testable extension is to compare human accuracy on the same verified modified questions; if humans also lose ground, part of the observed drop is ordinary ambiguity rather than VLM-specific brittleness.
Another extension is to vary only the syntactic placement or length of the modifier while holding content fixed, which would separate pattern-matching effects from reasoning effects.
The protocol could be applied to other benchmarks by appending verifiable answer-preserving modifiers, creating a robustness split that standard accuracy does not expose.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a modest but genuine empirical contribution. It takes the modifier-addition idea from Britton et al. (2022), applies it to three modern closed VLMs, runs an AMT human verification step, and frames it with Gricean pragmatics. The measurements are real, the cross-model generation matrix is a nice touch, and the paper is transparent about its limitations. If you care about VLM robustness or VLM-as-judge pipelines, this is worth a look.

The exact-match half (yes/no and numeric questions) is the solid core. The paper reports percentage change in accuracy across nine model/modifier combinations and pairs it with McNemar's test. The degradation is consistent: almost all cells are positive, and Gemini-1.5-Flash's modifiers are consistently hardest. That pattern is unlikely to be pure noise, and the authors do not oversell it.

Where the soft spots are, in order of severity:

1. The AMT verification protocol is weak. Workers are shown the original question, its answer, and the modified question, and asked 'Is the same answer correct for Q2?' That is a leading question. They are not asked to answer the modified question from scratch, and the paper never states what fraction of the three workers must agree. A worker who sees 'Yes' for Q1 is likely to say 'Yes' for Q2 even if the modifier introduces a false visual fact. If that happens, models that correctly answer the modified question (under the changed ground truth) get scored as wrong, inflating the reported %change. This is not a fatal flaw, but it means the size of the drop is uncertain, and the 'consistent degradation' could be partly a label-change artifact.

2. The open-ended half uses GPT-4o to score similarity, and GPT-4o is one of the evaluated models. The authors acknowledge this in Limitation 3, but it is still a circularity that limits the strength of Table 2. The numbers there are suggestive, not evidence.

3. Reporting gaps: the paper gives %change but not baseline accuracies, per-cell p-values, confidence intervals, or the raw modified-question set. Without that, independent verification is impossible. This is a clear request for revision, not a reason to reject.

The stress-test note about AMT anchoring is on target; I would not dismiss it. But I also would not call it 'fatal'. The central claim—that modifiers degrade VLM performance—probably holds in the yes/no and numeric half even with a stricter verification protocol; the magnitude might shrink. The Grice framing is light but not vacuous, and the paper's own limitations show honest thinking.

Who is this for? Researchers working on VQA robustness, multimodal evaluation, or VLM-as-judge validation. It does not resolve the field, but it is a useful data point. I would engage with it and would not desk reject it. If I were the editor, I would send it to a serious referee with instructions to ask for data release, a stricter AMT protocol (e.g., workers answer from scratch, majority agreement stated), and baseline accuracies.

Recommendation: accept for peer review; expect major revision.

Referee Report

5 major / 5 minor

Summary. The paper investigates the sensitivity of three vision-language models (GPT-4o, Gemini-1.5-Flash, and Claude-3.5-Sonnet) to pragmatically marked questions in VQA. The authors add visual or relational modifiers to human-written questions from the VQA v2.0 test set, ask AMT workers to verify that the modified questions preserve the original answer, and then compare each VLM's accuracy on original versus modified questions. They report percentage changes in accuracy for 500 yes/no and numeric questions and 500 open-ended questions, and claim a consistent performance degradation when modifiers are added, with Gemini-generated modifiers being the hardest and Claude-generated modifiers the easiest for all tested models.

Significance. If the claimed effect is robust, this is a useful behavioral probe for VLM robustness, connecting Gricean pragmatics to multimodal question answering and providing a reproducible protocol for generating answer-preserving perturbations. The paper's strengths include the use of a standard benchmark with exact ground truth for closed-form questions, a three-model generation-and-evaluation design, an AMT verification step, and paired significance tests. However, the central claim is not yet established because key verification details and baseline statistics are omitted, and the open-ended evaluation uses one of the tested models as the judge. The idea is testable and the main concerns are addressable with additional analysis, but the current reporting gaps prevent a firm conclusion.

major comments (5)

[Section 3.2 and Appendix B] The answer-preservation filter is the load-bearing step for the paper's central claim, but its implementation is under-specified. The HIT shows workers the original question, the original answer, and the modified question, and asks 'Is the same answer correct for Q2?'; this anchoring can lead workers to confirm the original answer even when the modifier introduces a visually false or ambiguous premise (e.g., a color or location that does not match the image). The manuscript does not report the agreement rule (e.g., 2 of 3 or 3 of 3), the number or rate of rejected modifications, or the distribution of modifier categories, and it does not provide free-form human answers to the modified questions on a validation subset. Without such details, the accuracy drops in Tables 1 and 2 could partly reflect changed ground truth rather than VLM sensitivity. Please report the agreement statistics and validate the protocol by collecting independent open answers to a sample of modified questions.
[Tables 1 and 2] The results tables report only percentage change in accuracy, without baseline accuracies, raw counts, confidence intervals, or per-cell p-values. Since the interpretation of a percentage change depends on the original accuracy, and since the paper's 'consistent degradation' claim rests on the pattern across 18 cells per table, the authors should report original and modified accuracy (or raw counts) for every model/modifier combination, together with 95% confidence intervals and the exact test statistic and p-value for each cell. This is necessary to assess whether negative or near-zero values (e.g., GPT-4o on Claude-3.5-Sonnet visual modifiers in Table 1: -0.26%) undermine the consistency claim.
[Section 3.2 and Table 2] Open-ended answers are scored by GPT-4o similarity, and GPT-4o is one of the three models being evaluated. This introduces a potential evaluation bias for the GPT-4o rows in Table 2 that is not addressed beyond the general statement in Limitations item 3. The authors should either evaluate open-ended answers with a judge that is not among the three tested models (a different LLM or human raters) or report a sensitivity analysis showing that the Table 2 pattern is unchanged when the similarity scores are produced by another model.
[Section 3.2.1] The sample description is internally inconsistent. The text states that questions were selected so that 'we had an equal number of instances corresponding to each question type (there are 55 question types)', but then says the sample consists of 500 yes/no and numeric questions and 500 open-ended questions; 1000 questions cannot be divided equally among 55 types. Please specify the exact sampling strata, counts per type, and the split between yes/no and numeric questions, since this affects the generalizability of the headline result.
[Section 3.2.1 and Abstract] The abstract's claim that VLMs 'consistently diminish' with modifiers is stronger than the evidence reported in the manuscript. The text itself says 'in almost all cases' and 'in most cases' for significance, and Table 1 contains a negative value (GPT-4o with Claude-3.5-Sonnet visual modifiers, -0.26%). Please state explicitly which cells are not significant and which are inconsistent with the degradation pattern, and adjust the abstract and conclusion to match the qualified results.

minor comments (5)

[Section 3] The phrase 'maximum of quality' should be 'maxim of quality', and the maxims should be introduced with consistent terminology throughout.
[Section 3.2.1] There is a typo: 'GTP-4o' should be 'GPT-4o' in the discussion of self-modified questions.
[Figure 2] The workflow figure is difficult to read; in particular, the labels 'Wrong Answer' and 'Correct Answer' under the VLM boxes are unclear and should be connected explicitly to the evaluation step.
[Section 5] Limitations item 2 appropriately notes possible bias in self-modified questions, but the paper should also report whether fresh sessions or API-level context isolation were used for the inference phase, not just for generation.
[General] The manuscript does not include a data or code availability statement; given the reliance on a sampled subset of VQA v2.0 and generated modifiers, releasing the sampled question IDs, the generated modified questions, and the AMT verification outcomes would substantially aid reproducibility.

Circularity Check

0 steps flagged · score 2.0 of 10

No construction-level circularity; the central yes/no and numeric results are anchored to VQA v2.0 ground truth and AMT verification, though GPT-4o self-scoring and self-modified questions add minor self-referential bias.

full rationale

The derivation chain is: human questions -> VLM-generated modifiers -> AMT verification that the same answer is correct -> evaluate VLMs against VQA v2.0 ground truth for yes/no and numeric questions -> report % change in accuracy. For this half of the data, correctness is defined by external VQA v2.0 ground truth, not by any fitted parameter, self-citation, or definitional identity. Asking a VLM to generate modifiers that preserve the answer and then filtering those modifiers through AMT does not make the measured accuracy drop true by construction, because the drop is measured against external labels. The self-citation to Britton et al. (2022) in Section 2 is related work and is not load-bearing. Two acknowledged self-referential edges exist: Limitations item 2 notes possible context leakage for self-modified questions, and Section 3.2.1 plus Limitations item 3 use GPT-4o to score open-ended answer similarity even though GPT-4o is one of the evaluated models. These are measurement-bias risks, not definitional circularity: the open-ended scores could be biased without making any equation tautological. No fitted parameter is renamed as a prediction, and no uniqueness theorem is imported from the authors' prior work. The claim is therefore self-contained against external benchmarks; score 2 reflects the minor self-evaluation and self-modification edges rather than a circular derivation.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The study contains no fitted mathematical parameters and introduces no entities beyond the label "visual/relational modifier". Its load-bearing assumptions are domain-level: that VQA v2.0 answers are a valid target, that AMT verification reliably detects answer-changing modifiers, and that closed API model responses are stable. These are measurement assumptions, not free parameters.

assumptions (5)

domain assumption Grice's maxims and the cooperative principle accurately model cognitive burden in task-oriented human-AI conversation.
Invoked in Section 3 to justify why added modifiers increase reasoning difficulty; the interpretation depends on it, though the raw accuracy comparison does not.
domain assumption Original human-crafted VQA v2.0 questions satisfy Grice's maxims.
Explicitly listed as Limitation 1; if false, the contrast between original and modified questions is not a clean maxim violation.
domain assumption AMT workers' judgments that the same answer holds for modified questions are accurate.
Section 3.2 and Appendix B use 3 workers per question; noisy verification could admit answer-changing modifiers and inflate the measured degradation.
domain assumption VQA v2.0 ground-truth answers provide a reliable exact-match target for yes/no and numeric questions.
Used in Section 3.2.1 for accuracy computation without independent validation.
domain assumption VLM API outputs are representative of the models and stable, with no uncontrolled context leakage between question-generation and answering.
No model snapshot, temperature, or session isolation is reported; Limitation 2 notes possible self-modification bias even with separate prompts.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Analyzing the Sensitivity of Vision Language Models in Visual Question Answering." pith.science (2026). https://pith.science/paper/NCW23S7J

@misc{pith2026250721335,
  author       = {Pith},
  title        = {Pith review of: Analyzing the Sensitivity of Vision Language Models in Visual Question Answering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NCW23S7J}},
  note         = {Machine review of arXiv:2507.21335}
}

read the original abstract

We can think of Visual Question Answering as a (multimodal) conversation between a human and an AI system. Here, we explore the sensitivity of Vision Language Models (VLMs) through the lens of cooperative principles of conversation proposed by Grice. Specifically, even when Grice's maxims of conversation are flouted, humans typically do not have much difficulty in understanding the conversation even though it requires more cognitive effort. Here, we study if VLMs are capable of handling violations to Grice's maxims in a manner that is similar to humans. Specifically, we add modifiers to human-crafted questions and analyze the response of VLMs to these modifiers. We use three state-of-the-art VLMs in our study, namely, GPT-4o, Claude-3.5-Sonnet and Gemini-1.5-Flash on questions from the VQA v2.0 dataset. Our initial results seem to indicate that the performance of VLMs consistently diminish with the addition of modifiers which indicates our approach as a promising direction to understand the limitations of VLMs.

Figures

Figures reproduced from arXiv: 2507.21335 by the authors.

**Figure 2.** Illustrating our workflow. We generate modified questions from human-crafted questions using a VLM. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 14 canonical work pages

[1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Anthropic. 2024. Claude 3.5 sonnet. https://claude.ai. Large language model

work page 2024
[4]

Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425--2433

2015
[5]

William Britton, Somdeb Sarkhel, and Deepak Venugopal. 2022. Question modifiers in visual question answering. In Language Resources and Evaluation Conference

work page 2022
[6]

Declan Campbell, Sunayana Rane, Tyler Giallanza, Camillo Nicol \`o De Sabbata, Kia Ghods, Amogh Joshi, Alexander Ku, Steven Frankland, Tom Griffiths, Jonathan D Cohen, and 1 others. 2024. Understanding the limits of vision language models through the lens of the binding problem. Advances in Neural Information Processing Systems, 37:113436--113460

work page 2024
[7]

Wei-Lin Chiang, Zhuohan Li, Ziqing Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, and 1 others. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90\ See https://vicuna. lmsys. org (accessed 14 April 2023), 2(3):6

2023
[8]

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, and 1 others. 2024. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70):1--53

2024

Show all 26 references

[9]

Marcella Cornia, Matteo Stefanini, Lorenzo Baraldi, and Rita Cucchiara. 2020. Meshed-memory transformer for image captioning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10578--10587

2020
[10]

Bethan Davies. 2000. Grice’s cooperative principle: Getting the meaning across. Leeds Working Papers in Linguistics and Phonetics, 8(1):26

2000
[11]

Thomas G Dietterich. 1998. Approximate statistical tests for comparing supervised classification learning algorithms. Neural computation, 10(7):1895--1923

1998
[12]

Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904--6913

2017
[13]

Herbert P Grice. 1975. Logic and conversation. In Speech acts, pages 41--58. Brill

1975
[14]

Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276

2024 arXiv
[15]

Baptiste Jacquet, Jean Baratgin, and Frank Jamet. 2018. The gricean maxims of quantity and of relation in the turing test. In 2018 11th international conference on human system interaction (hsi), pages 332--338. IEEE

2018
[16]

Marcel Adam Just, Patricia A Carpenter, Timothy A Keller, William F Eddy, and Keith R Thulborn. 1996. Brain activation modulated by sentence comprehension. Science, 274(5284):114--116

1996
[17]

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. Advances in neural information processing systems, 36:34892--34916

2023
[18]

Oscar Ma \ n as, Benno Krojer, and Aishwarya Agrawal. 2024. Improving automatic vqa evaluation using large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 4171--4179

2024
[19]

OpenAI. 2024. Hello gpt-4o (may 13 version). https://openai.com/index/hello-gpt-4o/. Large language model

2024
[20]

Ramprasaath R Selvaraju, Purva Tendulkar, Devi Parikh, Eric Horvitz, Marco Tulio Ribeiro, Besmira Nushi, and Ece Kamar. 2020. Squinting at vqa models: Introspecting vqa models with sub-questions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognit...

2020
[21]

Hao Tan and Mohit Bansal. 2019. Lxmert: Learning cross-modality encoder representations from transformers. arXiv preprint arXiv:1908.07490

2019 arXiv
[22]

Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, and 1 others. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530

2024 arXiv
[23]

A Trajman and RR Luiz. 2008. Mcnemar 2 test revisited: comparing sensitivity and specificity of diagnostic examinations. Scandinavian journal of clinical and laboratory investigation, 68(1):77--80

2008
[24]

Yiyu Wang, Jungang Xu, and Yingfei Sun. 2022. End-to-end transformer based model for image captioning. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pages 2585--2594

2022
[25]

Xu Yang, Kaihua Tang, Hanwang Zhang, and Jianfei Cai. 2019. Auto-encoding scene graphs for image captioning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10685--10694

2019
[26]

Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, and 1 others. 2023. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36:55006--55021

2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Anthropic. 2024. Claude 3.5 sonnet. https://claude.ai. Large language model

work page 2024

[4] [4]

Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425--2433

2015

[5] [5]

William Britton, Somdeb Sarkhel, and Deepak Venugopal. 2022. Question modifiers in visual question answering. In Language Resources and Evaluation Conference

work page 2022

[6] [6]

Declan Campbell, Sunayana Rane, Tyler Giallanza, Camillo Nicol \`o De Sabbata, Kia Ghods, Amogh Joshi, Alexander Ku, Steven Frankland, Tom Griffiths, Jonathan D Cohen, and 1 others. 2024. Understanding the limits of vision language models through the lens of the binding problem. Advances in Neural Information Processing Systems, 37:113436--113460

work page 2024

[7] [7]

Wei-Lin Chiang, Zhuohan Li, Ziqing Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, and 1 others. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90\ See https://vicuna. lmsys. org (accessed 14 April 2023), 2(3):6

2023

[8] [8]

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, and 1 others. 2024. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70):1--53

2024