Pith. sign in

REVIEW 2 major objections 3 minor 66 references

PARC: A Quantitative Framework Uncovering the Symmetries within Vision Language Models

T0 review · 2 major / 3 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read PARC claims that a single calibrated reliability score, carrying explicit guarantees on accuracy and confidence, reveals which prompt edits break vision-language models and which model families resist them.

desk verdict A serious large-scale VLM prompt-sensitivity study whose advertised centerpiece, the reliability score's 'guarantees', fails on its own equations; the empirical core survives, but the paper overclaims. read the letter →

arxiv 2506.14808 v1 pith:LENDWCEV submitted 2025-06-03 cs.LG

Jenny Schmalfuss , Nadine Chang , Vibashan VS , Maying Shen , Andres Bruhn , Jose M. Alvarez This is my paper
classification cs.LG
keywords visionlanguagemodelspromptsensitivityreliabilityscorecalibrationconformalpredictionvariationsmultiple-choicevisualquestionansweringmodelstability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Vision-language models answer user prompts made of text and images, and this paper tries to show that their sensitivity to prompt wording and image changes can be measured in a single comparable number. The authors propose PARC, which applies 11 realistic prompt variations across language and vision, computes a reliability score that combines accuracy with conformal-prediction certainty, and calibrates every score by the expected random performance of the prompt and dataset. The paper's central methodological claim is that the calibrated reliability score carries two explicit guarantees: for a positive score $r$, the model's certainty is at least $|r|$ and its calibrated accuracy is at least $r$, with the accuracy inequality reversed for negative scores. On 22 models and 7 datasets, PARC finds that VLMs are sensitive in both modalities, that semantic changes such as negations and image exchanges hurt most, and that the InternVL2 family is the most prompt-agnostic. If this is right, model selection and prompt auditing can start from one readable number rather than a spread of inconsistent metrics.

What carries the argument

The carrying object is the calibrated reliability score, a single number formed by multiplying a centered, calibrated accuracy with a conformal-prediction certainty: $\mathrm{rel} = (2\,\mathrm{acc}^m - 1)\,\mathrm{cert}$, where the exponent $m = \log 2 / \log(1/\mathrm{acc}_{\mathrm{rand}})$ shifts the neutral point to the prompt's random accuracy. Certainty is computed from conformal prediction sets over the multiple-choice options, and calibration rescales raw scores as $(s - s_{\mathrm{rand}})/(1 - s_{\mathrm{rand}})$ above random and $(s - s_{\mathrm{rand}})/s_{\mathrm{rand}}$ below random. The same calibration recipe is applied to accuracy, certainty, and consistency, which is what makes cross-dataset and cross-prompt comparisons possible; without it, negated prompts on an imbalanced dataset look artificially easy.

What would settle it

Re-run the full 22-model evaluation on the unfiltered comparison datasets, or on a fresh random subsample of image exchanges, and check whether VS-E remains among the most destructive variations and whether InternVL2 still leads; if the ordering flips, the filtering assumption is load-bearing and fails.

Watch

Extended reading notes

Core claim

PARC's core claim is that prompt sensitivity in VLMs becomes quantifiable and comparable once three pieces are in place: plausible prompt variations in text and image, a reliability score $\mathrm{rel} = (2\,\mathrm{acc}^m - 1)\,\mathrm{cert}$ with $m = \log 2 / \log(1/\mathrm{acc}_{\mathrm{rand}})$, and calibration that rescales accuracy, certainty, consistency, and reliability by their expected random baselines. The score is engineered so that a reading of $\mathrm{rel} = 0.3$ guarantees at least 30% calibrated accuracy improvement over random and at least 30% certainty, while $\mathrm{rel} = -0.5$ guarantees at least a 50% shortfall. Empirically, the paper finds that reformulations of a prompt are tolerated far better than changes that alter the expected answer, that the same gap appears for image reformulations versus image swaps and exchanges, and that reliability tracks consistency closely enough that choosing the most reliable model and the most consistent model are not in tension. The model-level result is that stability clusters by family rather than by size, with InternVL2 models leading and the best score at their largest 40B checkpoint, and the authors point to curated training data as the likely cause.

Load-bearing premise

The ranking of prompt variations assumes that the manual filtering that keeps only about 20% of samples (4.7% for one dataset) for the image-exchange variation does not change which variations are most destructive, even though the paper states that some filtered subsets are easier than the originals.

Editorial extensions

If this is right

  • A single calibrated reliability score lets a practitioner read both a lower bound on accuracy improvement over random and a lower bound on confidence from one number.
  • Comparisons across prompts and datasets are only meaningful after calibrating by expected random performance; this changes conclusions on datasets such as MMBench.
  • VLM prompt sensitivity mirrors LLM sensitivity and appears in the vision domain, so stability audits should include semantic visual edits, not just noise.
  • Prompt-agnosticism clusters by model family and improves with model size within a family, with curated training data as the indicated lever.
  • Combined language and vision variations behave like the worst single variation or worse, so per-variation evaluation is a practical proxy for combined-stress testing.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Going beyond the paper, the same calibration formula should transfer to any score with a known random baseline, including LLM-only prompt benchmarks, giving one currency for sensitivity across modalities.
  • The paper reports averages across prompts and datasets, but the at-a-glance guarantees are cleanest per prompt; a distribution-level report of calibrated reliability would preserve the guarantees and expose failure modes that means hide.
  • The training-data explanation is directly testable: take one architecture, fine-tune it on high-curation versus web-crawled data, and check whether reliability and consistency move in the predicted direction.
  • Because PARC needs softmax access, a black-box version would have to replace certainty with a sampled-output proxy, which would be a natural extension for closed models.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 3 minor

Summary. The paper introduces PARC, a framework for analyzing prompt sensitivity in vision-language models (VLMs). It defines 11 language and vision prompt variations, a reliability score that combines accuracy and certainty, and a calibration step intended to make accuracy, certainty, consistency, and reliability comparable across datasets and prompt variations. The framework is evaluated on 7 datasets with 22 VLMs from 7 model families. The main empirical claims are that VLMs are sensitive to both language and vision prompt variations, that semantic changes (especially those altering the expected answer) are most destructive, and that InternVL2 models are the most prompt-agnostic among those tested, with training data quality suggested as a possible cause.

Significance. The paper's empirical study is substantial: evaluating 22 VLMs across 7 datasets and 11 plausible prompt variations is a useful contribution, and the prompt-variation taxonomy is more realistic than prior noise-based perturbations. The proposed calibration idea addresses a real comparability problem. However, the central methodological novelty is the reliability score with its two stated guarantees in Eq. (6), and that claim is mathematically false as written. Because the advertised at-a-glance interpretability of the score rests on these guarantees, the methodological core of the paper needs correction. The empirical rankings may survive a corrected score, but the current formulation cannot be accepted as stated.

major comments (2)
  1. [Sec. 2.3, Eq. (6); also Sec. A8, Eq. (9)] The claimed guarantee acc_calib >= rel for rel > 0 does not follow from the definitions and is false. For example, with the paper's own MMBench value acc_rand = 0.27, m = log(2)/log(1/0.27) approx 0.529, take acc = 0.5 and cert = 1. Then rel = (2 * 0.5^0.529 - 1) approx 0.384, while acc_calib = (0.5 - 0.27)/(1 - 0.27) approx 0.315, so acc_calib < rel. This directly contradicts Eq. (6). The first inequality cert >= |rel| is trivially true, but the accuracy guarantee is load-bearing: the paper advertises that a positive reliability of 0.3 guarantees at least 30% improvement over random accuracy (Sec. 2.3), and this is not true. A corrected version could define the calibrated accuracy on acc^m rather than on acc, or replace the guarantee with a weaker correct statement; either way, the interpretation and the reported reliability values would need to be revisited.
  2. [Sec. 3.1, Sec. A7.3, Tab. A5] The conclusion that the vision semantic exchange (VS-E) is among the most destructive prompt variations is based on comparing VS-E, evaluated on a manually filtered subset retaining only about 20% of the original samples (and 4.7% for Fashionpedia), against other variations evaluated on the initial, unfiltered data. The paper acknowledges that filtering changes dataset difficulty, and Tab. A5 shows large retention differences across datasets. This comparison assumes that the relative difficulty ordering of prompt variations is invariant to the filtering, but that is asserted rather than demonstrated. Since VS-E is one of the headline 'most destructive' results, the authors should either evaluate all variations on a common filtered subset or otherwise control for the composition change.
minor comments (3)
  1. [Sec. A7.3, Tab. A5] Tab. A5 is labeled 'Error Rates', but the surrounding text describes the values as accuracies. The numbers for V-S and V-A increase after filtering, which is consistent with accuracies improving but not with error rates decreasing; please clarify the labeling and make the text and table consistent.
  2. [Sec. 2.1] There is a typo in the sentence introducing 'tailord variations'; it should read 'tailored variations'. Similar typos include 'Noteably' in Sec. 3.1, 'burnter' in the LLaMA prompt example in Sec. A7.2, and 'hightest' in the caption of Tab. A10.
  3. [Sec. 3.2, Fig. 5] The training-data-quality explanation is presented as suggestive ('indications'), which is appropriate, but the comparison in Fig. 5 [Right] is confounded by model family and architecture differences. Please state more explicitly that this is a correlational observation across non-controlled model families, not a causal conclusion.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation; Eq. (6) accuracy guarantee is false, not circular.

full rationale

PARC's derivation chain is largely self-contained. The reliability score rel=(2acc^m-1)cert is an explicit function of measured accuracy and conformal certainty, and the exponent m is solved analytically from the anchor condition 0=2acc_rand^m-1 rather than fitted to model outputs. The first advertised guarantee cert>=|rel| follows immediately from |2acc^m-1|<=1 and is a restatement of the definition. The second guarantee, acc_calib>=rel for rel>0, does not follow from the definitions and is in fact false: for MMBench acc_rand=0.27, m=log(2)/log(1/0.27)≈0.529, acc=0.5, and cert=1, one obtains rel≈0.384 but acc_calib=(0.5-0.27)/(1-0.27)≈0.315, violating Eq. (6). This is a mathematical soundness error rather than a circularity, because the guarantee is not assumed as an input, not fitted to data, and not derived from itself. The empirical comparisons are computed from the defined scores instead of engineered to force a desired ranking, and the calibration validation against balanced NYU-Depth V2 trends provides independent evidence. The VS-E filtering retains only about 20% of samples and is explicitly acknowledged as a limitation; this affects external validity, not circularity. The only relevant self-citation, [53], appears in the supplementary's perturbation-class selection alongside the independent reference [16] and is not load-bearing. Overall, no derivation step reduces to its own input; score 2 reflects only the minor non-load-bearing self-citation.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

No invented entities. The framework introduces no new physical or model entities. The reliability score has an analytically derived exponent, not a free fit. The only hand-set parameters are the conformal miscoverage level and the dataset filters for VS-E.

free parameters (1)
  • conformal prediction miscoverage level alpha = 0.1
    The paper sets the allowed error rate for conformal prediction sets to 10% (Sec. 2.2). This affects certainty scores for every model and every prompt.
assumptions (3)
  • domain assumption Conformal prediction sets from softmax scores are valid under exchangeability for all evaluated VLMs.
    Invoked in Sec. 2.2 when computing certainty via the LAC conformal method. If softmax scores are not exchangeable, certainty values are unreliable.
  • domain assumption Expected random performance is the correct baseline for cross-dataset and cross-prompt calibration.
    Sec. 2.3 Eq. (5) rescales all scores by acc_rand. The linear rescaling assumes the gap between observed and random performance is equally meaningful across different answer distributions.
  • ad hoc to paper The manual VS-E filtering preserves the relative difficulty of prompt variations.
    Sec. A7.3 discards a large fraction of samples to create exchangeable image pairs; the paper acknowledges filtered V-S/V-A are easier but asserts the insight is unaffected.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PARC: A Quantitative Framework Uncovering the Symmetries within Vision Language Models." pith.science (2026). https://pith.science/paper/LENDWCEV

@misc{pith2026250614808,
  author       = {Pith},
  title        = {Pith review of: PARC: A Quantitative Framework Uncovering the Symmetries within Vision Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LENDWCEV}},
  note         = {Machine review of arXiv:2506.14808}
}
read the original abstract

Vision language models (VLMs) respond to user-crafted text prompts and visual inputs, and are applied to numerous real-world problems. VLMs integrate visual modalities with large language models (LLMs), which are well known to be prompt-sensitive. Hence, it is crucial to determine whether VLMs inherit this instability to varying prompts. We therefore investigate which prompt variations VLMs are most sensitive to and which VLMs are most agnostic to prompt variations. To this end, we introduce PARC (Prompt Analysis via Reliability and Calibration), a VLM prompt sensitivity analysis framework built on three pillars: (1) plausible prompt variations in both the language and vision domain, (2) a novel model reliability score with built-in guarantees, and (3) a calibration step that enables dataset- and prompt-spanning prompt variation analysis. Regarding prompt variations, PARC's evaluation shows that VLMs mirror LLM language prompt sensitivity in the vision domain, and most destructive variations change the expected answer. Regarding models, outstandingly robust VLMs among 22 evaluated models come from the InternVL2 family. We further find indications that prompt sensitivity is linked to training data. The code will be at https://github.com/NVlabs/PARC.

Figures

Figures reproduced from arXiv: 2506.14808 by the authors.

Figure 1
Figure 1. PARC prompt sensitivity analysis framework overview. Given a collection of VLMs and datasets, PARC identifies [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Reliability score in PARC. The plots visualize how accuracy and certainty are mapped to the reliability score. A re￾liability of 1 [blue] highlights a confidently correct model, while -1 [red] flags confidently incorrect models. [Left] Mapping for a balanced dataset like NYU-Depth V2 [57], where expected ran￾dom accuracy is 0.5. [Right] Calibrated reliability scores for accrand = 0.27, which represents MMBench [37].… view at source ↗
Figure 3
Figure 3. Calibration effect in PARC. On MMBench [37] before calibration, models appear more accurate on negated prompts LS￾N [Orange, dashed] than on the original prompts O [Blue, solid], LS-N>O. After calibrating by measuring the improvement over the expected random performance, the order switches to O>LS￾N. Why should we expect VLMs to be better on O than LS-N? We take a look at NYU-Depth V2 [57] – a balanced dataset with … view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: VLMs understand reformulations [blue] better than se￾mantic changes [orange], across language and vision modalities. Trends are aligned on the six comparative datasets and MMBench. ingly, for image swaps (VS-S), the average model accuracy and reliability improve, indic…
Figure 5
Figure 5. Figure 5: [Left] Larger models within the same family are less prompt sensitive. [Right] More high-quality data yields more prompt-agnostic models. Comparison shows models with 7B or 8B LLMs per family; with Qwen-VL for Qwen, LLaVA-1.6 7B vic for LLava-1.6 and CogVLM Chat for Co…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

66 extracted references · 53 canonical work pages

  1. [1]

    3, 9, 11

    LLAMA-3: https://ai.meta.com/blog/meta-llama-3/. 3, 9, 11

    work page

  2. [2]

    Phi-3 technical report: A highly capable language model locally on your phone

    Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024. 8

    arXiv 2024

  3. [3]

    Flamingo: A visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: A visual language model for few-shot learning. Proc. Confer- ence on Neural Information Processing Systems (NeurIPS) , 35:23716–23736, 2022. 8

    work page 2022

  4. [4]

    How susceptible are LLMs to influence in prompts? In Proc

    Sotiris Anagnostidis and Jannis Bulian. How susceptible are LLMs to influence in prompts? In Proc. Conference on Lan- guage Modeling (CoLM), 2024. 1, 2

    work page 2024

  5. [5]

    A gentle in- troduction to conformal prediction and distribution-free un- certainty quantification

    Anastasios N Angelopoulos and Stephen Bates. A gentle in- troduction to conformal prediction and distribution-free un- certainty quantification. arXiv preprint arXiv:2107.07511 ,

    arXiv

  6. [6]

    This is not correct! Negation-aware evaluation of lan- guage generation systems

    Miriam Ansch ¨utz, Diego Miguel Lozano, and Georg Groh. This is not correct! Negation-aware evaluation of lan- guage generation systems. In Proc. International Natural Language Generation Conference (INLG) , pages 163–175, Prague, Czechia, 2023. Association for Computational Lin- guistics. 2, 8

    work page 2023

  7. [7]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023. 5, 8, 9

    arXiv 2023

  8. [8]

    Benchmarking robustness of adaptation methods on pre-trained vision-language models

    Shuo Chen, Jindong Gu, Zhen Han, Yunpu Ma, Philip Torr, and V olker Tresp. Benchmarking robustness of adaptation methods on pre-trained vision-language models. In Thirty- seventh Conference on Neural Information Processing Sys- tems Datasets and Benchmarks Track, 2023. 8

    work page 2023

Show all 66 references
  1. [9]

    PaLI: A jointly-scaled multilingual language- image model

    Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish V Thapliyal, James...

    2023

  2. [10]

    InternVL: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. InternVL: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proc. IEE...

    2024

  3. [11]

    Measuring and improving consistency in pre- trained language models

    Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Sch ¨utze, and Yoav Goldberg. Measuring and improving consistency in pre- trained language models. Transactions of the Association for Computational Linguistics (TACL), 9:1012–1031, 2021. 1, 2, 8

    2021

  4. [12]

    IsoBench: Benchmarking multimodal founda- tion models on isomorphic representations

    Deqing Fu, Ruohao Guo, Ghazal Khalighinejad, Ollie Liu, Bhuwan Dhingra, Dani Yogatama, Robin Jia, and Willie Neiswanger. IsoBench: Benchmarking multimodal founda- tion models on isomorphic representations. arXiv preprint arXiv:2404.01266, 2024. 2

    2024 arXiv

  5. [13]

    Sensitivity and ro- bustness of large language models to prompt template in Japanese text classification tasks

    Chengguang Gan and Tatsunori Mori. Sensitivity and ro- bustness of large language models to prompt template in Japanese text classification tasks. In Proc. Pacific Asia Con- ference on Language, Information and Computation , pages 1–11, Hong Kong, China, 2023. Association for ...

    2023

  6. [14]

    Demystifying prompts in language models via perplexity estimation

    Hila Gonen, Srini Iyer, Terra Blevins, Noah Smith, and Luke Zettlemoyer. Demystifying prompts in language models via perplexity estimation. In Findings of the Association for Computational Linguistics: EMNLP 2023 , pages 10136– 10148, Singapore, 2023. Association for Computati...

    2023

  7. [15]

    Robustness of learn- ing from task instructions

    Jiasheng Gu, Hongyu Zhao, Hanzi Xu, Liangyu Nie, Hongyuan Mei, and Wenpeng Yin. Robustness of learn- ing from task instructions. In Findings of the Association for Computational Linguistics: ACL , pages 13935–13948, Toronto, Canada, 2023. Association for Computational Lin- guistics. 8

    2023

  8. [16]

    Benchmarking neu- ral network robustness to common corruptions and perturba- tions

    Dan Hendrycks and Thomas Dietterich. Benchmarking neu- ral network robustness to common corruptions and perturba- tions. In Proc. International Conference on Learning Repre- sentations (ICLR), 2019. 3, 8, 9

    2019

  9. [17]

    CogVLM2: Visual language mod- els for image and video understanding

    Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang, Yean Cheng, Shiyu Huang, Jun- hui Ji, Zhao Xue, et al. CogVLM2: Visual language mod- els for image and video understanding. arXiv preprint arXiv:2408.16500, 2024. 5, 9

    2024 arXiv

  10. [18]

    Under- standing by understanding Not: Modeling negation in lan- guage models

    Arian Hosseini, Siva Reddy, Dzmitry Bahdanau, R Devon Hjelm, Alessandro Sordoni, and Aaron Courville. Under- standing by understanding Not: Modeling negation in lan- guage models. In Proc. Conference of the North American Chapter of the Association for Computational Linguistic...

  11. [19]

    A tutorial on calibration measurements and calibration models for clinical prediction models

    Yingxiang Huang, Wentao Li, Fima Macheret, Rodney A Gabriel, and Lucila Ohno-Machado. A tutorial on calibration measurements and calibration models for clinical prediction models. Journal of the American Medical Informatics Asso- ciation (JAMIA), 27(4):621–633, 2020. 8

    2020

  12. [20]

    GQA: A new dataset for real-world visual reasoning and composi- tional question answering

    Drew A Hudson and Christopher D Manning. GQA: A new dataset for real-world visual reasoning and composi- tional question answering. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6700–6709, 2019. 2, 4, 8

    2019

  13. [21]

    Dis- covering states and transformations in image collections

    Phillip Isola, Joseph J Lim, and Edward H Adelson. Dis- covering states and transformations in image collections. In Proc. IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 1383–1391, 2015. 5, 11

    2015

  14. [22]

    BECEL: Benchmark for consistency evaluation of language models

    Myeongjun Jang, Deuk Sin Kwon, and Thomas Lukasiewicz. BECEL: Benchmark for consistency evaluation of language models. In Proc. International Conference on Computational Linguistics (COLING) , pages 3680–3696, Gyeongju, Republic of Korea, 2022. International Committee on Compu...

    2022

  15. [23]

    Scaling up visual and vision-language representa- tion learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representa- tion learning with noisy text supervision. In Proc. Inter- national Conference on Machine Learning (ICML) ...

    2021

  16. [24]

    Fashionpedia: Ontology, segmentation, and an at- tribute localization dataset

    Menglin Jia, Mengyun Shi, Mikhail Sirotenko, Yin Cui, Claire Cardie, Bharath Hariharan, Hartwig Adam, and Serge Belongie. Fashionpedia: Ontology, segmentation, and an at- tribute localization dataset. InProc. European Conference on Computer Vision (ECCV), pages 316–332. Spring...

    2020

  17. [25]

    Evaluating VLMs for score-based, multi-probe annotation of 3D objects

    Rishabh Kabra, Loic Matthey, Alexander Lerchner, and Niloy Mitra. Evaluating VLMs for score-based, multi-probe annotation of 3D objects. In NeurIPS 2023 Workshop on Synthetic Data Generation with Generative AI, 2023. 8

    2023

  18. [26]

    3D common corruptions and data augmentation

    O ˘guzhan Fatih Kar, Teresa Yeo, Andrei Atanov, and Amir Zamir. 3D common corruptions and data augmentation. In Proc. IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 18963–18974, 2022. 9

    2022

  19. [27]

    Consistency and uncertainty: Identi- fying unreliable responses from black-box vision-language models for selective visual question answering

    Zaid Khan and Yun Fu. Consistency and uncertainty: Identi- fying unreliable responses from black-box vision-language models for selective visual question answering. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10854–10863, 2024. 1, 2, 8

    2024

  20. [28]

    MLLM-CompBench: A comparative reasoning benchmark for multimodal llms

    Jihyung Kil, Zheda Mai, Justin Lee, Zihe Wang, Kerrie Cheng, Lemeng Wang, Ye Liu, Arpita Chowdhury, and Wei- Lun Chao. MLLM-CompBench: A comparative reasoning benchmark for multimodal llms. Proc. Conference on Neu- ral Information Processing Systems (NeurIPS) , 2024. 2, 3, 5, ...

    2024

  21. [29]

    Uncertainty-aware evaluation for vision- language models

    Vasily Kostumov, Bulat Nutfullin, Oleg Pilipenko, and Eu- gene Ilyushin. Uncertainty-aware evaluation for vision- language models. arXiv preprint arXiv:2402.14418 , 2024. 2, 4, 8, 12

    2024 arXiv

  22. [30]

    Confor- mal prediction with large language models for multi-choice 17 question answering

    Bhawesh Kumar, Charlie Lu, Gauri Gupta, Anil Palepu, David Bellamy, Ramesh Raskar, and Andrew Beam. Confor- mal prediction with large language models for multi-choice 17 question answering. arXiv preprint arXiv:2305.18404, 2023. 8

    2023 arXiv

  23. [31]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In Proc. International Conference on Machine Learning (ICML) , pages 19730–19742. PMLR, 2023. 8

    2023

  24. [32]

    Zhenxin Li, Kailin Li, Shihao Wang, Shiyi Lan, Zhiding Yu, Yishen Ji, Zhiqi Li, Ziyue Zhu, Jan Kautz, Zuxuan Wu, Yu-Gang Jiang, and Jose M. Alvarez. Hydra-MDP: End-to- end multimodal planning with multi-target hydra-distillation. arXiv preprint arXiv:2406.06978, 2024. 1

    2024 arXiv

  25. [33]

    External validation of four dementia prediction models for use in the general community-dwelling population: a comparative analysis from the Rotterdam Study

    Silvan Licher, Pınar Yilmaz, Maarten JG Leening, Frank J Wolters, Meike W Vernooij, Blossom CM Stephan, M Kam- ran Ikram, and M Arfan Ikram. External validation of four dementia prediction models for use in the general community-dwelling population: a comparative analysis from...

    2018

  26. [34]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26296–26306, 2024. 5, 8, 9

    2024

  27. [35]

    LLaV A-NeXT: Improved reasoning, OCR, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. LLaV A-NeXT: Improved reasoning, OCR, and world knowledge, 2024. https://llava-vl.github.io/blog/2024-01-30-llava-next/. 5, 9

    2024

  28. [36]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Proc. Conference on Neural Infor- mation Processing Systems (NeurIPS), 36, 2024. 8

    2024

  29. [37]

    MMBench: Is your multi-modal model an all-around player? In Proc

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. MMBench: Is your multi-modal model an all-around player? In Proc. European Conference on Com- puter Vision (ECCV), pages 216–233. Springer, 2025. 2, ...

    2025

  30. [38]

    Ex- ploring the sensitivity of LLMs’ decision-making capabili- ties: Insights from prompt variations and hyperparameters

    Manikanta Loya, Divya Sinha, and Richard Futrell. Ex- ploring the sensitivity of LLMs’ decision-making capabili- ties: Insights from prompt variations and hyperparameters. In Findings of the Association for Computational Linguis- tics: EMNLP, pages 3711–3716, Singapore, 2023. ...

    2023

  31. [39]

    MathVista: Evaluating mathemat- ical reasoning of foundation models in visual contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. MathVista: Evaluating mathemat- ical reasoning of foundation models in visual contexts. In Proc. International Conference on Learning Rep...

    2024

  32. [40]

    Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitiv- ity

    Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitiv- ity. In Proc. Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers)...

    2022

  33. [41]

    Ecker, Matthias Bethge, and Wieland Brendel

    Claudio Michaelis, Benjamin Mitzkus, Robert Geirhos, Evgenia Rusak, Oliver Bringmann, Alexander S. Ecker, Matthias Bethge, and Wieland Brendel. Benchmarking ro- bustness in object detection: Autonomous driving when winter is coming. In NeurIPS 2019 Workshop on Ma- chine Learni...

    2019

  34. [42]

    Promptaid: Prompt exploration, perturbation, testing and iteration using visual analytics for large language models

    Aditi Mishra, Utkarsh Soni, Anjana Arunkumar, Jinbin Huang, Bum Chul Kwon, and Chris Bryan. Promptaid: Prompt exploration, perturbation, testing and iteration using visual analytics for large language models. arXiv preprint arXiv:2304.01964, 2023. 8

    2023 arXiv

  35. [43]

    State of what art? A call for multi-prompt LLM evaluation

    Moran Mizrahi, Guy Kaplan, Dan Malkin, Rotem Dror, Dafna Shahaf, and Gabriel Stanovsky. State of what art? A call for multi-prompt LLM evaluation. Transactions of the Association for Computational Linguistics, 12:933–949,

  36. [44]

    Can you trust your model’s uncertainty? Evaluating predictive uncertainty under dataset shift

    Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshmi- narayanan, and Jasper Snoek. Can you trust your model’s uncertainty? Evaluating predictive uncertainty under dataset shift. Proc. Conference on Neural Informatio...

    2019

  37. [45]

    Learning to pre- dict visual attributes in the wild

    Khoi Pham, Kushal Kafle, Zhe Lin, Zhihong Ding, Scott Co- hen, Quan Tran, and Abhinav Shrivastava. Learning to pre- dict visual attributes in the wild. In Proc. IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR) , pages 13018–13028, 2021. 5, 11

    2021

  38. [46]

    What is the limitation of multimodal LLMs? A deeper look into multimodal LLMs through prompt probing

    Shuhan Qi, Zhengying Cao, Jun Rao, Lei Wang, Jing Xiao, and Xuan Wang. What is the limitation of multimodal LLMs? A deeper look into multimodal LLMs through prompt probing. Information Processing & Management, 60 (6):103510, 2023. 1, 2, 8

    2023

  39. [47]

    Med- ical image understanding with pretrained vision language models: A comprehensive study

    Ziyuan Qin, Hua Hui Yi, Qicheng Lao, and Kang Li. Med- ical image understanding with pretrained vision language models: A comprehensive study. InProc. International Con- ference on Learning Representations (ICLR), 2023. 1

    2023

  40. [48]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In Proc. International Conference on Machine Le...

    2021

  41. [49]

    Allen Z. Ren, Anushri Dixit, Alexandra Bodrova, Sumeet Singh, Stephen Tu, Noah Brown, Peng Xu, Leila Takayama, Fei Xia, Jake Varley, Zhenjia Xu, Dorsa Sadigh, Andy Zeng, and Anirudha Majumdar. Robots that ask for help: Uncer- tainty alignment for large language model planners....

    2023

  42. [50]

    MultiMedEval: A benchmark and a toolkit for evaluating medical vision-language models

    Corentin Royer, bjoern menze, and Anjany Sekuboyina. MultiMedEval: A benchmark and a toolkit for evaluating medical vision-language models. Medical Imaging with Deep Learning, 2024. 1

    2024

  43. [51]

    Least ambiguous set-valued classifiers with bounded error levels

    Mauricio Sadinle, Jing Lei, and Larry Wasserman. Least ambiguous set-valued classifiers with bounded error levels. Journal of the American Statistical Association (JASA), 114 (525):223–234, 2019. 4

    2019

  44. [52]

    Robustness analysis of video- language models against visual and language perturbations

    Madeline Schiappa, Shruti Vyas, Hamid Palangi, Yogesh Rawat, and Vibhav Vineet. Robustness analysis of video- language models against visual and language perturbations. 18 Proc. Conference on Neural Information Processing Systems (NeurIPS), 35:34405–34420, 2022. 3, 8, 9

    2022

  45. [53]

    RobustSpring: Benchmarking robustness to image corrup- tions for optical flow, scene flow and stereo

    Jenny Schmalfuss, Victor Oei, Lukas Mehl, Madlen Bartsch, Shashank Agnihotri, Margret Keuper, and Andr ´es Bruhn. RobustSpring: Benchmarking robustness to image corrup- tions for optical flow, scene flow and stereo. arXiv preprint arXiv:2505.09368, 2025. 9

    2025 arXiv

  46. [54]

    Quantifying language models’ sensitivity to spurious fea- tures in prompt design or: How i learned to start worrying about prompt formatting

    Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying language models’ sensitivity to spurious fea- tures in prompt design or: How i learned to start worrying about prompt formatting. In Proc. International Conference on Learning Representations (ICLR), 2024. 1, 8

    2024

  47. [55]

    BLEURT: Learning robust metrics for text generation

    Thibault Sellam, Dipanjan Das, and Ankur P Parikh. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881–7892, 2020. 8

    2020

  48. [56]

    Lm- nav: Robotic navigation with large pre-trained models of language, vision, and action

    Dhruv Shah, Bła ˙zej Osi ´nski, Sergey Levine, et al. Lm- nav: Robotic navigation with large pre-trained models of language, vision, and action. In Proc. Conference on Robot Learning (CoRL), pages 492–504. PMLR, 2023. 1

    2023

  49. [57]

    Indoor segmentation and support inference from rgbd images

    Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In Proc. European Conference on Computer Vision (ECCV), pages 746–760. Springer, 2012. 4, 5, 11, 12, 13

    2012

  50. [58]

    Evalu- ating the zero-shot robustness of instruction-tuned language models

    Jiuding Sun, Chantal Shaib, and Byron C Wallace. Evalu- ating the zero-shot robustness of instruction-tuned language models. In Proc. International Conference on Learning Rep- resentations (ICLR), 2024. 2, 8

    2024

  51. [59]

    Measuring ro- bustness to natural distribution shifts in image classification

    Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Car- lini, Benjamin Recht, and Ludwig Schmidt. Measuring ro- bustness to natural distribution shifts in image classification. Proc. Conference on Neural Information Processing Systems (NeurIPS), 33:18583–18599, 2020. 8

    2020

  52. [60]

    Cambrian-1: A fully open, vision-centric exploration of multimodal LLMs

    Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, Austin Wang, Rob Fergus, Yann LeCun, and Saining Xie. Cambrian-1: A fully open, vision-centric exploration of multimodal LLMs. arXiv...

    2024 arXiv

  53. [61]

    Mind your format: Towards consistent evaluation of in-context learning improvements

    Anton V oronov, Lena Wolf, and Max Ryabinin. Mind your format: Towards consistent evaluation of in-context learning improvements. arXiv preprint arXiv:2401.06766, 2024. 1, 8

    2024 arXiv

  54. [62]

    BLINK: Multimodal large language models can see but not perceive

    Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei- Chiu Ma, and Ranjay Krishna. BLINK: Multimodal large language models can see but not perceive. InProc. European Conference on Computer Vision (ECCV). Springer, 2024. 2

    2024

  55. [63]

    CogVLM: Visual expert for pretrained language models

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. CogVLM: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03079 , 2023. 5, 8, 9

    2023 arXiv

  56. [64]

    Benchmarking LLMs via uncertainty quantification

    Fanghua Ye, Mingming Yang, Jianhui Pang, Longyue Wang, Derek F Wong, Emine Yilmaz, Shuming Shi, and Zhaopeng Tu. Benchmarking LLMs via uncertainty quantification. arXiv preprint arXiv:2401.12794, 2024. 2, 4, 8, 12

    2024 arXiv

  57. [65]

    Be- yond positive scaling: How negation impacts scaling trends of language models

    Yuhui Zhang, Michihiro Yasunaga, Zhengping Zhou, Jeff Z HaoChen, James Zou, Percy Liang, and Serena Yeung. Be- yond positive scaling: How negation impacts scaling trends of language models. In Findings of the Association for Com- putational Linguistics: ACL 2023 , pages 7479–7...

    2023

  58. [2021]

    Association for Computational Linguistics. 2, 8

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.