Pith. sign in

REVIEW 3 major objections 5 minor 47 references

Mild image corruption can improve a vision-language model's top-1 accuracy while the model's internal prediction structure—confidence and calibration—is simultaneously deteriorating; the paper proposes RAS, a metric that detects this hidden

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 20:34 UTC pith:GSEHUD5F

load-bearing objection BENCH-C is a genuinely useful discriminative robustness testbed, and RAS is a promising but over-named metric; the headline finding is coherent only once you accept the per-sample calibration proxy at face value. the 3 major comments →

arxiv 2511.19032 v2 pith:GSEHUD5F submitted 2025-11-24 cs.CV

Diagnosing Corruption-Induced Reliability Failures in Vision-Language Models

Xiangjie Sui , Songyang Li , Hanwei Zhu , Baoliang Chen , Yuming Fang , Xin Sun This is my paper
classification cs.CV
keywords vision-language modelscorruption robustnesscalibrationprediction uncertaintymultiple-choice benchmarkreliabilityBench-CRAS
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Vision-language models can keep giving the same answer under image corruption while the distribution of confidence they assign to options quietly degrades—or they can stumble into a correct answer through instability. This paper argues that top-1 accuracy is therefore an incomplete robustness metric, and builds a testbed (Bench-C) and a metric (RAS) to see what accuracy hides. It tries to establish that mild corruptions can raise accuracy even as the model's prediction structure reliably worsens: across 13 VLMs, all 28 severity-1 cases with accuracy gains had negative RAS. A sympathetic reader should care because if this holds, accuracy-based robustness leaderboards can invert and misorder models, and 'improvements' from mild corruption are better read as fragility.

Core claim

The central claim is that the internal prediction structure of VLMs—the probability mass spread over answer options—typically degrades under visual corruption even when final-answer accuracy rises. The authors support this by measuring, on 80,655 corrupted samples, changes in normalized entropy and per-sample calibration error, and combining them into RAS. They find that the 28 model-corruption pairs with improved top-1 accuracy at severity 1 all have negative RAS, which they interpret as global structural deterioration. They also show that RAS and accuracy shift can rank models differently, and that decomposing robustness into destructive (on originally correct samples) and corrective (on o

What carries the argument

RAS, the Robustness Alignment Score, combines two shifts: ΔS, the change in normalized entropy of the option probabilities, and ΔC, the change in per-sample calibration error |1{ŷ=y} − max p|. The score is RAS = −ΔC − [ΔC]₊[−ΔS]₊ − [ΔS]₊[−ΔC]₊, rewarding calibration improvements and penalizing two failure regimes: erroneous overconfidence (confidence rises while calibration worsens) and hesitation (uncertainty rises while calibration improves). Bench-C selects discriminative samples using the Gini impurity of predictions across corruptions and selector models, plus semantic diversity, yielding 849 questions that expose robustness gaps accuracy alone misses.

Load-bearing premise

The load-bearing premise is that per-sample calibration error C = |1{ŷ=y} − max p| measures reliability; it conflates correctness with calibration and can be inflated whenever corruption rescales the logits, so ΔC and ΔS may partly reflect scale artifacts rather than genuine structural change.

What would settle it

Take one of the 28 severity-1 accuracy-gain cases (e.g., a model under brightness corruption), re-scale the corrupted logits so that max p matches a well-calibrated reference (temperature scaling), and recompute ΔC and RAS. If RAS becomes non-negative after rescaling across a majority of these cases, the structural-degradation claim is not robust to the scale-artifact critique; if RAS stays negative, the conclusion survives.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Accuracy-based robustness rankings of VLMs can invert when measured with RAS; the paper's Table 1 gives one such reversal.
  • A model that keeps the same final answer under corruption can still lose structural reliability, so identical outputs can correspond to very different internal states.
  • Improvements under mild corruption should not be trusted as robustness; all 28 observed accuracy-gain cases carry negative RAS.
  • The destructive/corrective decomposition suggests that a model that appears to recover on previously wrong samples may simply be unstable, not genuinely robust.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because per-sample calibration error conflates correctness with calibration, temperature-scaling the logits before computing ΔC could be a cheap test of whether RAS's degradation signal is partly a scale artifact—my inference, not the paper's.
  • The MCQ-only setup limits RAS to option distributions; a structural analog for open-ended generation would need token-level probability changes, which the paper does not address.
  • The paper's observation that higher-accuracy models show lower RAS points to a possible stability-capability trade-off, testable by fine-tuning one base model and evaluating the variants on Bench-C.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces Bench-C, a multiple-choice corruption-robustness benchmark built by selecting semantically diverse and prediction-inconsistent samples from six LVLM benchmarks, and the Robustness Alignment Score (RAS), a metric that combines uncertainty shift (Eq. 7) with a per-sample calibration shift (Eq. 8). The authors evaluate 13 LVLMs under 19 corruption types at 5 severity levels and report three main findings: (i) mild accuracy gains under subtle corruption are accompanied by negative RAS in all 28 such cases, so the gains are argued to be illusory; (ii) accuracy-based rankings can reverse when RAS is used (e.g., Falcon2-VLM vs. mPLUG-Owl3 in Table 1); and (iii) destructive vs. corrective robustness decompose into complementary failure and recovery patterns. The paper includes a proof that text-similarity metrics collapse to accuracy on MCQ tasks (Appendix 9.1).

Significance. If the central empirical claim is robust, the paper makes a useful contribution: a discriminative corruption benchmark for LVLMs and a metric that exposes reliability changes invisible to top-1 accuracy. The Appendix 9.1 proof is a clean, useful observation, and the release of code/data supports reproducibility. However, the headline finding—that all accuracy-gain cases still correspond to structural degradation—rests on the specific per-sample calibration proxy in Eq. (8), which is not the standard group-level calibration error. The paper would be substantially strengthened by showing that the negative-RAS pattern persists under a standard ECE-style calibration measure, and by resolving an internal inconsistency in Table 3. The current evidence is consistent with the authors' interpretation, but not yet decisive.

major comments (3)
  1. [Section 5, Eq. (8)] The per-sample 'calibration error' C = |1{ŷ=y} − max p| is not a calibration error in the standard, group-level sense. For correct samples it equals 1 − confidence, and for wrong samples it equals confidence. Thus ΔC and RAS conflate correctness transitions with confidence drift, and any monotone rescaling of logits (which corruptions such as blur, brightness, and JPEG compression can induce) changes C even when calibrated reliability is unchanged. Since the paper's central claim—negative RAS in all 28 accuracy-gain cases (Sec. 6.3, Table 3)—is mediated entirely by this proxy, the claim as stated is not yet supported. Please re-run the RAS analysis with a standard bin-based ECE or a proper per-sample strictly proper scoring rule (e.g., Brier-score decomposition), and report whether all 28 cases remain negative; alternatively, justify why this one-sided score is the intended reliability m
  2. [Table 3 / Section 9.3] There is an internal inconsistency between Eq. (10) and the 'Stable' rows in Table 3. For example, Molmo Contrast has mean ΔS = −0.004 and mean ΔC = −0.006, and SAIL-VL-1.6 Brightness has ΔS = −0.016, ΔC = −0.001; both are labeled Stable and yet report negative mean RAS values. According to Eq. (10), when ΔC < 0 and ΔS < 0 the overconfidence and hesitation penalty terms vanish and RAS = −ΔC > 0 for each such sample, so the sample-level mean should be positive. Either the aggregation is nonlinear in a way that must be stated explicitly, or the equation/table labels are misreported. This directly affects the summary that all 28 improved-accuracy cases have negative RAS, and needs to be resolved.
  3. [Table 1 and Section 6.1] Table 1 reports aggregate RAS and ΔAcc. values without confidence intervals, bootstrap errors, or repeated-seed variation. The key rank-reversal example (Falcon2-VLM ΔRAS = −0.067 vs. mPLUG-Owl3 ΔRAS = −0.059) is a difference of 0.008, which may be within sampling noise for a benchmark of 849 samples and a nonlinear metric. The paper should include bootstrap CIs or a paired significance test for the model-level RAS differences, especially for the claims that accuracy-based rankings invert and that higher clean accuracy correlates with greater RAS degradation.
minor comments (5)
  1. [Eq. (10)] The typesetting of Eq. (10) is ambiguous: the positive-part operators and the minus signs preceding the penalty terms are not visually separated. Please write the formula in a standard form, e.g., RAS = −ΔC − [ΔC]_+[−ΔS]_+ − [ΔS]_+[−ΔC]_+, and define [x]_+ clearly.
  2. [Section 4.3 / Eq. (3)] The selection weights α1 and α2 are both set to 1 with no sensitivity analysis. Since the benchmark is defined by this selection, it would be useful to report how strongly the composition and the final findings vary with α1/α2, or at least to state why the default is natural.
  3. [Figure 6] The caption refers to 'cooler hue' to indicate lower RAS values, but the figure appears in grayscale. Please add a colorbar or use a perceptually uniform colormap and make the direction explicit.
  4. [Appendix 9.2 / Table 2] The t-statistics are extremely large (35–70) and the p-values are correspondingly tiny. The text correctly notes this arises after z-score normalization, but it would help to also report the effect size (e.g., mean paired difference and its standard deviation), rather than only t and p, to allow readers to judge practical significance.
  5. [Throughout] There are several typos, e.g., 'Deversity' in the Fig. 2 caption, 'sveral' in Appendix 9.4, and 'follow' in Sec. 7 ('are follow'). A careful proofread is recommended.

Circularity Check

0 steps flagged

No significant circularity: RAS is a fixed, a priori-defined metric; the 28-case accuracy-gain finding is empirical, and the benchmark's κ re-weighting does not force the result.

full rationale

The paper's derivation chain is self-contained. RAS is defined directly in Eqs. (6)-(10) from normalized entropy and the per-sample score C=|1{ŷ=y} − max p|; no parameter is fitted to the 13 evaluated models or to the 28 accuracy-gain cases. The central empirical claim—that mild corruptions can improve top-1 accuracy while RAS is negative—is not forced by construction: RAS is not designed so that an accuracy gain guarantees a negative value, and the paper itself shows a wrong-to-correct sample with positive RAS (Fig. 5c, ΔAcc.=+1, RAS=+0.500). The benchmark includes a mild self-referential design, since the selection objective in Eq. (3) uses prediction inconsistency κ(q) and the same κ(q) is later used as a weighting term in Eq. (11); this enriches the benchmark for unstable samples and may amplify observed patterns, but it does not mathematically determine the sign of RAS or the 28-case result. No load-bearing self-citation or imported uniqueness theorem is present; the cited 'visual quality paradox' [40] is an external prior result. Concerns that Eq. (8) conflates correctness with confidence and may reflect logit-scale artifacts are validity and interpretation questions, not circularity, because the metric's values could in principle have moved in the opposite direction. Therefore the paper exhibits no significant circularity.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

RAS is a free design rather than a derived quantity: its component shifts are standard, but the combination and its unit weights are chosen by hand. The benchmark's selection criterion, size, and the re-weighting scheme are also free choices. The per-sample calibration proxy and the small-selector generalization are the two assumptions carrying the most weight.

free parameters (4)
  • Selection weights α1, α2 = 1, 1
    Hand-set in Eq. (3), Sec. 4.3; the balance between discriminative power and semantic diversity is arbitrary.
  • BENCH-C size threshold = 849 (samples whose final scores exceed the mean)
    Sec. 4.4: 'preserve the samples whose final scores are above the mean'; determines the benchmark's composition.
  • Candidate pool size = 4,000 (~200 per task)
    Sec. 4.4: balanced sampling from six source benchmarks; hand-set.
  • RAS penalty terms (unit weights) = 1 (implicit)
    Eq. (10): the positive-part products [ΔC]+[−ΔS]+ and [ΔS]+[−ΔC]+ enter with weight 1; the relative weighting is a design choice, not derived.
axioms (5)
  • domain assumption Softmax probabilities over the K options are meaningful calibrated confidence measures for all 13 evaluated LVLMs.
    Sec. 3 and Sec. 5, Eqs. (6)–(9): ΔS and ΔC treat max p and normalized entropy as model confidence; LVLMs are generative and option logits are an artifact of the evaluation harness.
  • ad hoc to paper Per-sample |1{ŷ=y} − max p| is a calibration error.
    Eq. (8): standard calibration error is a group-level expectation; the per-sample definition conflates correctness with calibration and is not standard.
  • domain assumption Prediction inconsistency of three small selector models generalizes to discriminative power for the 13 evaluated models.
    Eqs. (4)–(5), Sec. 4.3: κ(q) is computed from MoonDream2, InternVL3-1B, and Ovis2-1B only, then used to select samples and re-weight results (Eq. 11).
  • domain assumption ImageNet-C corruption types and severity levels transfer to arbitrary VLM multiple-choice inputs.
    Sec. 4.2: 19 corruptions × 5 severities applied directly to source benchmark images.
  • ad hoc to paper Similarity between any two distinct option labels is approximately a constant c ≪ 1.
    App. 9.1, Eqs. (15)–(16): the claim that text-similarity metrics collapse to accuracy requires this unverified approximation.

pith-pipeline@v1.3.0-alltime-deepseek · 27187 in / 17874 out tokens · 176514 ms · 2026-08-03T20:34:48.598241+00:00 · methodology

0 comments
read the original abstract

Visual corruptions can change vision--language model (VLM) behavior in ways that top-1 accuracy does not capture. A model may keep the same answer while losing distributional support, or improve accuracy through unstable wrong-to-correct changes. We introduce Bench-C, a controlled multiple-choice testbed for studying these effects. It selects semantically diverse samples whose predictions respond to corruption, and evaluates them under 19 corruption types and five severity levels. To measure how corruption changes the option distribution, we introduce the Robustness Alignment Score (RAS), which combines confidence-correctness alignment with uncertainty direction. We further separate originally correct samples from originally wrong samples, and track whether changes are temporary or persistent across severity. Experiments across 13 VLMs reveal a counterintuitive pattern: mild corruptions can improve top-1 accuracy while degrading prediction structure. These failures include silent degradation, erroneous overconfidence, and severity-dependent persistence. Bench-C therefore supports robustness evaluation that goes beyond final answers and attributes where reliability changes occur. Code and data are available at https://github.com/xiangjieSui/Bench-C.

Figures

Figures reproduced from arXiv: 2511.19032 by Baoliang Chen, Hanwei Zhu, Songyang Li, Xiangjie Sui, Xin Sun, Yuming Fang.

Figure 1
Figure 1. Figure 1: Motivation of this paper. We address two primary limita [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: First, we collect a diverse set of source bench￾marks (Sec. 4.1), covering a wide range of tasks and do￾mains. Second, a visual corruption module is introduced to apply diverse corruption types and severity levels levels on visual inputs (Sec. 4.2). Finally, we perform a discrim￾inative sample selection strategy by jointly considering the semantic diversity and discriminative power (Sec. 4.3). 4.1. Source … view at source ↗
Figure 3
Figure 3. Figure 3: Representative prediction structure shift under corruption. Case 1: [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Behavioral analysis of MPLUG-OWL3 under Snow corruption (ℓ=5). Each scatter plot visualizes samples in the (∆S, ∆C) plane, divided into four behavioral types: (Erroneous) Overconfident, Degraded, Stable, and Hesitant. Panels (b)–(c) isolate samples that maintain or shift predictions. While ∆Acc. measures correctness change, RAS exposes how uncertainty and calibration jointly evolve, revealing structural de… view at source ↗
Figure 5
Figure 5. Figure 5: Illustration of robustness analysis of DeepSeek-VL2-Small [ [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Destructive and corrective robustness analysis accross different corruptions. (a) Destructive robustness analysis, where the RAS [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: The composition of BENCH-C. which shows that BENCH-C maintains broad tasks. Cate￾gories such as Image Retrieval, Web, Map, Chart and Sci￾ence & Technology contribute a substantial number of dis￾criminative samples, indicate these tasks are visually de￾manding or corruption-sensitive. Although some tasks are highly visual reliance, e.g., Keypoint Detection, they con￾tribute fewer likely due to they are stil… view at source ↗
Figure 8
Figure 8. Figure 8: Category-wise sample distribution in BENCH-C. Each bar shows the total number of candidate samples (light gray) and the finally selected high-discriminative samples (hatched) [PITH_FULL_IMAGE:figures/full_fig_p012_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Examples of samples with low, medium, and high discriminative power [PITH_FULL_IMAGE:figures/full_fig_p012_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Boxplots of RAS across corruption types and severity levels. Each panel corresponds to one corruption family: Photometric, [PITH_FULL_IMAGE:figures/full_fig_p015_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

47 extracted references · 14 linked inside Pith

  1. [1]

    Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras.CoRR, abs/2503.01743, 2025

    Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, et al. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras.CoRR, abs/2503.01743, 2025. 2, 5, 3

    Pith/arXiv arXiv 2025

  2. [2]

    Qwen2.5-vl tech- nical report.CoRR, abs/2502.13923, 2025

    Shuai Bai, Keqin Chen, Xuejing Liu, et al. Qwen2.5-vl tech- nical report.CoRR, abs/2502.13923, 2025. 2, 5, 6, 3

    Pith/arXiv arXiv 2025

  3. [3]

    train on the test set

    Ellis Brown, Jihan Yang, Shusheng Yang, et al. Benchmark designers should “train on the test set”’ to expose exploitable non-visual shortcuts.CoRR, abs/2511.04655, 2025. 2

    arXiv 2025

  4. [4]

    Are we on the right way for evaluating large vision–language models? In Advances in Neural Information Processing Systems, 2024

    Lin Chen, Jinsong Li, Xiaoyi Dong, et al. Are we on the right way for evaluating large vision–language models? In Advances in Neural Information Processing Systems, 2024. 3

    2024

  5. [5]

    Internvl: Scal- ing up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, et al. Internvl: Scal- ing up vision foundation models and aligning for generic visual-linguistic tasks. InIEEE Conference on Computer Vi- sion and Pattern Recognition, pages 24185–24198, 2024. 2, 4

    2024

  6. [6]

    RobustBench: a standardized adversarial ro- bustness benchmark

    Francesco Croce, Maksym Andriushchenko, Vikash Se- hwag, et al. RobustBench: a standardized adversarial ro- bustness benchmark. InThirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2021. 2

    2021

  7. [7]

    On the robustness of large multimodal models against image adversarial attacks

    Xuanming Cui, Alejandro Aparcedo, Young Kyun Jang, et al. On the robustness of large multimodal models against image adversarial attacks. InIEEE Conference on Computer Vision and Pattern Recognition, pages 24625–24634, 2024. 2

    2024

  8. [8]

    Molmo and pixmo: Open weights and open data for state-of-the-art vision-language models

    Matt Deitke, Christopher Clark, Sangho Lee, et al. Molmo and pixmo: Open weights and open data for state-of-the-art vision-language models. InIEEE Conference on Computer Vision and Pattern Recognition, pages 91–104, 2025. 2, 5, 3

    2025

  9. [9]

    Scalable vi- sion language model training via high quality data curation

    Hongyuan Dong, Zijian Kang, Weijie Yin, et al. Scalable vi- sion language model training via high quality data curation. InProceedings of the Annual Meeting of the Association for Computational Linguistics, pages 1595–1612, 2025. 2, 5, 3

    2025

  10. [10]

    An image is worth 16x16 words: Transformers for im- age recognition at scale.International Conference on Learn- ing Representations, 2021

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, et al. An image is worth 16x16 words: Transformers for im- age recognition at scale.International Conference on Learn- ing Representations, 2021. 1

    2021

  11. [11]

    VLMEvalKit : An open-source toolkit for evaluating large multi-modality models

    Haodong Duan, Junming Yang, Yuxuan Qiao, et al. VLMEvalKit : An open-source toolkit for evaluating large multi-modality models. InACM International Conference on Multimedia, pages 11198–11201, 2024. 5

    2024

  12. [12]

    Tracing the ori- gin of adversarial attack for forensic investigation and de- terrence

    Han Fang, Jiyi Zhang, Yupeng Qiu, et al. Tracing the ori- gin of adversarial attack for forensic investigation and de- terrence. InIEEE International Conference on Computer Vision, pages 4335–4344, 2023. 2

    2023

  13. [13]

    The best defense is a good of- fense: Adversarial augmentation against adversarial attacks

    Iuri Frosio and Jan Kautz. The best defense is a good of- fense: Adversarial augmentation against adversarial attacks. InIEEE conference on computer vision and pattern recogni- tion, pages 4067–4076, 2023. 2

    2023

  14. [14]

    Benchmarking neu- ral network robustness to common corruptions and pertur- bations.International Conference on Learning Representa- tions, 2019

    Dan Hendrycks and Thomas Dietterich. Benchmarking neu- ral network robustness to common corruptions and pertur- bations.International Conference on Learning Representa- tions, 2019. 1, 2, 3

    2019

  15. [15]

    The many faces of robustness: A critical analysis of out-of-distribution generalization

    Dan Hendrycks, Steven Basart, Norman Mu, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. InIEEE international conference on com- puter vision, pages 8340–8349, 2021. 2

    2021

  16. [16]

    Visual robustness benchmark for visual question answering (VQA)

    Md Farhan Ishmam, Ishmam Tashdeed, Talukder Asir Saa- dat, et al. Visual robustness benchmark for visual question answering (VQA). InIEEE Winter Conference on Applica- tions of Computer Vision, pages 6623–6633, 2025. 2

    2025

  17. [17]

    Building and better understanding vision-language models: insights and future directions.CoRR, abs/2408.12637, 2024

    Hugo Laurenc ¸on, Andr ´es Marafioti, Victor Sanh, et al. Building and better understanding vision-language models: insights and future directions.CoRR, abs/2408.12637, 2024. 2, 5, 7, 8, 3

    Pith/arXiv arXiv 2024

  18. [18]

    SEED-bench-2-plus: Benchmarking multimodal large language models with text- rich visual comprehension

    Bohao Li, Yuying Ge, Yi Chen, et al. SEED-bench-2-plus: Benchmarking multimodal large language models with text- rich visual comprehension. InIEEE Conference on Com- puter Vision and Pattern Recognition, pages 13299–13306,

  19. [19]

    R-Bench: Are your large multimodal model robust to real-world cor- ruptions?CoRR, abs/2410.05474, 2024

    Chunyi Li, Jianbo Zhang, Zicheng Zhang, et al. R-Bench: Are your large multimodal model robust to real-world cor- ruptions?CoRR, abs/2410.05474, 2024. 1, 2, 5, 8

    Pith/arXiv arXiv 2024

  20. [20]

    OODRobust- Bench: A benchmark and large-scale analysis of adversar- ial robustness under distribution shift

    Lin Li, Yifei Wang, Chawin Sitawarin, et al. OODRobust- Bench: A benchmark and large-scale analysis of adversar- ial robustness under distribution shift. InInternational Con- ference on Machine Learning, pages 28830–28869. PMLR,

  21. [21]

    Sibling-attack: Re- thinking transferable adversarial attacks against face recog- nition

    Zexin Li, Bangjie Yin, Taiping Yao, et al. Sibling-attack: Re- thinking transferable adversarial attacks against face recog- nition. InIEEE Conference on Computer Vision and Pattern Recognition, pages 24626–24637, 2023. 2

    2023

  22. [22]

    Monkey: Image resolution and text label are important things for large multi- modal models

    Zhang Li, Biao Yang, Qiang Liu, et al. Monkey: Image resolution and text label are important things for large multi- modal models. InIEEE Conference on Computer Vision and Pattern Recognition, pages 26763–26773, 2024. 2, 5, 3

    2024

  23. [23]

    TruthfulQA: Measuring how models mimic human falsehoods

    Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. InPro- ceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pages 3214–3252, 2022. 2

    2022

  24. [24]

    MM- bench: Is your multi-modal model an all-around player? In European Conference on Computer Vision, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, et al. MM- bench: Is your multi-modal model an all-around player? In European Conference on Computer Vision, pages 216–233. Springer, 2024. 3

    2024

  25. [25]

    Ovis: Structural embedding alignment for multimodal large language model

    Shiyin Lu, Yang Li, Qing-Guo Chen, et al. Ovis: Structural embedding alignment for multimodal large language model. CoRR, abs/2405.20797, 2024. 2, 4

    Pith/arXiv arXiv 2024

  26. [26]

    Jailbreakv: A benchmark for assessing the robustness of multimodal large language models against jailbreak attacks

    Weidi Luo, Siyuan Ma, Xiaogeng Liu, et al. Jailbreakv: A benchmark for assessing the robustness of multimodal large language models against jailbreak attacks. InFirst Confer- ence on Language Modeling, 2024. 2

    2024

  27. [27]

    M87 Labs

    Inc. M87 Labs. Moondream. Inhttps://moondream.ai, 2025. 2, 4 9

    2025

  28. [28]

    Falcon2-11b technical report.CoRR, abs/2407.14885, 2024

    Quentin Malartic, Nilabhra Roy Chowdhury, Ruxandra Co- jocaru, et al. Falcon2-11b technical report.CoRR, abs/2407.14885, 2024. 2, 5, 8, 3

    Pith/arXiv arXiv 2024

  29. [29]

    Adver- sarial pruning: A survey and benchmark of pruning methods for adversarial robustness.Pattern Recognition, 168:111788,

    Giorgio Piras, Maura Pintor, Ambra Demontis, et al. Adver- sarial pruning: A survey and benchmark of pruning methods for adversarial robustness.Pattern Recognition, 168:111788,

  30. [30]

    Do CIFAR-10 classifiers generalize to CIFAR-10?CoRR, abs/1806.00451, 2018

    Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do CIFAR-10 classifiers generalize to CIFAR-10?CoRR, abs/1806.00451, 2018. 1

    Pith/arXiv arXiv 2018

  31. [31]

    Sentence-BERT: Sen- tence embeddings using siamese bert-networks

    Nils Reimers and Iryna Gurevych. Sentence-BERT: Sen- tence embeddings using siamese bert-networks. InProceed- ings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing, pages 3980–3990, 2019. 1

    2019

  32. [32]

    Robustness analysis of video-language models against visual and language perturbations

    Madeline Chantry Schiappa, Shruti Vyas, Hamid Palangi, et al. Robustness analysis of video-language models against visual and language perturbations. InThirty-sixth Confer- ence on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. 2

    2022

  33. [33]

    Eagle: Explor- ing the design space for multimodal llms with mixture of encoders

    Min Shi, Fuxiao Liu, Shihao Wang, et al. Eagle: Explor- ing the design space for multimodal llms with mixture of encoders. InInternational Conference on Learning Repre- sentations, 2025. 2, 5, 3

    2025

  34. [34]

    LEGO- Puzzles: How good are mllms at multi-step spatial reason- ing?CoRR, abs/2503.19990, 2025

    Kexian Tang, Junyao Gao, Yanhong Zeng, et al. LEGO- Puzzles: How good are mllms at multi-step spatial reason- ing?CoRR, abs/2503.19990, 2025. 3

    Pith/arXiv arXiv 2025

  35. [35]

    Gemma 3 technical report.CoRR, abs/2503.19786, 2025

    Gemma Team, Aishwarya Kamath, Johan Ferret, et al. Gemma 3 technical report.CoRR, abs/2503.19786, 2025. 2, 5, 3

    Pith/arXiv arXiv 2025

  36. [36]

    Kimi-vl techni- cal report.CoRR, abs/2504.07491, 2025

    Kimi Team, Angang Du, Bohong Yin, et al. Kimi-vl techni- cal report.CoRR, abs/2504.07491, 2025. 2, 5, 3

    Pith/arXiv arXiv 2025

  37. [37]

    Analysing the robustness of vision-language-models to common corruptions.CoRR, abs/2504.13690, 2025

    Muhammad Usama, Syeda Aishah Asim, Syed Bilal Ali, et al. Analysing the robustness of vision-language-models to common corruptions.CoRR, abs/2504.13690, 2025. 1, 2

    Pith/arXiv arXiv 2025

  38. [38]

    Emu3: Next-token prediction is all you need.CoRR, abs/2409.18869, 2024

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, et al. Emu3: Next-token prediction is all you need.CoRR, abs/2409.18869, 2024. 2, 5, 6, 3

    Pith/arXiv arXiv 2024

  39. [39]

    Deepseek- vl2: Mixture-of-experts vision-language models for ad- vanced multimodal understanding.CoRR, abs/2412.10302,

    Zhiyu Wu, Xiaokang Chen, Zizheng Pan, et al. Deepseek- vl2: Mixture-of-experts vision-language models for ad- vanced multimodal understanding.CoRR, abs/2412.10302,

    Pith/arXiv arXiv

  40. [40]

    Demystify- ing the visual quality paradox in multimodal large language models.CoRR, abs/2506.15645, 2025

    Shuo Xing, Lanqing Guo, Hongyuan Hua, et al. Demystify- ing the visual quality paradox in multimodal large language models.CoRR, abs/2506.15645, 2025. 1, 2, 7

    Pith/arXiv arXiv 2025

  41. [41]

    mPLUG-owl3: Towards long image-sequence understanding in multi-modal large language models

    Jiabo Ye, Haiyang Xu, Haowei Liu, et al. mPLUG-owl3: Towards long image-sequence understanding in multi-modal large language models. InInternational Conference on Learning Representations, 2025. 2, 5, 6, 3

    2025

  42. [42]

    MMT- Bench: A comprehensive multimodal benchmark for eval- uating large vision-language models towards multitask agi

    Kaining Ying, Fanqing Meng, Jin Wang, et al. MMT- Bench: A comprehensive multimodal benchmark for eval- uating large vision-language models towards multitask agi. InInternational Conference on Machine Learning, pages 57116–57198, 2024. 3

    2024

  43. [43]

    Yi-Fan Zhang, Huanyu Zhang, Haochen Tian, et al. MME- RealWorld: Could your multimodal llm challenge high- resolution real-world scenarios that are difficult for hu- mans?International Conference on Learning Representa- tions, 2025. 3

    2025

  44. [44]

    On evaluating adversarial robustness of large vision-language models

    Yunqing Zhao, Tianyu Pang, Chao Du, et al. On evaluating adversarial robustness of large vision-language models. In Thirty-seventh Conference on Neural Information Process- ing Systems, 2023. 2 10 Benchmarking Corruption Robustness of LVLMs: A Discriminative Benchmark and Robustness Alignment Metric Supplementary Material

    2023

  45. [45]

    Discriminative Samples Selection Diversity formulation.In Eq

    More about BENCH-C 8.1. Discriminative Samples Selection Diversity formulation.In Eq. 3, the termsD s encourage diversity of the selected subset in both visual and textual spaces. Herein, we provide a detailed description below. Letz m(q)∈R d denote the normalized semantic embed- ding of samplequnder modalitym∈ {img,text}, obtained from a pretrained encod...

    2000

  46. [46]

    A”, “B”, “C

    More about Experimental Results 9.1. Similarity-Based Metrics Collapse to Accuracy Herein, we demonstrate that for MCQ tasks, text similarity based metrics – which measure the average similarity be- tween model generated text and the ground-truth option – will collapse to accuracy. Consider an MCQ task withKoptions, where the model predicts a text respons...

  47. [47]

    lead to an unexpected overall increase in accuracy. Al- though these distortions seem to “improve” model perfor- mance in terms of accuracy, a closer inspection of shifts in internal prediction structure reveals such an improvement Table 2. Statistical comparison ofz-scorenormalized RAS and ∆Acc.across all corruption types under subtle quality difference ...