Pith. sign in

REVIEW 4 major objections 5 minor 4 cited by

MLLMs are Deeply Affected by Modality Bias

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Multimodal AI leans on language, shortchanges images

desk verdict A competent, well-structured position paper on modality bias in MLLMs; the five-factor taxonomy is the main value, but the case study lacks a text-only baseline and is illustrative rather than proof. read the letter →

arxiv 2505.18657 v1 pith:OTX3XVNG submitted 2025-05-24 cs.AI

Xu Zheng , Chenfei Liao , Yuqian Fu , Kaiyu Lei , Yuanhuiyi Lyu , Lutao Jiang , Bin Ren , Jialei Chen
show 10 more authors
Jiawen Wang Chengxin Li Linfeng Zhang Danda Pani Paudel Xuanjing Huang Yu-Gang Jiang Nicu Sebe Dacheng Tao Luc Van Gool Xuming Hu
This is my paper · ORCID
classification cs.AI
keywords modalitybiasmultimodallargelanguagemodelspriorsvisualunder-utilizationMMMU-Provision-languagebenchmarkanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper takes the position that multimodal large language models (MLLMs) are deeply affected by modality bias: their answers are often driven by language priors rather than by visual input. To support the position, the authors run a case study on the MMMU-Pro benchmark with Qwen2.5-VL models, replacing images with white, black, or noisy frames and comparing accuracy with complete inputs. They attribute the bias to three main factors—dataset imbalances, asymmetric backbone capabilities, and training objectives—and outline a research roadmap for measuring, avoiding, and reducing it. The stakes are practical: if the claim is right, current MLLMs will keep failing when images are missing or misleading, and debiasing the visual side becomes central to building trustworthy multimodal systems.

What carries the argument

The paper's formal object is the modality imbalance ratio $\Delta_{\text{modality}} = C(M_{\text{dominant}})/C(M_{\text{underutilized}})$, a relative measure of how much one modality's contribution dominates another; the authors note it is for illustration rather than for direct calculation. The load-bearing mechanism is the case study design: replacing real images with white, black, or noisy frames while keeping the question text unchanged, then comparing accuracy and prediction consistency across complete, text-only, and image-only conditions. High accuracy and consistency in text-only conditions, combined with low consistency in image-only conditions, is what carries the claim that language priors dominate.

What would settle it

Take a set of MMMU-Pro questions verified to require the image, so that text-only accuracy is at chance, blank the images for a range of MLLMs, and check whether accuracy collapses toward chance; if a model keeps complete-input accuracy on visually dependent items despite blank images, the paper's claim of deep modality bias would be falsified.

Watch

Extended reading notes

Core claim

The central discovery this paper asserts is that MLLMs are deeply affected by modality bias: given an image-text question, the model's answer is often determined by the language component, not by what the image shows. Evidence is the MMMU-Pro case study, where Qwen2.5-VL-7B's accuracy on Standard 4 falls from 48.32% with real images to 28.73% with white images, while text-only inputs match complete inputs on 56.53% of samples. The paper reads these results as showing that textual input alone is a strong predictor and that the image is underused, supporting its three-factor explanation of the bias.

Load-bearing premise

The claim rests on assuming that blanking the image cleanly isolates how much the model relies on vision; if the model reacts to the corruption itself rather than to the absence of visual information, the measured accuracy drops are not a clean measure of modality bias, and the argument also assumes that Qwen2.5-VL on MMMU-Pro represents MLLMs generally.

Editorial extensions

If this is right

  • If MLLMs mostly rely on text, then current benchmarks that allow language-only shortcuts overestimate how well models actually understand images.
  • Blank-image checks should become a standard, cheap robustness probe for every MLLM evaluation.
  • Debiasing efforts need to act on data balance, backbone capacity, and training objectives together, since the paper ties the bias to all three.
  • Real deployments will see failures when images are missing, blurry, or contradictory to the text, exactly the scenarios the blanked-image experiments simulate.
  • Preference optimization on modality-biased samples and visually dependent datasets are the paper's concrete routes for mitigation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The white/black/noise probe likely shows a lower bound on modality bias, because a model can detect that a blank frame is corrupted and may hedge; naturally misleading images could trigger even stronger language-prior reliance.
  • Scaling may worsen the bias: the 32B model shows larger absolute drops than the 7B model in several conditions, hinting that bigger language backbones inject stronger priors.
  • A direct extension would apply the same blank-image protocol to audio-text and video-text MLLMs to see whether the bias generalizes beyond vision-language pairs.
  • Turning the illustrative ratio $\Delta_{\text{modality}}$ into a practical metric would require estimating per-modality contributions on real inputs, which the paper leaves open.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This position paper argues that multimodal large language models (MLLMs) are deeply affected by modality bias, meaning their outputs are often driven by language priors rather than by visual input. It defines modality bias (Sec. 2), organizes prior work into a roadmap with three directions (prove, solve via datasets, solve via methods; Sec. 3.1), and attributes the bias to three primary factors (dataset imbalances, asymmetric modal backbone capabilities, training objectives) plus two secondary factors (convergence rates and modal interactions; Sec. 3.2). To support the position, the authors run a case study on MMMU-Pro with Qwen2.5-VL-7B and 32B, replacing images with white, black, or noisy images and measuring accuracy and prediction consistency (Sec. 4, Tables 1 and 2). They then discuss current mitigation strategies and future directions (Sec. 5). The paper is explicitly a position paper and leans substantially on previously published benchmarks and methods.

Significance. If the central position is accepted, the paper provides a useful synthesis of the modality-bias literature and a structured research agenda for a community that is actively working on robust multimodal evaluation. The paper's strengths include its broad coverage of existing benchmarks (MMStar, MMMU-Pro, VLind-Bench, CMM, MMVP), its explicit separation of an illustrative mathematical definition from a computational one (Footnote 3), and its use of public models and a public benchmark in the case study, which improves reproducibility in principle. However, the case study as reported does not cleanly establish the headline claim about language priors, and the internal mapping from consistency numbers to the three named factors is not justified by the experimental design. The paper is therefore more valuable as a survey and roadmap than as a standalone empirical demonstration.

major comments (4)
  1. [Sec. 4, Table 1] The case study lacks a text-only accuracy baseline. The central claim is that MLLM outputs are often determined by language priors, but Table 1 only compares complete image-text inputs against text-with-corrupted-image and image-only conditions. If language priors dominate, then replacing the image with white, black, or noise should leave accuracy close to the text-only accuracy; if instead the model actively integrates the corrupted image as misleading evidence, the observed drops (e.g., 48.32 to 28.73 for Qwen2.5-VL-7B Direct, Standard 4, white) are also consistent with the data. Since Table 2 already appears to have been computed from text-only predictions, the text-only accuracy is readily available and should be reported in Table 1. Without it, the case study conflates corrupted-image sensitivity with language-prior reliance.
  2. [Sec. 4, Table 2] Consistency between complete and text-only outputs is not evidence of reliance on language priors, because agreement does not imply correctness. The text states that 'over half of the samples show consistency' and concludes that 'textual information alone is a strong foundation,' but MMMU-Pro is explicitly designed so that many questions require visual information. A model could consistently choose the same incorrect answer from text alone, which would show high consistency but would not demonstrate that the language prior is a sound or dominant basis for the response. The paper should report text-only accuracy and, ideally, the accuracy of complete inputs conditioned on whether the text-only prediction was correct, to distinguish between 'language prior drives a correct answer' and 'language prior drives a confidently wrong answer.'
  3. [Sec. 4, Findings] The mapping from each finding to a specific factor is not justified by the experiment. For example, 'Lower Consistency with Image-only Inputs' is attributed to dataset imbalances, but the image-only condition uses a different basic prompt ('Based on the provided images, please answer the question') and may also reflect the model's visual backbone limitations or the absence of the question text embedded in the image. Similarly, 'Inconsistency between Text-only and Image-only Inputs' is attributed to training objectives without any manipulation of training objectives. The three factors are not independently varied, so the case study is at best illustrative; the paper should either soften the causal language or acknowledge that the factor attribution is speculative.
  4. [Sec. 4, Tables 1 and 2] The case study has no error bars, multiple runs, or seeds, and it only uses two checkpoints from a single model family (Qwen2.5-VL-7B and 32B). The headline claim is about MLLMs in general, but the reported evidence is limited to one family on one benchmark. The paper should either add repeated evaluations with variance estimates and another model family (e.g., an InternVL or LLaVA variant), or explicitly restrict the claim to 'Qwen2.5-VL on MMMU-Pro' and present the broader position as a synthesis of the prior benchmarks cited in Sec. 3.1.
minor comments (5)
  1. [Sec. 1.3] The phrase 'we analysis the key factors' should be 'we analyze the key factors'.
  2. [Sec. 6] The sentence 'which is proved and explored by both the theoretical analysis and case study' is grammatically awkward and should be revised; the analysis in Sec. 2 is explicitly illustrative, so 'proved' is too strong.
  3. [Sec. 3.1] There are missing spaces in several citations, e.g., 'Parket al.' and 'Lenget al.'; a careful proofread of inline author names is needed.
  4. [Table 1] The column headers 'Text /w white Image', 'Text w/ white Image', and 'w/ B-P' are inconsistently formatted; using the same notation throughout would improve readability.
  5. [Sec. 4] The numbers cited in the findings paragraph (e.g., 27.17%, 56.53%, 26.76%) are not all present in the immediate text before Table 2, so a pointer to the relevant table rows would help the reader map each finding to the data.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the formal measure is explicitly illustrative and the case study is an external-benchmark measurement.

full rationale

This is a position paper whose central claim is supported by cited external work and by a direct case study, not by a derivation from its own definitions. The only formal definitions, Eq. (1) and Eq. (2) in Sec. 2, are explicitly labeled 'for better illustration of modality bias, not for calculating' (Footnote 3), so no result is derived from them by construction. The case study in Sec. 4 uses the public MMMU-Pro benchmark and the public Qwen2.5-VL model with no fitted parameters; the accuracy drops and consistency percentages are reported measurements rather than outputs of the proposed formalism. The cited evidence for the central position, such as Zhang et al. [24] and Leng et al. [51], is external to the present author set, and self-citations such as [33], [38], [39], [40], and [50] appear only as contextual support in survey-style passages, not as load-bearing justifications for the headline claim. The absence of a pure text-only accuracy baseline in Table 1 is a possible experimental-control concern for how strongly the drops indicate language priors, but that is a validity issue, not circularity. Accordingly, the derivation chain is self-contained and no circular step is present.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No free parameters are fitted; the formal definitions are explicitly illustrative. The case study uses a public benchmark and model. The listed axioms are the domain assumptions on which the measurement logic rests.

assumptions (5)
  • domain assumption Modality contributions C(M_i) are additive and well-defined, so that C_total = sum of C(M_i) (Eq. 1).
    The formal definition of modality bias assumes each modality contributes a scalar quantity that sums meaningfully. The paper itself states in Footnote 3 that these definitions are for illustration, not calculation.
  • domain assumption Substituting the image with white, black, or noise does not change the model's behavior except through loss of visual information.
    The case study in Sec. 4 relies on this to attribute accuracy drops to modality loss rather than to distribution shift or prompt effects.
  • domain assumption Prediction consistency between complete, text-only, and image-only inputs measures the extent of modality reliance.
    Table 2 interprets consistency percentages as evidence for text dominance and objective-induced shortcuts.
  • domain assumption The performance of Qwen2.5-VL on MMMU-Pro is representative of MLLMs broadly.
    Sec. 4 generalizes from one model family to 'MLLMs' when stating the position.
  • domain assumption Prior benchmark results (MMVP, VLind-Bench, CMM) are accepted as valid evidence for modality bias.
    The position leans on these external results to support the general claim beyond the paper's own case study.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MLLMs are Deeply Affected by Modality Bias." pith.science (2026). https://pith.science/paper/OTX3XVNG

@misc{pith2026250518657,
  author       = {Pith},
  title        = {Pith review of: MLLMs are Deeply Affected by Modality Bias},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OTX3XVNG}},
  note         = {Machine review of arXiv:2505.18657}
}
read the original abstract

Recent advances in Multimodal Large Language Models (MLLMs) have shown promising results in integrating diverse modalities such as texts and images. MLLMs are heavily influenced by modality bias, often relying on language while under-utilizing other modalities like visual inputs. This position paper argues that MLLMs are deeply affected by modality bias. Firstly, we diagnose the current state of modality bias, highlighting its manifestations across various tasks. Secondly, we propose a systematic research road-map related to modality bias in MLLMs. Thirdly, we identify key factors of modality bias in MLLMs and offer actionable suggestions for future research to mitigate it. To substantiate these findings, we conduct experiments that demonstrate the influence of each factor: 1. Data Characteristics: Language data is compact and abstract, while visual data is redundant and complex, creating an inherent imbalance in learning dynamics. 2. Imbalanced Backbone Capabilities: The dominance of pretrained language models in MLLMs leads to overreliance on language and neglect of visual information. 3. Training Objectives: Current objectives often fail to promote balanced cross-modal alignment, resulting in shortcut learning biased toward language. These findings highlight the need for balanced training strategies and model architectures to better integrate multiple modalities in MLLMs. We call for interdisciplinary efforts to tackle these challenges and drive innovation in MLLM research. Our work provides a fresh perspective on modality bias in MLLMs and offers insights for developing more robust and generalizable multimodal systems-advancing progress toward Artificial General Intelligence.

Figures

Figures reproduced from arXiv: 2505.18657 by the authors.

Figure 1
Figure 1. The big picture of our position on modality bias in MLLMs. (a) We define modality bias in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Further definition of modality bias and three potential results. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Case study for exploring modality bias in MLLMs. Dataset: MMMU-Pro, MLLM: [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Targeted solutions of modality bias in MLLMs, including current works and future directions. ⃝2 Turn the focus of MLLMs from textual information into visual information: Considering the ignorance of visual modality in the inference of MLLMs, it is an intuitive approach…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Allocation Before Ranking: Decoupled Token Compression for OmniLLMs

    cs.AI 2026-08 conditional novelty 6.0 of 10

    MACER decouples cross-modal token-budget allocation from within-modality token ranking and improves accuracy over shared top-K compression in Qwen2.5-Omni and OmniVinci models.

  2. Can Large Multimodal Models Actively Recognize Faulty Inputs? A Systematic Evaluation Framework of Their Input Scrutiny Ability

    cs.CV 2025-08 unverdicted novelty 6.0 of 10

    Large multimodal models mostly fail to proactively detect flawed textual premises, and their performance depends on error type and on how they weight text versus images.

  3. When Vision Overrides Language: Evaluating and Mitigating Counterfactual Failures in VLAs

    cs.CV 2026-02 conditional novelty 5.0 of 10

    VLAs fail most counterfactual instructions because vision shortcuts dominate language; the new LIBERO-CF benchmark quantifies this, and CAG, an inference-time action mixer, improves grounding and success.

  4. Omnidirectional Spatial Modeling from Correlated Panoramas

    cs.CV 2025-09 conditional novelty 5.0 of 10

    The authors create a cross-frame panoramic VQA benchmark from 3D scene data and show that GRPO fine-tuning of Qwen2.5-VL raises its score on that benchmark.

Reference graph

Works this paper leans on

89 extracted references · 45 canonical work pages · cited by 4 Pith papers

  1. [1]

    Qwen2. 5-vl technical report,

    S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang,et al., “Qwen2. 5-vl technical report,”arXiv preprint arXiv:2502.13923, 2025

    arXiv 2025

  2. [2]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution,

    P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge,et al., “Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution,” arXiv preprint arXiv:2409.12191, 2024

    arXiv 2024

  3. [3]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models,

    J. Zhu, W. Wang, Z. Chen, Z. Liu, S. Ye, L. Gu, Y . Duan, H. Tian, W. Su, J. Shao,et al., “Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models,”arXiv preprint arXiv:2504.10479, 2025

    arXiv 2025

  4. [4]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling,

    Z. Chen, W. Wang, Y . Cao, Y . Liu, Z. Gao, E. Cui, J. Zhu, S. Ye, H. Tian, Z. Liu,et al., “Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling,”arXiv preprint arXiv:2412.05271, 2024

    arXiv 2024

  5. [5]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,

    P. Georgiev, V . I. Lei, R. Burnell, L. Bai, A. Gulati, G. Tanzer, D. Vincent, Z. Pan, S. Wang, et al., “Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,” arXiv preprint arXiv:2403.05530, 2024

    arXiv 2024

  6. [6]

    Gpt-4o system card,

    A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford,et al., “Gpt-4o system card,”arXiv preprint arXiv:2410.21276, 2024

    arXiv 2024

  7. [7]

    Tactile sensing—from humans to humanoids,

    R. S. Dahiya, G. Metta, M. Valle, and G. Sandini, “Tactile sensing—from humans to humanoids,” IEEE transactions on robotics, vol. 26, no. 1, pp. 1–20, 2009

    2009

  8. [8]

    Novel tactile sensor technology and smart tactile sensing systems: A review,

    L. Zou, C. Ge, Z. J. Wang, E. Cretu, and X. Li, “Novel tactile sensor technology and smart tactile sensing systems: A review,”Sensors, vol. 17, no. 11, p. 2653, 2017

    2017

Show all 89 references
  1. [9]

    Recent progress in technologies for tactile sensors,

    C. Chi, X. Sun, N. Xue, T. Li, and C. Liu, “Recent progress in technologies for tactile sensors,” Sensors, vol. 18, no. 4, p. 948, 2018

    2018

  2. [10]

    Event-based vision: A survey,

    G. Gallego, T. Delbrück, G. Orchard, C. Bartolozzi, B. Taba, A. Censi, S. Leutenegger, A. J. Davison, J. Conradt, K. Daniilidis,et al., “Event-based vision: A survey,”IEEE transactions on pattern analysis and machine intelligence, vol. 44, no. 1, pp. 154–180, 2020

    2020

  3. [11]

    Deep learn- ing for event-based vision: A comprehensive survey and benchmarks,

    X. Zheng, Y . Liu, Y . Lu, T. Hua, T. Pan, W. Zhang, D. Tao, and L. Wang, “Deep learn- ing for event-based vision: A comprehensive survey and benchmarks,”arXiv preprint arXiv:2302.08890, 2023

    2023 arXiv

  4. [12]

    High speed and high dynamic range video with an event camera,

    H. Rebecq, R. Ranftl, V . Koltun, and D. Scaramuzza, “High speed and high dynamic range video with an event camera,”IEEE transactions on pattern analysis and machine intelligence, vol. 43, no. 6, pp. 1964–1980, 2019

    1964

  5. [13]

    360sfuda++: Towards source-free uda for panoramic segmentation by learning reliable category prototypes,

    X. Zheng, P. Y . Zhou, A. V . Vasilakos, and L. Wang, “360sfuda++: Towards source-free uda for panoramic segmentation by learning reliable category prototypes,”IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    2024

  6. [14]

    Semantics distortion and style matter: Towards source-free uda for panoramic segmentation,

    X. Zheng, P. Zhou, A. V . Vasilakos, and L. Wang, “Semantics distortion and style matter: Towards source-free uda for panoramic segmentation,” inProceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 27885–27895, 2024

    2024

  7. [15]

    Omnisam: Omnidirectional segment anything model for uda in panoramic semantic segmentation,

    D. Zhong, X. Zheng, C. Liao, Y . Lyu, J. Chen, S. Wu, L. Zhang, and X. Hu, “Omnisam: Omnidirectional segment anything model for uda in panoramic semantic segmentation,”arXiv preprint arXiv:2503.07098, 2025

    2025

  8. [16]

    Mmbench: Is your multi-modal model an all-around player?,

    Y . Liu, H. Duan, Y . Zhang, B. Li, S. Zhang, W. Zhao, Y . Yuan, J. Wang, C. He, Z. Liu,et al., “Mmbench: Is your multi-modal model an all-around player?,” inEuropean conference on computer vision, pp. 216–233, Springer, 2024. 10

    2024

  9. [17]

    Mmbench-video: A long- form multi-shot benchmark for holistic video understanding,

    X. Fang, K. Mao, H. Duan, X. Zhao, Y . Li, D. Lin, and K. Chen, “Mmbench-video: A long- form multi-shot benchmark for holistic video understanding,”Advances in Neural Information Processing Systems, vol. 37, pp. 89098–89124, 2024

    2024

  10. [18]

    Docvqa: A dataset for vqa on document images,

    M. Mathew, D. Karatzas, and C. Jawahar, “Docvqa: A dataset for vqa on document images,” in Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 2200– 2209, 2021

    2021

  11. [19]

    A survey on multimodal benchmarks: In the era of large ai models,

    L. Li, G. Chen, H. Shi, J. Xiao, and L. Chen, “A survey on multimodal benchmarks: In the era of large ai models,”arXiv preprint arXiv:2409.18142, 2024

    2024 arXiv

  12. [20]

    A survey on benchmarks of multimodal large language models,

    J. Li, W. Lu, H. Fei, M. Luo, M. Dai, M. Xia, Y . Jin, Z. Gan, D. Qi, C. Fu,et al., “A survey on benchmarks of multimodal large language models,”arXiv preprint arXiv:2408.08632, 2024

    2024 arXiv

  13. [21]

    Visual prompting in multimodal large language models: A survey,

    J. Wu, Z. Zhang, Y . Xia, X. Li, Z. Xia, A. Chang, T. Yu, S. Kim, R. A. Rossi, R. Zhang, et al., “Visual prompting in multimodal large language models: A survey,”arXiv preprint arXiv:2409.15310, 2024

    2024 arXiv

  14. [22]

    Survey of adversarial robustness in multimodal large language models,

    C. Jiang, Z. Wang, M. Dong, and J. Gui, “Survey of adversarial robustness in multimodal large language models,”arXiv preprint arXiv:2503.13962, 2025

    2025 arXiv

  15. [23]

    When continue learning meets multimodal large language model: A survey,

    Y . Huo and H. Tang, “When continue learning meets multimodal large language model: A survey,”arXiv preprint arXiv:2503.01887, 2025

    2025 arXiv

  16. [24]

    Debiasing multimodal large language models,

    Y .-F. Zhang, W. Yu, Q. Wen, X. Wang, Z. Zhang, L. Wang, R. Jin, and T. Tan, “Debiasing multimodal large language models,”arXiv preprint arXiv:2403.05262, 2024

    2024 arXiv

  17. [25]

    Assessing modality bias in video question answering benchmarks with multimodal large language models,

    J. Park, K. J. Jang, B. Alasaly, S. Mopidevi, A. Zolensky, E. Eaton, I. Lee, and K. Johnson, “Assessing modality bias in video question answering benchmarks with multimodal large language models,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 39, pp. 19...

    2025

  18. [26]

    Eyes wide shut? exploring the visual shortcomings of multimodal llms,

    S. Tong, Z. Liu, Y . Zhai, Y . Ma, Y . LeCun, and S. Xie, “Eyes wide shut? exploring the visual shortcomings of multimodal llms,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9568–9578, 2024

    2024

  19. [27]

    Are we on the right way for evaluating large vision-language models?,

    L. Chen, J. Li, X. Dong, P. Zhang, Y . Zang, Z. Chen, H. Duan, J. Wang, Y . Qiao, D. Lin, et al., “Are we on the right way for evaluating large vision-language models?,”arXiv preprint arXiv:2403.20330, 2024

    2024 arXiv

  20. [28]

    Quantifying and mitigating unimodal biases in multimodal large language models: A causal perspective,

    M. Chen, Y . Cao, Y . Zhang, and C. Lu, “Quantifying and mitigating unimodal biases in multimodal large language models: A causal perspective,”arXiv preprint arXiv:2403.18346, 2024

    2024 arXiv

  21. [29]

    Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark,

    X. Yue, T. Zheng, Y . Ni, Y . Wang, K. Zhang, S. Tong, Y . Sun, B. Yu, G. Zhang, H. Sun,et al., “Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark,”arXiv preprint arXiv:2409.02813, 2024

    2024 arXiv

  22. [30]

    Multimodal learning with transformers: A survey,

    P. Xu, X. Zhu, and D. A. Clifton, “Multimodal learning with transformers: A survey,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 10, pp. 12113–12132, 2023

    2023

  23. [31]

    A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets,

    K. Bayoudh, R. Knani, F. Hamdaoui, and A. Mtibaa, “A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets,”The Visual Computer, vol. 38, no. 8, pp. 2939–2970, 2022

    2022

  24. [32]

    A review on methods and applications in multimodal deep learning,

    S. Jabeen, X. Li, M. S. Amin, O. Bourahla, S. Li, and A. Jabbar, “A review on methods and applications in multimodal deep learning,”ACM Transactions on Multimedia Computing, Communications and Applications, vol. 19, no. 2s, pp. 1–41, 2023

    2023

  25. [33]

    Learning modality-agnostic representation for semantic segmentation from any modalities,

    X. Zheng, Y . Lyu, and L. Wang, “Learning modality-agnostic representation for semantic segmentation from any modalities,” inEuropean Conference on Computer Vision, pp. 146–165, Springer, 2024. 11

    2024

  26. [34]

    Memorysam: Memorize modalities and semantics with segment anything model 2 for multi-modal semantic segmentation,

    C. Liao, X. Zheng, Y . Lyu, H. Xue, Y . Cao, J. Wang, K. Yang, and X. Hu, “Memorysam: Memorize modalities and semantics with segment anything model 2 for multi-modal semantic segmentation,”arXiv preprint arXiv:2503.06700, 2025

    2025 arXiv

  27. [35]

    Cafuser: Condition-aware multimodal fusion for robust semantic perception of driving scenes,

    T. Brödermann, C. Sakaridis, Y . Fu, and L. Van Gool, “Cafuser: Condition-aware multimodal fusion for robust semantic perception of driving scenes,”IEEE Robotics and Automation Letters, 2025

    2025

  28. [36]

    Multimodality represen- tation learning: A survey on evolution, pretraining and its applications,

    M. A. Manzoor, S. Albarri, Z. Xian, Z. Meng, P. Nakov, and S. Liang, “Multimodality represen- tation learning: A survey on evolution, pretraining and its applications,”ACM Transactions on Multimedia Computing, Communications and Applications, vol. 20, no. 3, pp. 1–34, 2023

    2023

  29. [37]

    Enhancing multimodal cooperation via sample-level modality valuation,

    Y . Wei, R. Feng, Z. Wang, and D. Hu, “Enhancing multimodal cooperation via sample-level modality valuation,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 27338–27347, 2024

    2024

  30. [38]

    Centering the value of every modality: Towards efficient and resilient modality-agnostic semantic segmentation,

    X. Zheng, Y . Lyu, J. Zhou, and L. Wang, “Centering the value of every modality: Towards efficient and resilient modality-agnostic semantic segmentation,” inEuropean Conference on Computer Vision, pp. 192–212, Springer, 2024

    2024

  31. [39]

    Reducing unimodal bias in multi-modal semantic segmentation with multi-scale functional entropy regularization,

    X. Zheng, Y . Lyu, L. Jiang, D. P. Paudel, L. Van Gool, and X. Hu, “Reducing unimodal bias in multi-modal semantic segmentation with multi-scale functional entropy regularization,”arXiv preprint arXiv:2505.06635, 2025

    2025 arXiv

  32. [40]

    Learn- ing robust anymodal segmentor with unimodal and cross-modal distillation,

    X. Zheng, H. Xue, J. Chen, Y . Yan, L. Jiang, Y . Lyu, K. Yang, L. Zhang, and X. Hu, “Learn- ing robust anymodal segmentor with unimodal and cross-modal distillation,”arXiv preprint arXiv:2411.17141, 2024

    2024 arXiv

  33. [41]

    Balanced multimodal learning via on-the-fly gradient modulation,

    X. Peng, Y . Wei, A. Deng, D. Wang, and D. Hu, “Balanced multimodal learning via on-the-fly gradient modulation,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8238–8247, 2022

    2022

  34. [42]

    Clip the bias: How useful is balancing data in multimodal learning?,

    I. Alabdulmohsin, X. Wang, A. P. Steiner, P. Goyal, A. D’Amour, and X. Zhai, “Clip the bias: How useful is balancing data in multimodal learning?,” inThe Twelfth International Conference on Learning Representations

  35. [43]

    Learning transferable visual models from natural language supervi- sion,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark,et al., “Learning transferable visual models from natural language supervi- sion,” inInternational conference on machine learning, pp. 8748–8763, PmLR, 2021

    2021

  36. [44]

    Clippo: Image-and-language understanding from pixels only,

    M. Tschannen, B. Mustafa, and N. Houlsby, “Clippo: Image-and-language understanding from pixels only,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11006–11017, 2023

    2023

  37. [45]

    Clip-kd: An empirical study of clip model distillation,

    C. Yang, Z. An, L. Huang, J. Bi, X. Yu, H. Yang, B. Diao, and Y . Xu, “Clip-kd: An empirical study of clip model distillation,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15952–15962, 2024

    2024

  38. [46]

    Tinyclip: Clip distillation via affinity mimicking and weight inheritance,

    K. Wu, H. Peng, Z. Zhou, B. Xiao, M. Liu, L. Yuan, H. Xuan, M. Valenzuela, X. S. Chen, X. Wang,et al., “Tinyclip: Clip distillation via affinity mimicking and weight inheritance,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 21970–21980, 2023

    2023

  39. [47]

    An inverse scaling law for clip training,

    X. Li, Z. Wang, and C. Xie, “An inverse scaling law for clip training,”Advances in Neural Information Processing Systems, vol. 36, pp. 49068–49087, 2023

    2023

  40. [48]

    Balancebenchmark: A survey for multimodal imbalance learning,

    S. Xu, M. Cui, C. Huang, H. Wang, and D. Hu, “Balancebenchmark: A survey for multimodal imbalance learning,”arXiv preprint arXiv:2502.10816, 2025

    2025 arXiv

  41. [49]

    Facilitating multimodal classification via dynamically learning modality gap,

    Y . Yang, F. Wan, Q.-Y . Jiang, and Y . Xu, “Facilitating multimodal classification via dynamically learning modality gap,”Advances in Neural Information Processing Systems, vol. 37, pp. 62108– 62122, 2024. 12

    2024

  42. [50]

    Benchmarking multi-modal semantic segmentation under sensor failures: Missing and noisy modality robustness,

    C. Liao, K. Lei, X. Zheng, J. Moon, Z. Wang, Y . Wang, D. P. Paudel, L. Van Gool, and X. Hu, “Benchmarking multi-modal semantic segmentation under sensor failures: Missing and noisy modality robustness,”arXiv preprint arXiv:2503.18445, 2025

    2025 arXiv

  43. [51]

    The curse of multi-modalities: Evaluating hallucinations of large multimodal models across language, visual, and audio,

    S. Leng, Y . Xing, Z. Cheng, Y . Zhou, H. Zhang, X. Li, D. Zhao, S. Lu, C. Miao, and L. Bing, “The curse of multi-modalities: Evaluating hallucinations of large multimodal models across language, visual, and audio,”arXiv preprint arXiv:2410.12787, 2024

    arXiv

  44. [52]

    On modality bias recognition and reduction,

    Y . Guo, L. Nie, H. Cheng, Z. Cheng, M. Kankanhalli, and A. Del Bimbo, “On modality bias recognition and reduction,”ACM Transactions on Multimedia Computing, Communications and Applications, vol. 19, no. 3, pp. 1–22, 2023

    2023

  45. [53]

    Cross modality bias in visual question answering: A causal view with possible worlds vqa,

    A. V osoughi, S. Deng, S. Zhang, Y . Tian, C. Xu, and J. Luo, “Cross modality bias in visual question answering: A causal view with possible worlds vqa,”IEEE Transactions on Multimedia, 2024

    2024

  46. [54]

    Counterfactual vqa: A cause-effect look at language bias,

    Y . Niu, K. Tang, H. Zhang, Z. Lu, X.-S. Hua, and J.-R. Wen, “Counterfactual vqa: A cause-effect look at language bias,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12700–12710, 2021

    2021

  47. [55]

    Removing bias in multi-modal classifiers: Regularization by maximizing functional entropies,

    I. Gat, I. Schwartz, A. Schwing, and T. Hazan, “Removing bias in multi-modal classifiers: Regularization by maximizing functional entropies,”Advances in Neural Information Processing Systems, vol. 33, pp. 3197–3208, 2020

    2020

  48. [56]

    Overcoming language priors in visual question answering with adversarial regularization,

    S. Ramakrishnan, A. Agrawal, and S. Lee, “Overcoming language priors in visual question answering with adversarial regularization,”Advances in neural information processing systems, vol. 31, 2018

    2018

  49. [57]

    Vlind-bench: Measuring language priors in large vision-language models,

    K.-i. Lee, M. Kim, S. Yoon, M. Kim, D. Lee, H. Koh, and K. Jung, “Vlind-bench: Measuring language priors in large vision-language models,”arXiv preprint arXiv:2406.08702, 2024

    2024 arXiv

  50. [58]

    Insight over sight? exploring the vision-knowledge conflicts in multimodal llms,

    X. Liu, W. Wang, Y . Yuan, J.-t. Huang, Q. Liu, P. He, and Z. Tu, “Insight over sight? exploring the vision-knowledge conflicts in multimodal llms,”arXiv preprint arXiv:2410.08145, 2024

    2024 arXiv

  51. [59]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi,

    X. Yue, Y . Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y . Sun, et al., “Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Reco...

    2024

  52. [60]

    Mmicl: Empowering vision-language model with multi-modal in-context learning,

    H. Zhao, Z. Cai, S. Si, X. Ma, K. An, L. Chen, Z. Liu, S. Wang, W. Han, and B. Chang, “Mmicl: Empowering vision-language model with multi-modal in-context learning,” inICLR, 2024

    2024

  53. [61]

    Strengthening multimodal large language model with bootstrapped preference optimization,

    R. Pi, T. Han, W. Xiong, J. Zhang, R. Liu, R. Pan, and T. Zhang, “Strengthening multimodal large language model with bootstrapped preference optimization,” inEuropean Conference on Computer Vision, pp. 382–398, Springer, 2024

    2024

  54. [62]

    Paying more attention to image: A training-free method for alleviating hallucination in lvlms,

    S. Liu, K. Zheng, and W. Chen, “Paying more attention to image: A training-free method for alleviating hallucination in lvlms,” inEuropean Conference on Computer Vision, pp. 125–140, Springer, 2024

    2024

  55. [63]

    Looking beyond text: Reducing language bias in large vision-language models via multimodal dual-attention and soft-image guidance,

    H. Zhao, S. Si, L. Chen, Y . Zhang, M. Sun, M. Zhang, and B. Chang, “Looking beyond text: Reducing language bias in large vision-language models via multimodal dual-attention and soft-image guidance,”arXiv preprint arXiv:2411.14279, 2024

    2024 arXiv

  56. [64]

    Debiasing multimodal large language models via noise-aware preference optimization,

    Z. Zhang, H. Tang, J. Sheng, Z. Zhang, Y . Ren, Z. Li, D. Yin, D. Ma, and T. Liu, “Debiasing multimodal large language models via noise-aware preference optimization,”arXiv preprint arXiv:2503.17928, 2025

    2025 arXiv

  57. [65]

    The devil is in the details: Tackling unimodal spurious correlations for generalizable multimodal reward models,

    Z. Li, X. Wen, J. Lou, Y . Ji, Y . Lu, X. Han, D. Zhang, and L. Sun, “The devil is in the details: Tackling unimodal spurious correlations for generalizable multimodal reward models,”arXiv preprint arXiv:2503.03122, 2025

    2025 arXiv

  58. [66]

    Deepseek-v3 technical report,

    A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan,et al., “Deepseek-v3 technical report,”arXiv preprint arXiv:2412.19437, 2024. 13

    2024 arXiv

  59. [67]

    Deepseek llm: Scaling open-source language models with longtermism,

    X. Bi, D. Chen, G. Chen, S. Chen, D. Dai, C. Deng, H. Ding, K. Dong, Q. Du, Z. Fu, et al., “Deepseek llm: Scaling open-source language models with longtermism,”arXiv preprint arXiv:2401.02954, 2024

    2024 arXiv

  60. [68]

    A comprehensive overview of large language models,

    H. Naveed, A. U. Khan, S. Qiu, M. Saqib, S. Anwar, M. Usman, N. Akhtar, N. Barnes, and A. Mian, “A comprehensive overview of large language models,”arXiv preprint arXiv:2307.06435, 2023

    2023 arXiv

  61. [69]

    Masked jigsaw puzzle: A versatile position embedding for vision transformers,

    B. Ren, Y . Liu, Y . Song, W. Bi, R. Cucchiara, N. Sebe, and W. Wang, “Masked jigsaw puzzle: A versatile position embedding for vision transformers,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20382–20391, 2023

    2023

  62. [70]

    A survey on vision transformer,

    K. Han, Y . Wang, H. Chen, X. Chen, J. Guo, Z. Liu, Y . Tang, A. Xiao, C. Xu, Y . Xu,et al., “A survey on vision transformer,”IEEE transactions on pattern analysis and machine intelligence, vol. 45, no. 1, pp. 87–110, 2022

    2022

  63. [71]

    Bringing masked autoencoders explicit contrastive properties for point cloud self-supervised learning,

    B. Ren, G. Mei, D. P. Paudel, W. Wang, Y . Li, M. Liu, R. Cucchiara, L. Van Gool, and N. Sebe, “Bringing masked autoencoders explicit contrastive properties for point cloud self-supervised learning,” inProceedings of the Asian Conference on Computer Vision, pp. 2034–2052, 2024

    2024

  64. [72]

    Swin transformer: Hierarchical vision transformer using shifted windows,

    Z. Liu, Y . Lin, Y . Cao, H. Hu, Y . Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” inProceedings of the IEEE/CVF international conference on computer vision, pp. 10012–10022, 2021

    2021

  65. [73]

    Sharing key semantics in transformer makes efficient image restoration,

    B. Ren, Y . Li, J. Liang, R. Ranjan, M. Liu, R. Cucchiara, L. V . Gool, M.-H. Yang, and N. Sebe, “Sharing key semantics in transformer makes efficient image restoration,”Advances in Neural Information Processing Systems, vol. 37, pp. 7427–7463, 2024

    2024

  66. [74]

    Learning disentangled identifiers for action-customized text-to-image generation,

    S. Huang, B. Gong, Y . Feng, X. Chen, Y . Fu, Y . Liu, and D. Wang, “Learning disentangled identifiers for action-customized text-to-image generation,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7797–7806, 2024

    2024

  67. [75]

    Languagebind: Extending video-language pretraining to n-modality by language-based semantic alignment,

    B. Zhu, B. Lin, M. Ning, Y . Yan, J. Cui, H. Wang, Y . Pang, W. Jiang, J. Zhang, Z. Li, C. Zhang, Z. Li, W. Liu, and L. Yuan, “Languagebind: Extending video-language pretraining to n-modality by language-based semantic alignment,” inICLR, OpenReview.net, 2024

    2024

  68. [76]

    Unibind: Llm-augmented unified and balanced representation space to bind them all,

    Y . Lyu, X. Zheng, J. Zhou, and L. Wang, “Unibind: Llm-augmented unified and balanced representation space to bind them all,” inCVPR, pp. 26742–26752, IEEE, 2024

    2024

  69. [77]

    Optimizing intersection-over-union in deep neural networks for image segmentation,

    M. A. Rahman and Y . Wang, “Optimizing intersection-over-union in deep neural networks for image segmentation,” inInternational symposium on visual computing, pp. 234–244, Springer, 2016

    2016

  70. [78]

    Generalized inter- section over union: A metric and a loss for bounding box regression,

    H. Rezatofighi, N. Tsoi, J. Gwak, A. Sadeghian, I. Reid, and S. Savarese, “Generalized inter- section over union: A metric and a loss for bounding box regression,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 658–666, 2019

    2019

  71. [79]

    Subjective and objective quality assessment for image restoration: A critical survey,

    B. Hu, L. Li, J. Wu, and J. Qian, “Subjective and objective quality assessment for image restoration: A critical survey,”Signal Processing: Image Communication, vol. 85, p. 115839, 2020

    2020

  72. [80]

    Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models,

    P. Xu, W. Shao, K. Zhang, P. Gao, S. Liu, M. Lei, F. Meng, S. Huang, Y . Qiao, and P. Luo, “Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models,”IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    2024

  73. [81]

    Valor: Vision-audio-language omni-perception pretraining model and dataset,

    J. Liu, S. Chen, X. He, L. Guo, X. Zhu, W. Wang, and J. Tang, “Valor: Vision-audio-language omni-perception pretraining model and dataset,”IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    2024

  74. [82]

    Multimodal visual-tactile representation learning through self-supervised contrastive pre-training,

    V . Dave, F. Lygerakis, and E. Rueckert, “Multimodal visual-tactile representation learning through self-supervised contrastive pre-training,” in2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 8013–8020, IEEE, 2024. 14

    2024

  75. [83]

    Omnibind: Teach to build unequal-scale modality interaction for omni-bind of all,

    Y . Lyu, X. Zheng, D. Kim, and L. Wang, “Omnibind: Teach to build unequal-scale modality interaction for omni-bind of all,”arXiv preprint arXiv:2405.16108, 2024

    2024 arXiv

  76. [84]

    Soft robotic hand with tactile palm-finger coordination,

    N. Zhang, J. Ren, Y . Dong, X. Yang, R. Bian, J. Li, G. Gu, and X. Zhu, “Soft robotic hand with tactile palm-finger coordination,”Nature Communications, vol. 16, no. 1, p. 2395, 2025

    2025

  77. [85]

    Categorizing robots by performance fitness into the tree of robots,

    R. J. Kirschner, K. Karacan, A. Melone, and S. Haddadin, “Categorizing robots by performance fitness into the tree of robots,”Nature Machine Intelligence, pp. 1–12, 2025

    2025

  78. [86]

    Vision-based tactile sensor design using physically based rendering,

    A. Agarwal, A. Wilson, T. Man, E. Adelson, I. Gkioulekas, and W. Yuan, “Vision-based tactile sensor design using physically based rendering,”Communications Engineering, vol. 4, no. 1, p. 21, 2025

    2025

  79. [87]

    Bi-vla: Vision-language-action model-based system for bimanual robotic dexterous manipulations,

    K. F. Gbagbe, M. A. Cabrera, A. Alabbas, O. Alyunes, A. Lykov, and D. Tsetserukou, “Bi-vla: Vision-language-action model-based system for bimanual robotic dexterous manipulations,” in 2024 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp. 2864–2869, IEEE, 2024

    2024

  80. [88]

    A practical tutorial on explainable ai techniques,

    A. Bennetot, I. Donadello, A. El Qadi El Haouari, M. Dragoni, T. Frossard, B. Wagner, A. Sarranti, S. Tulli, M. Trocan, R. Chatila,et al., “A practical tutorial on explainable ai techniques,”ACM Computing Surveys, vol. 57, no. 2, pp. 1–44, 2024

    2024

  81. [89]

    Explainable ai (xai): Core ideas, techniques, and solutions,

    R. Dwivedi, D. Dave, H. Naik, S. Singhal, R. Omer, P. Patel, B. Qian, Z. Wen, T. Shah, G. Morgan,et al., “Explainable ai (xai): Core ideas, techniques, and solutions,”ACM Computing Surveys, vol. 55, no. 9, pp. 1–33, 2023. 15

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.