Pith. sign in

REVIEW 3 cited by

Benchmarking Large Multimodal Models against Common Corruptions

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2401.11943 v1 pith:STLCNAFA submitted 2024-01-22 cs.LG cs.CLcs.CRcs.CVcs.MM

Jiawei Zhang , Tianyu Pang , Chao Du , Yi Ren , Bo Li , Min Lin This is my paper · ORCID
classification cs.LGcs.CLcs.CRcs.CVcs.MM
keywords commoncorruptionslmmsbenchmarkinglargemmcbenchmodelsmultimodal
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

This technical report aims to fill a deficiency in the assessment of large multimodal models (LMMs) by specifically examining the self-consistency of their outputs when subjected to common corruptions. We investigate the cross-modal interactions between text, image, and speech, encompassing four essential generation tasks: text-to-image, image-to-text, text-to-speech, and speech-to-text. We create a comprehensive benchmark, named MMCBench, that covers more than 100 popular LMMs (totally over 150 model checkpoints). A thorough evaluation under common corruptions is critical for practical deployment and facilitates a better understanding of the reliability of cutting-edge LMMs. The benchmarking code is available at https://github.com/sail-sg/MMCBench

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Can Multimodal Large Language Models Understand OCT?

    cs.CV 2026-07 conditional novelty 6.0 of 10

    OCT-Bench, a 20-task benchmark across 10,076 questions, shows current MLLMs score up to 62% overall but only 43% on clinical reasoning over OCT images.

  2. From Individuals to Interactions: Benchmarking Gender Bias in Multimodal Large Language Models from the Lens of Social Relationship

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Dual-character narrative prompts reveal gender biases in six multimodal LLMs that are largely invisible in single-character evaluations, and GENRES provides a structured benchmark to measure them.

  3. Are Any-to-Any Models More Consistent Across Modality Transfers Than Specialists?

    cs.CL 2025-05 conditional novelty 6.0 of 10

    On the ACON benchmark, any-to-any models do not consistently beat specialist model pairs on cyclic consistency, but show weak latent-space consistency in equivariance tests.

Pith tools