Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Benchmarking Multimodal Knowledge Conflict for Large Multimodal Models

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Large multimodal models detect knowledge conflicts but usually answer from memory, a new benchmark shows.

desk verdict A useful benchmark resource, but the headline claim that LMMs favor parametric knowledge is contradicted by the paper's own clean entity-knowledge numbers. read the letter →

arxiv 2505.19509 v1 pith:AFG4P4IN submitted 2025-05-26 cs.LG cs.AI

Yifan Jia , Kailin Jiang , Yuyang Liang , Qihan Ren , Yi Xin , Rui Yang , Fenze Feng , Mingcai Chen
show 6 more authors
Hengyang Lu Haozhe Wang Xiaoye Qu Dongrui Liu Lizhen Cui Yuntao Du
This is my paper
classification cs.LGcs.AI
keywords multimodalknowledgeconflictlargemodelsretrieval-augmentedgenerationcontext-memoryinter-contextdetectioncounterfactualeditingbenchmarkconstruction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces MMKC-Bench, a human-verified benchmark of 1,573 multimodal knowledge conflicts, and uses it to ask how large multimodal models behave when external evidence contradicts what they already know. The paper claims that current models can identify when a conflict exists, but when answering they tend to reproduce their internal parametric knowledge rather than follow the external evidence, a pattern that holds across open-source and closed-source models. The authors care because retrieval-augmented generation depends on models using retrieved evidence; if models quietly override that evidence with memory, RAG systems become unreliable. The benchmark covers three conflict types (entity recognition, entity knowledge, and visual semantics) under both context-memory and inter-context settings.

What carries the argument

The load-bearing object is MMKC-Bench itself: a counterfactually edited dataset in which each original knowledge item (image plus text) is paired with one or two conflicting versions, produced by replacing entity names, attribute values, or semantic labels. Each conflict is cast as a multiple-choice or open-ended question whose answer options include the original, the conflicting, and a distractor answer. Behavior is summarized by three ratios (Original Answer Ratio, Counter Answer Ratio, and Irrelevant Answer Ratio) computed against a non-conflict baseline, and conflict detection is scored as binary accuracy in coarse-grained and fine-grained settings.

What would settle it

Run the context-memory conflict task again with the image removed from the conflicting evidence, or replaced by a different image, and compare the Original Answer Ratio; if it drops substantially, the apparent preference for internal knowledge is actually a preference for the unchanged visual input.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is an asymmetry: multimodal knowledge conflicts are recognized but not followed. Across nine models in the Qwen2.5-VL, InternVL3, and GPT-4o mini families, conflict-detection accuracy averages roughly 75 to 79 percent, while in behavior analysis the Original Answer Ratio (answers matching the model's non-conflict response) exceeds the Counter Answer Ratio (answers matching the conflicting external evidence) in nearly every configuration. The paper interprets this as models favoring internal parametric knowledge over external evidence, and notes that this contrasts with text-only LLMs, which generally yield to external knowledge. It also finds that sensitivity is higher for knowledge-level conflicts (such as birth-year contradictions) than for recognition-level conflicts (such as which building is shown), and that larger models show an even stronger preference for internal knowledge.

Load-bearing premise

The paper's main behavioral conclusion assumes that a model's answer when given the correct image and correct text is a pure measure of what it remembers; because the same correct image is also present when the conflicting text arrives, a model that simply follows the image would be miscounted as following memory.

Editorial extensions

If this is right

  • In multimodal RAG pipelines, retrieved evidence may not shift model answers as intended, so systems should verify whether the model actually used the retrieved context.
  • Model scaling within a family increases agreement with internal knowledge, so bigger is not automatically better for evidence-following.
  • Knowledge-level conflicts such as birth years are more likely to make models switch answers than recognition-level conflicts such as which entity is in the image, suggesting that perception-style conflicts need different mitigation.
  • The benchmark provides a reusable protocol: counterfactual editing plus human verification, with behavior analysis and conflict detection as two separable evaluation tasks.
  • Average detection accuracy around 75 to 79 percent means current LMMs can flag conflicts but still need better mechanisms for resolving them.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension would remove or change the image in the conflict setting; because the baseline uses the correct image and the conflict setting keeps it, the current Original Answer Ratio may partly measure image-following rather than parametric memory.
  • The synthetic counterfactual design, acknowledged by the authors, means real-world conflicts such as naturally contradictory web pages could behave differently, so building a naturalistic multimodal conflict set would test whether the memory preference persists.
  • If the memory-over-evidence pattern holds, multimodal RAG training should include interleaved multi-source contexts rather than isolated image-text pairs, directly addressing the training-data gap the paper proposes as an explanation.
  • The finding that models detect conflicts better in non-conflict scenarios than in conflict scenarios may reflect a bias to answer 'yes' only when evidence is emphatic; per-type error analysis could confirm this.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces MMKC-Bench, a multimodal knowledge conflict benchmark covering context-memory and inter-context conflicts across three conflict types (entity recognition, entity knowledge, and visual semantic), with 1,573 knowledge instances and 3,381 images constructed through an automated pipeline with human verification. Nine LMMs from three families are evaluated on model-behavior analysis and conflict-detection tasks. The headline empirical claim, stated in the abstract and Section 5.2, is that while current LMMs can recognize knowledge conflicts, they tend to favor internal parametric knowledge over external evidence.

Significance. The benchmark itself addresses a real gap: existing multimodal conflict datasets mostly cover intra-memory conflicts, and the paper's pipeline for generating context-memory and inter-context conflicts with human verification is useful for the multimodal RAG community. The paper ships a public code repository and evaluates nine models across two question formats and two granularities of conflict detection, which is a solid engineering contribution. However, the central behavioral finding is not established because the OAR baseline is confounded, and the paper's own fine-grained results contradict the headline claim on the subset that actually isolates parametric knowledge. If the central claim were corrected, the benchmark could still be a valuable resource, but the current manuscript overstates what the experiments demonstrate.

major comments (4)
  1. [5.1, 5.2, Table 3] The OAR baseline is confounded. OAR is defined as consistency with the model's answer in the non-conflict setting, where the non-conflict setting contains the correct image and the correct text. In entity recognition and visual semantic conflicts, the correct image itself determines the answer (e.g., an image of the Empire State Building or an OK gesture), so in the conflict setting the model can produce the 'original' answer by following visual evidence rather than by retrieving parametric knowledge. The clean subset, entity knowledge (e.g., birth year), does not have this confound because the image does not contain the queried fact. Yet Tables 2 and 3 show that on EK, CAR is comparable to or higher than OAR: in Table 3 open-ended context-memory, Qwen2.5-VL-7B OAR=0.26 vs CAR=0.62, InternVL3-8B OAR=0.27 vs CAR=0.53, GPT-4o mini OAR=0.36 vs CAR=0.47; in inter-context, Qwen2.5-VL-7B OAR=0.14 vs CAR=0.76. Thus the central claim that LMMs favor parametric knowledge is reversed on the subset that isolates parametric knowledge from image evidence. The paper must redefine the parametric-knowledge baseline (e.g., answer with the image only, or with no image) and report conflict-type-specific results, not averages across confounded types.
  2. [5.2, Table 3] Observation 1 selectively reports context-memory results. The inter-context open-ended rows in Table 3 show the opposite of the claim for two open-source models: Qwen2.5-VL-7B has OAR=0.38 vs CAR=0.50, and InternVL3-8B has OAR=0.30 vs CAR=0.54. Since the abstract and conclusion state generally that LMMs favor internal parametric knowledge over external evidence, the paper should either qualify the claim to the context-memory setting only or present all settings transparently and explain the discrepancy.
  3. [5.2, Observation 2] The statement 'entity recognition conflicts yield an OAR as low as 0.26 on Qwen2.5-VL-7B' is inconsistent with Table 3, where Qwen2.5-VL-7B ER OAR is 0.66 (context-memory) and 0.65 (inter-context); the 0.26 value corresponds to entity knowledge OAR. This appears to be a factual error in the text and undermines the reliability of the qualitative observations.
  4. [5.2, overall] No error bars, confidence intervals, or significance tests are reported for any of the OAR/CAR comparisons. Several differences are small (e.g., Table 3 Qwen2.5-VL-7B context-memory avg OAR=0.44 vs CAR=0.40), and some categories contain as few as 147 instances (visual semantic). The paper should report variance across repeated runs (if any) or statistical significance to support the claimed trends.
minor comments (6)
  1. [4.1] In Section 4.1, 'from the MMKC-Bench dataset' should read 'from the MMKE-Bench dataset' (reference [37]); the sentence also says 'Since these types are already included in MMKC-Bench,' which is confusing because MMKC-Bench is the current benchmark being built.
  2. [Table 1] The row label 'Visual Entity Conflict' is inconsistent with the term 'Entity Recognition Conflict' used throughout the text and Figure 1; please align the terminology.
  3. [5.1] The metric is defined as 'Irrelevant Answer Ratio' with acronym IAO, but Tables 2 and 3 and the surrounding text use IAR; please use one acronym consistently.
  4. [5.4] In the case study, 'curvewaringsign' appears to be a typo; it should likely be 'curve warning sign'.
  5. [5.2, Observation 4] The text mentions sizes '13B' and '70B' for Qwen2.5-VL, but the paper's model list says Qwen2.5-VL (3B, 7B, 32B, 72B); please correct the model sizes.
  6. [6] In the conclusion, 'real-world multimodal knowledge conflict benchmarks, such as [10]' cites WikiContradict, which is a text-only benchmark; please either use a multimodal real-world benchmark or clarify that [10] is text-only.

Circularity Check

1 steps flagged · score 6.0 of 10

The OAR metric defines 'internal knowledge' as the non-conflict answer, which for entity-recognition and visual-semantic conflicts is just the image's content; the headline preference-for-parametric-knowledge finding is partly an artifact of that definition.

  1. self definitional [Sec. 5.1 (Evaluation Metrics), Sec. 5.2 (Observation 1), with Sec. 4.2 defining conflict inputs.]
    "we retain the original image while modifying the textual component. ... Each model prediction under a conflict scenario is categorized into one of three types: (1) consistent with the model's answer in the non-conflict setting ... To enable this, we first perform QA under a non-conflict setting to establish the model's internal knowledge. ... under context-memory conflicts, the average OAR exceeds CAR in all cases (6 out of 6), indicating that LMMs tend to favor internal knowledge."

    OAR is defined as agreement with the answer in the non-conflict setting, which the paper equates with 'the model's internal knowledge.' For entity recognition and visual semantic conflicts, that non-conflict answer is the entity name or gesture meaning read directly off the image. The conflict setting retains the same original image and only replaces the text, so a model that continues to trust the visual input reproduces the non-conflict answer and is scored OAR even though it is following external visual evidence, not parametric memory. High OAR on these types, and on the averages that include them, is thus enforced by the metric's construction for any image-faithful model.

full rationale

This is a benchmark-construction paper rather than a formal derivation, so most of the pipeline (counterfactual editing, human verification, and QA generation) is not circular. The one genuinely circular-by-construction element is the OAR measure used for the central behavioral claim: for entity-recognition and visual-semantic conflicts, the 'internal knowledge' baseline is the answer to the image itself, and the same image remains in the conflict prompt, so OAR counts image-following as parametric preference. The paper's own per-type open-ended results on entity knowledge (Table 3) show CAR greater than OAR on two of the three representative models, indicating that the headline conclusion is not independently supported on the subset that actually isolates parametric knowledge. A minor self-citation trail also exists: the visual-semantic instances are taken directly from the authors' earlier MMKE-Bench [37] (Section 4.1, which typo's the name as MMKC-Bench), and the LLM comparison relies on the authors' ConflictBank [11]; however, those are published, independently evaluated resources and are not the main source of the score. The score of 6 reflects that one of the paper's central 'findings' reduces, for two of its three conflict types, to a definitional property of its metric rather than to an empirical discovery.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

No fitted numeric parameters are present; the paper contributes a dataset, so the central claim does not depend on fitted constants. The two domain assumptions listed above are the load-bearing premises about how the benchmark measures conflict and validates its labels.

assumptions (2)
  • domain assumption The answer a model gives under the non-conflict setting (image plus matching text) is treated as its internal parametric knowledge and used as the OAR baseline.
    Section 5.1 defines OAR as consistency with the answer in the non-conflict setting. For recognition-based conflicts the same image is present in both settings, so this baseline may incorporate visual evidence rather than pure parametric memory.
  • domain assumption Counterfactually edited text descriptions and LLM-generated questions are assumed to create valid conflict labels after human verification.
    Sections 4.2 and 4.4 rely on LLM generation plus human review, but no inter-annotator agreement or verification statistics are reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Benchmarking Multimodal Knowledge Conflict for Large Multimodal Models." pith.science (2026). https://pith.science/paper/AFG4P4IN

@misc{pith2026250519509,
  author       = {Pith},
  title        = {Pith review of: Benchmarking Multimodal Knowledge Conflict for Large Multimodal Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AFG4P4IN}},
  note         = {Machine review of arXiv:2505.19509}
}
read the original abstract

Large Multimodal Models(LMMs) face notable challenges when encountering multimodal knowledge conflicts, particularly under retrieval-augmented generation(RAG) frameworks where the contextual information from external sources may contradict the model's internal parametric knowledge, leading to unreliable outputs. However, existing benchmarks fail to reflect such realistic conflict scenarios. Most focus solely on intra-memory conflicts, while context-memory and inter-context conflicts remain largely investigated. Furthermore, commonly used factual knowledge-based evaluations are often overlooked, and existing datasets lack a thorough investigation into conflict detection capabilities. To bridge this gap, we propose MMKC-Bench, a benchmark designed to evaluate factual knowledge conflicts in both context-memory and inter-context scenarios. MMKC-Bench encompasses three types of multimodal knowledge conflicts and includes 1,573 knowledge instances and 3,381 images across 23 broad types, collected through automated pipelines with human verification. We evaluate three representative series of LMMs on both model behavior analysis and conflict detection tasks. Our findings show that while current LMMs are capable of recognizing knowledge conflicts, they tend to favor internal parametric knowledge over external evidence. We hope MMKC-Bench will foster further research in multimodal knowledge conflict and enhance the development of multimodal RAG systems. The source code is available at https://github.com/MLLMKCBENCH/MLLMKC.

Figures

Figures reproduced from arXiv: 2505.19509 by the authors.

Figure 1
Figure 1. Three types of multimodal knowledge conflict in MMKC-Bench. It is noted that the original [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The construction pipeline of MMKC-Bench. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. The data types of MMKC-Bench. For entity recognition and entity knowledge conflicts, we manually define multiple candidate visual entity types (e.g., person, building). For each type, we use a large language model (LLM) to generate a list of the most prominent entities (e.g., Messi under the “per￾son” category). Once the entity list is obtained, we crawl their images from Google and retrieve entity de￾scriptions fro… view at source ↗
Figures from the paper (21 more)
Figure 4
Figure 4. Figure 4: The results of Qwen2.5-VL with different model sizes under context-memory conflict with [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: The results of Qwen2.5-VL with different model sizes under inter-context conflict with [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Case study of context-memory conflict and inter-context conflict involving entity recognition [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 8
Figure 8. Figure 8: Figure8 [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 7
Figure 7. Figure 7: Prompt for summary of original knowledge [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Prompt for Generate Entity Recognition conflicting knowledge [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Prompt for Generate character time knowledge conflicting knowledge [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Prompt for Generate character country of citizenship knowledge conflicting knowledge [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 11
Figure 11. Figure 11: Prompt for Generate character occupation knowledge conflicting knowledge [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 12
Figure 12. Figure 12: The results of Qwen2.5-VL with different model sizes under context-memory conflict [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]
Figure 13
Figure 13. Figure 13: The results of Qwen2.5-VL with different model sizes under inter-context conflict with [PITH_FULL_IMAGE:figures/full_fig_p017_13.png]
Figure 14
Figure 14. Figure 14: The results of InternVL3 with different model sizes under context-memory conflict with [PITH_FULL_IMAGE:figures/full_fig_p017_14.png]
Figure 15
Figure 15. Figure 15: The results of InternVL3 with different model sizes under inter-context conflict with [PITH_FULL_IMAGE:figures/full_fig_p018_15.png]
Figure 16
Figure 16. Figure 16: The results of InternVL3 with different model sizes under context-memory conflict with [PITH_FULL_IMAGE:figures/full_fig_p018_16.png]
Figure 17
Figure 17. Figure 17: The results of InternVL3 with different model sizes under inter-context conflict with [PITH_FULL_IMAGE:figures/full_fig_p018_17.png]
Figure 18
Figure 18. Figure 18: Data instance display diagram of eight entity recognition types. [PITH_FULL_IMAGE:figures/full_fig_p021_18.png]
Figure 19
Figure 19. Figure 19: Data instance display diagram of three visual semantic types. [PITH_FULL_IMAGE:figures/full_fig_p021_19.png]
Figure 20
Figure 20. Figure 20: A data example showing three dimensions of character knowledge. [PITH_FULL_IMAGE:figures/full_fig_p022_20.png]
Figure 21
Figure 21. Figure 21: Data example display of three dimensions of brand knowledge. [PITH_FULL_IMAGE:figures/full_fig_p022_21.png]
Figure 22
Figure 22. Figure 22: A diagram showing cases of different data instances under Context-Memory Conflict. [PITH_FULL_IMAGE:figures/full_fig_p023_22.png]
Figure 23
Figure 23. Figure 23: Case diagram showing different data instances under Inter-Context Conflict. [PITH_FULL_IMAGE:figures/full_fig_p024_23.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MissingBench-Verified: Probing Vision-Language Models' Inability to Detect Missing Object Parts

    cs.CV 2026-07 conditional novelty 5.0 of 10

    Ten leading VLMs mostly fail to report removed essential object parts as missing, and simulated detector evidence, image tools, longer reasoning, and an easier fine-tune barely improve accuracy.

Reference graph

Works this paper leans on

38 extracted references · 21 canonical work pages · cited by 1 Pith paper

  1. [1]

    Qwen2.5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    arXiv 2025

  2. [2]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 24185–24198, 2024

    2024

  3. [3]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    2023

  4. [4]

    A survey on multimodal large language models for autonomous driving

    Can Cui, Yunsheng Ma, Xu Cao, Wenqian Ye, Yang Zhou, Kaizhao Liang, Jintai Chen, Juanwu Lu, Zichong Yang, Kuei-Da Liao, et al. A survey on multimodal large language models for autonomous driving. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 958–979, 2024

    2024

  5. [5]

    Openthinkimg: Learning to think with images via visual tool reinforcement learning.arXiv preprint arXiv:2505.08617, 2025

    Zhaochen Su, Linjie Li, Mingyang Song, Yunzhuo Hao, Zhengyuan Yang, Jun Zhang, Guanjie Chen, Jiawei Gu, Juntao Li, Xiaoye Qu, et al. Openthinkimg: Learning to think with images via visual tool reinforcement learning.arXiv preprint arXiv:2505.08617, 2025

    arXiv 2025

  6. [6]

    A survey on rag meeting llms: Towards retrieval-augmented large language models

    Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. A survey on rag meeting llms: Towards retrieval-augmented large language models. InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 6491–6501, 2024

    work page 2024

  7. [7]

    A survey of multimodal retrieval- augmented generation.arXiv preprint arXiv:2504.08748, 2025

    Lang Mei, Siyu Mo, Zhihan Yang, and Chong Chen. A survey of multimodal retrieval- augmented generation.arXiv preprint arXiv:2504.08748, 2025

    arXiv 2025

  8. [8]

    Knowledge conflicts for llms: A survey.EMNLP, 2024

    Rongwu Xu, Zehan Qi, Zhijiang Guo, Cunxiang Wang, Hongru Wang, Yue Zhang, and Wei Xu. Knowledge conflicts for llms: A survey.EMNLP, 2024

    work page 2024

Show all 38 references
  1. [9]

    Resolving knowledge conflicts in large language models.COLM, 2024

    Yike Wang, Shangbin Feng, Heng Wang, Weijia Shi, Vidhisha Balachandran, Tianxing He, and Yulia Tsvetkov. Resolving knowledge conflicts in large language models.COLM, 2024

    2024

  2. [10]

    Wikicontradict: A benchmark for evaluating llms on real-world knowledge conflicts from wikipedia.Advances in Neural Information Processing Systems, 37:109701–109747, 2024

    Yufang Hou, Alessandra Pascale, Javier Carnerero-Cano, Tigran Tchrakian, Radu Marinescu, Elizabeth Daly, Inkit Padhi, and Prasanna Sattigeri. Wikicontradict: A benchmark for evaluating llms on real-world knowledge conflicts from wikipedia.Advances in Neural Information Process...

    2024

  3. [11]

    Conflictbank: A benchmark for evaluating the influence of knowledge conflicts in llms.Advances in Neural Information Processing Systems, 37:103242–103268, 2024

    Zhaochen Su, Jun Zhang, Xiaoye Qu, Tong Zhu, Yanshu Li, Jiashuo Sun, Juntao Li, Min Zhang, and Yu Cheng. Conflictbank: A benchmark for evaluating the influence of knowledge conflicts in llms.Advances in Neural Information Processing Systems, 37:103242–103268, 2024

    2024

  4. [12]

    Insight over sight? exploring the vision-knowledge conflicts in multimodal llms

    Xiaoyuan Liu, Wenxuan Wang, Youliang Yuan, Jen-tse Huang, Qiuzhi Liu, Pinjia He, and Zhaopeng Tu. Insight over sight? exploring the vision-knowledge conflicts in multimodal llms. arXiv preprint arXiv:2410.08145, 2024

    2024 arXiv

  5. [13]

    Is cognition consistent with perception? assessing and mitigating multimodal knowledge conflicts in document understanding.arXiv preprint arXiv:2411.07722, 2024

    Zirui Shao, Chuwei Luo, Zhaoqing Zhu, Hangdi Xing, Zhi Yu, Qi Zheng, and Jiajun Bu. Is cognition consistent with perception? assessing and mitigating multimodal knowledge conflicts in document understanding.arXiv preprint arXiv:2411.07722, 2024

    2024

  6. [14]

    Unraveling cross-modality knowledge conflicts in large vision-language models.arXiv preprint arXiv:2410.03659, 2024

    Tinghui Zhu, Qin Liu, Fei Wang, Zhengzhong Tu, and Muhao Chen. Unraveling cross-modality knowledge conflicts in large vision-language models.arXiv preprint arXiv:2410.03659, 2024

    2024 arXiv

  7. [15]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479, 2025

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479, 2025

    2025 arXiv

  8. [16]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024. 10

    2024 arXiv

  9. [17]

    The revolution of multimodal large language models: a survey.ACL, 2024

    Davide Caffagni, Federico Cocchi, Luca Barsellotti, Nicholas Moratelli, Sara Sarto, Lorenzo Baraldi, Marcella Cornia, and Rita Cucchiara. The revolution of multimodal large language models: a survey.ACL, 2024

    2024

  10. [18]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  11. [19]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  12. [20]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  13. [21]

    An image is worth 16x16 words: Transformers for image recognition at scale.ICLR, 2020

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale.ICLR, 2020

    2020

  14. [22]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271, 2024

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271, 2024

    2024 arXiv

  15. [23]

    Llava-onevision: Easy visual task transfer.Transactions on Machine Learning Research, 2025

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer.Transactions on Machine Learning Research, 2025

    2025

  16. [24]

    Mllm-compbench: A comparative reasoning bench- mark for multimodal llms.Advances in Neural Information Processing Systems, 37:28798– 28827, 2024

    Jihyung Kil, Zheda Mai, Justin Lee, Arpita Chowdhury, Zihe Wang, Kerrie Cheng, Lemeng Wang, Ye Liu, and Wei-Lun Harry Chao. Mllm-compbench: A comparative reasoning bench- mark for multimodal llms.Advances in Neural Information Processing Systems, 37:28798– 28827, 2024

    2024

  17. [25]

    A survey on evaluation of multimodal large language models

    Jiaxing Huang and Jingyi Zhang. A survey on evaluation of multimodal large language models. arXiv preprint arXiv:2408.15769, 2024

    2024 arXiv

  18. [26]

    Clova: A closed-loop visual assistant with tool usage and update

    Zhi Gao, Yuntao Du, Xintong Zhang, Xiaojian Ma, Wenjuan Han, Song-Chun Zhu, and Qing Li. Clova: A closed-loop visual assistant with tool usage and update. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13258–13268, 2024

    2024

  19. [27]

    Videoagent: A memory-augmented multimodal agent for video understanding

    Yue Fan, Xiaojian Ma, Rujie Wu, Yuntao Du, Jiaqi Li, Zhi Gao, and Qing Li. Videoagent: A memory-augmented multimodal agent for video understanding. InEuropean Conference on Computer Vision, pages 75–92. Springer, 2024

    2024

  20. [28]

    Visual programming: Compositional visual reasoning without training

    Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14953–14962, 2023

    2023

  21. [29]

    Rich knowledge sources bring complex knowledge conflicts: Recalibrating models to reflect conflicting evidence.EMNLP, 2022

    Hung-Ting Chen, Michael JQ Zhang, and Eunsol Choi. Rich knowledge sources bring complex knowledge conflicts: Recalibrating models to reflect conflicting evidence.EMNLP, 2022

    2022

  22. [30]

    Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models in knowledge conflicts

    Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su. Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models in knowledge conflicts. InThe Twelfth International Conference on Learning Representations, 2023

    2023

  23. [31]

    Entity-based knowledge conflicts in question answering.EMNLP, 2021

    Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. Entity-based knowledge conflicts in question answering.EMNLP, 2021

    2021

  24. [32]

    Intuitive or dependent? investigating llms’ behavior style to conflicting prompts.ACL, 2024

    Jiahao Ying, Yixin Cao, Kai Xiong, Yidong He, Long Cui, and Yongbin Liu. Intuitive or dependent? investigating llms’ behavior style to conflicting prompts.ACL, 2024

    2024

  25. [33]

    Contradoc: understanding self-contradictions in documents with large language models.NAACL, 2024

    Jierui Li, Vipul Raheja, and Dhruv Kumar. Contradoc: understanding self-contradictions in documents with large language models.NAACL, 2024. 11

    2024

  26. [34]

    Trueteacher: Learning factual consistency evaluation with large language models.EMNLP, 2023

    Zorik Gekhman, Jonathan Herzig, Roee Aharoni, Chen Elkind, and Idan Szpektor. Trueteacher: Learning factual consistency evaluation with large language models.EMNLP, 2023

    2023

  27. [35]

    Dola: Decoding by contrasting layers improves factuality in large language models.ICLR, 2024

    Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. Dola: Decoding by contrasting layers improves factuality in large language models.ICLR, 2024

    2024

  28. [36]

    Factllama: Optimizing instruction-following language models with external knowledge for automated fact-checking

    Tsun-Hin Cheung and Kin-Man Lam. Factllama: Optimizing instruction-following language models with external knowledge for automated fact-checking. In2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pages 846–853. IEEE, 2023

    2023

  29. [37]

    Mmke- bench: A multimodal editing benchmark for diverse visual knowledge.ICLR, 2025

    Yuntao Du, Kailin Jiang, Zhi Gao, Chenrui Shi, Zilong Zheng, Siyuan Qi, and Qing Li. Mmke- bench: A multimodal editing benchmark for diverse visual knowledge.ICLR, 2025

    2025

  30. [38]

    happy" with

    Liang Chen, Yichi Zhang, Shuhuai Ren, Haozhe Zhao, Zefan Cai, Yuchi Wang, Peiyi Wang, Xiangdi Meng, Tianyu Liu, and Baobao Chang. Pca-bench: Evaluating multimodal large language models in perception-cognition-action chain.Findings of ACL, 2024. 12 G DBENCHMARK CONSTRUCTION G.1...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.