Pith. sign in

REVIEW 10 cited by

Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Survey

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.05391 v4 pith:W6LD32XS submitted 2024-02-08 cs.AI cs.CVcs.IRcs.LG

Zhuo Chen , Yichi Zhang , Yin Fang , Yuxia Geng , Lingbing Guo , Xiang Chen , Qian Li , Wen Zhang
show 7 more authors
Jiaoyan Chen Yushan Zhu Jiaqi Li Xiaoze Liu Jeff Z. Pan Ningyu Zhang Huajun Chen
This is my paper · ORCID
classification cs.AIcs.CVcs.IRcs.LG
keywords multi-modalresearchknowledgelearningtasksmmkgsurveycomprehensive
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Knowledge Graphs (KGs) play a pivotal role in advancing various AI applications, with the semantic web community's exploration into multi-modal dimensions unlocking new avenues for innovation. In this survey, we carefully review over 300 articles, focusing on KG-aware research in two principal aspects: KG-driven Multi-Modal (KG4MM) learning, where KGs support multi-modal tasks, and Multi-Modal Knowledge Graph (MM4KG), which extends KG studies into the MMKG realm. We begin by defining KGs and MMKGs, then explore their construction progress. Our review includes two primary task categories: KG-aware multi-modal learning tasks, such as Image Classification and Visual Question Answering, and intrinsic MMKG tasks like Multi-modal Knowledge Graph Completion and Entity Alignment, highlighting specific research trajectories. For most of these tasks, we provide definitions, evaluation benchmarks, and additionally outline essential insights for conducting relevant research. Finally, we discuss current challenges and identify emerging trends, such as progress in Large Language Modeling and Multi-modal Pre-training strategies. This survey aims to serve as a comprehensive reference for researchers already involved in or considering delving into KG and multi-modal learning research, offering insights into the evolving landscape of MMKG research and supporting future work.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 30 citations worldwide. Full citation record

  1. RADD: Retrieval-Augmented Discrete Diffusion for Multi-Modal Knowledge Graph Completion

    cs.AI 2026-04 unverdicted novelty 6.0 of 10

    RADD decouples retrieval and reranking in multi-modal KGC via a relation-aware KGE retriever and conditional discrete denoiser, reporting state-of-the-art results on three benchmarks.

  2. NLKI: A lightweight Natural Language Knowledge Integration Framework for Improving Small VLMs in Commonsense VQA Tasks

    cs.CL 2025-08 conditional novelty 6.0 of 10

    NLKI combines fine-tuned dense retrieval, LLM-generated explanations, and noise-robust losses to improve small VLMs on commonsense VQA.

  3. Dark Side of Modalities: Reinforced Multimodal Distillation for Multimodal Knowledge Graph Reasoning

    cs.MM 2025-07 conditional novelty 6.0 of 10

    A unimodal student model, taught by reinforcement-selected combinations of multimodal teachers via neighbor-decoupled knowledge distillation, sets new state-of-the-art results on five multimodal knowledge graph reason...

  4. Abstractive Visual Understanding of Multi-modal Structured Knowledge: A New Perspective for MLLM Evaluation

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A new benchmark, M3STR, renders knowledge-graph subgraphs as images and shows current MLLMs score near random on anomaly detection and poorly on entity counting.

  5. HERGC: Heterogeneous Experts Representation and Generative Completion for Multimodal Knowledge Graphs

    cs.CL 2025-06 conditional novelty 6.0 of 10

    HERGC combines a multimodal expert-based retriever with a fine-tuned LLM re-ranker to achieve state-of-the-art multimodal knowledge graph completion on MKG-W, MKG-Y, and DB15K.

  6. FinRipple: Aligning Large Language Models with Financial Market for Event Ripple Effect Awareness

    cs.SI 2025-05 reject novelty 6.0 of 10

    FinRipple aligns LLMs with financial markets via knowledge-graph adapters and PPO using CAPM residuals as reward, claiming strong ripple-effect prediction, but the evaluation is circular and artifacts are unavailable.

  7. K-ON: Stacking Knowledge On the Head Layer of Large Language Model

    cs.CL 2025-02 conditional novelty 6.0 of 10

    K-ON stacks K prediction heads onto an LLM to generate entity tokens in one step and uses entity-level contrastive learning, achieving new state-of-the-art results on two knowledge graph completion benchmarks.

  8. I2CR: Intra- and Inter-modal Collaborative Reflections for Multimodal Entity Linking

    cs.CV 2025-08 conditional novelty 5.0 of 10

    A text-first, multi-round visual feedback framework reports state-of-the-art top-1 accuracy on WikiMEL, WikiDiverse, and RichMEL.

  9. Position Paper: Metadata Enrichment Model: Integrating Neural Networks and Semantic Knowledge Graphs for Cultural Heritage Applications

    cs.CV 2025-05 conditional novelty 4.0 of 10

    A conceptual metadata enrichment framework integrating iterative vision analysis with LLM-driven decisions and RDF knowledge graphs is proposed, with a small annotated incunabula dataset released.

  10. Towards Structure-aware Model for Multi-modal Knowledge Graph Completion

    cs.MM 2025-05 conditional novelty 4.0 of 10

    TSAM combines token-level fusion of visual and textual data with structure-anchored contrastive learning, outperforming prior multi-modal KGC models on DB15K, MKG-W, and MKG-Y.

Pith tools