pith. sign in

arxiv: 2308.12038 · v3 · pith:D3MVJ7JSnew · submitted 2023-08-23 · 💻 cs.CL · cs.CV

Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages

Jinyi Hu , Yuan Yao , Chongyi Wang , Shan Wang , Yinxu Pan , Qianyu Chen , Tianyu Yu , Hanghao Wu
show 8 more authors
Yue Zhao Haoye Zhang Xu Han Yankai Lin Jiao Xue Dahai Li Zhiyuan Liu Maosong Sun
This is my paper
classification 💻 cs.CL cs.CV
keywords languagesmultimodalmodelsdatalargeimage-textlearningmultilingual
0
0 comments X
read the original abstract

Recently there has been a significant surge in multimodal learning in terms of both image-to-text and text-to-image generation. However, the success is typically limited to English, leaving other languages largely behind. Building a competitive counterpart in other languages is highly challenging due to the low-resource nature of non-English multimodal data (i.e., lack of large-scale, high-quality image-text data). In this work, we propose MPM, an effective training paradigm for training large multimodal models in non-English languages. MPM demonstrates that Multilingual language models can Pivot zero-shot Multimodal learning across languages. Specifically, based on a strong multilingual large language model, multimodal models pretrained on English-only image-text data can well generalize to other languages in a (quasi)-zero-shot manner, even surpassing models trained on image-text data in native languages. Taking Chinese as a practice of MPM, we build large multimodal models VisCPM in image-to-text and text-to-image generation, which achieve state-of-the-art (open-source) performance in Chinese. To facilitate future research, we open-source codes and model weights at https://github.com/OpenBMB/VisCPM.git.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?

    cs.AI 2024-07 accept novelty 7.0

    WE-MATH benchmark reveals most LMMs rely on rote memorization for visual math while GPT-4o has shifted toward knowledge generalization.

  2. Exploring Adversarial Robustness and Safety Alignment in Multilingual Multi-Modal Large Language Models

    cs.CL 2026-06 unverdicted novelty 6.0

    Adversarial images transfer across languages in MLLMs while apparent safety in weaker languages stems from comprehension and visual-grounding failures rather than genuine alignment.

  3. Task-Aware Structured Memory for Dynamic Multi-modal In-Context Learning

    cs.CV 2026-06 unverdicted novelty 5.0

    TASM proposes a task-aware structured memory framework using task-vector compression, bipartite token merging, and a Core Memory plus Latent Bank hierarchy to enable efficient dynamic multi-modal in-context learning.

  4. MiniCPM-V: A GPT-4V Level MLLM on Your Phone

    cs.CV 2024-08 conditional novelty 5.0

    MiniCPM-Llama3-V 2.5 delivers GPT-4V-level multimodal performance on phones through architecture, pretraining, and alignment optimizations.

  5. A Survey on Multimodal Large Language Models

    cs.CV 2023-06 accept novelty 3.0

    This survey organizes the architectures, training strategies, data, evaluation methods, extensions, and challenges of Multimodal Large Language Models.