Pith. sign in

REVIEW 16 cited by

ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalities

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2305.11172 v1 pith:NFLRSKYG submitted 2023-05-18 cs.CV cs.CLcs.SDeess.AS

ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalities

Peng Wang , Shijie Wang , Junyang Lin , Shuai Bai , Xiaohuan Zhou , Jingren Zhou , Xinggang Wang , Chang Zhou This is my paper
classification cs.CV cs.CLcs.SDeess.AS
keywords modalitiesone-peacemodelaudiotasksunlimitedadaptersalign
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
read the original abstract

In this work, we explore a scalable way for building a general representation model toward unlimited modalities. We release ONE-PEACE, a highly extensible model with 4B parameters that can seamlessly align and integrate representations across vision, audio, and language modalities. The architecture of ONE-PEACE comprises modality adapters, shared self-attention layers, and modality FFNs. This design allows for the easy extension of new modalities by adding adapters and FFNs, while also enabling multi-modal fusion through self-attention layers. To pretrain ONE-PEACE, we develop two modality-agnostic pretraining tasks, cross-modal aligning contrast and intra-modal denoising contrast, which align the semantic space of different modalities and capture fine-grained details within modalities concurrently. With the scaling-friendly architecture and pretraining tasks, ONE-PEACE has the potential to expand to unlimited modalities. Without using any vision or language pretrained model for initialization, ONE-PEACE achieves leading results on a wide range of uni-modal and multi-modal tasks, including image classification (ImageNet), semantic segmentation (ADE20K), audio-text retrieval (AudioCaps, Clotho), audio classification (ESC-50, FSD50K, VGGSound), audio question answering (AVQA), image-text retrieval (MSCOCO, Flickr30K), and visual grounding (RefCOCO/+/g). Code is available at https://github.com/OFA-Sys/ONE-PEACE.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 16 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Fusion Embedding: A Unified Embedding Space for Text, Image, Video, and Audio

    cs.CL 2026-07 conditional novelty 7.0

    Trained connectors and audio-only gated adapters integrate audio into a frozen vision-language embedding space, preserving base outputs bit-exactly and yielding emergent audio-image retrieval.

  2. EmergentBridge: Improving Zero-Shot Cross-Modal Transfer in Unified Multimodal Embedding Models

    cs.AI 2026-04 unverdicted novelty 7.0

    EmergentBridge improves zero-shot cross-modal transfer for unpaired modality pairs by learning noisy bridge anchors and enforcing proxy alignment only in the orthogonal subspace to preserve existing anchor alignments.

  3. GME: Improving Universal Multimodal Retrieval by Multimodal LLMs

    cs.CL 2024-12 unverdicted novelty 7.0

    GME achieves state-of-the-art results in universal multimodal retrieval by training on a balanced synthetic multimodal dataset.

  4. Qwen-RobotWorld Technical Report: Unifying Embodied World Modeling through Language-Conditioned Video Generation

    cs.CV 2026-06 unverdicted novelty 6.0

    Qwen-RobotWorld is a language-conditioned video world model using Double-Stream MMDiT, an 8.6M-frame embodied corpus, and progressive curriculum training that ranks first on EWMBench and DreamGen Bench.

  5. EmergentBridge: Improving Zero-Shot Cross-Modal Transfer in Unified Multimodal Embedding Models

    cs.AI 2026-04 unverdicted novelty 6.0

    EmergentBridge enhances zero-shot cross-modal performance on unpaired modalities by learning noisy bridge anchors from existing alignments and enforcing proxy alignment only in the orthogonal subspace to avoid gradien...

  6. InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency

    cs.CV 2025-08 unverdicted novelty 6.0

    InternVL3.5 advances open-source multimodal models with Cascade RL for +16% reasoning gains and ViR for 4x inference speedup, with the 241B model reaching SOTA among open-source MLLMs on multimodal, reasoning, and age...

  7. InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models

    cs.CV 2025-04 conditional novelty 6.0

    InternVL3-78B sets a new open-source SOTA of 72.2 on MMMU via native joint multimodal pre-training, V2PE, MPO, and test-time scaling while remaining competitive with proprietary models.

  8. Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling

    cs.CV 2024-12 unverdicted novelty 6.0

    InternVL 2.5 is the first open-source MLLM to surpass 70% on the MMMU benchmark via model, data, and test-time scaling, with a 3.7-point gain from chain-of-thought reasoning.

  9. Qwen-Audio-VAE Technical Report

    eess.AS 2026-07 conditional novelty 5.0

    A 12.5 Hz continuous audio VAE reconstructs speech, music, and sound well while encoding 64×30s clips in 541 ms after latency-aware encoder pruning.

  10. Grounded 3D-Aware Spatial Vision-Language Modeling

    cs.CV 2026-05 unverdicted novelty 5.0

    GR3D is a VLM that combines explicit 2D, implicit 2D, and monocular 3D grounding mechanisms to improve performance on spatial understanding benchmarks.

  11. AeroRAG: Structured Multimodal Retrieval-Augmented LLM for Fine-Grained Aerial Visual Reasoning

    cs.CV 2026-04 unverdicted novelty 5.0

    AeroRAG improves fine-grained aerial visual question answering by converting images to scene graphs and using retrieval-augmented generation to create compact LLM prompts.

  12. AeroRAG: Structured Multimodal Retrieval-Augmented LLM for Fine-Grained Aerial Visual Reasoning

    cs.CV 2026-04 conditional novelty 5.0

    Scene-graph-guided multimodal RAG improves aerial VQA by feeding query-relevant structured visual knowledge to a text LLM instead of dense visual tokens.

  13. Beyond Surface Artifacts: Capturing Shared Latent Forgery Knowledge Across Modalities

    cs.CV 2026-04 unverdicted novelty 5.0

    Introduces MAF framework and DeepModal-Bench to capture universal cross-modal forgery traces for better generalization in multimodal deepfake detection.

  14. DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding

    cs.CV 2024-12 accept novelty 5.0

    DeepSeek-VL2 is a series of MoE vision-language models using dynamic tiling and latent attention that reach competitive or state-of-the-art results on VQA, OCR, document understanding and grounding with 1.0B to 4.5B a...

  15. InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks

    cs.CV 2023-12 unverdicted novelty 5.0

    InternVL scales a vision model to 6B parameters and aligns it with LLMs using web data to achieve state-of-the-art results on 32 visual-linguistic benchmarks.

  16. Multiple Domain Generalization Using Category Information Independent of Domain Differences

    cs.CV 2026-04 unverdicted novelty 4.0

    A domain generalization method extracts domain-independent category features and applies SQ-VAE to bridge gaps, yielding accuracy gains on vascular and cell nucleus segmentation datasets.