Pith. sign in

REVIEW 13 cited by

Meta-Transformer: A Unified Framework for Multimodal Learning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2307.10802 v1 pith:QGYNBJY2 submitted 2023-07-20 cs.CV cs.AIcs.CLcs.LGcs.MM

Yiyuan Zhang , Kaixiong Gong , Kaipeng Zhang , Hongsheng Li , Yu Qiao , Wanli Ouyang , Xiangyu Yue This is my paper · ORCID
classification cs.CVcs.AIcs.CLcs.LGcs.MM
keywords datameta-transformermultimodalunifiedmodalitiesencoderframeworklearning
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

Multimodal learning aims to build models that can process and relate information from multiple modalities. Despite years of development in this field, it still remains challenging to design a unified network for processing various modalities ($\textit{e.g.}$ natural language, 2D images, 3D point clouds, audio, video, time series, tabular data) due to the inherent gaps among them. In this work, we propose a framework, named Meta-Transformer, that leverages a $\textbf{frozen}$ encoder to perform multimodal perception without any paired multimodal training data. In Meta-Transformer, the raw input data from various modalities are mapped into a shared token space, allowing a subsequent encoder with frozen parameters to extract high-level semantic features of the input data. Composed of three main components: a unified data tokenizer, a modality-shared encoder, and task-specific heads for downstream tasks, Meta-Transformer is the first framework to perform unified learning across 12 modalities with unpaired data. Experiments on different benchmarks reveal that Meta-Transformer can handle a wide range of tasks including fundamental perception (text, image, point cloud, audio, video), practical application (X-Ray, infrared, hyperspectral, and IMU), and data mining (graph, tabular, and time-series). Meta-Transformer indicates a promising future for developing unified multimodal intelligence with transformers. Code will be available at https://github.com/invictus717/MetaTransformer

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 13 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CSTrack: Enhancing RGB-X Tracking via Compact Spatiotemporal Features

    cs.CV 2025-05 conditional novelty 7.0 of 10

    CSTrack proposes compact spatial and temporal feature modules for RGB-X tracking, reporting new state-of-the-art results on DepthTrack, VOT-RGBD2022, LasHeR, RGBT234, and VisEvent.

  2. Mixture of Probes: Learning from Privileged Modalities in Multimodal LLMs Through Probing

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Structured intermediate probing that separates modality-specific from modality-general signals lets privileged training modalities improve single-modality MLLM inference by large margins over naive multimodal training.

  3. UniPre3D: Unified Pre-training of 3D Point Cloud Models with Cross-Modal Gaussian Splatting

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A point cloud pre-training method that uses 3D Gaussian splatting rendering and cross-modal image features to work for both objects and scenes.

  4. VisCon-100K: Leveraging Contextual Web Data for Fine-tuning Vision Language Models

    cs.CL 2025-02 conditional novelty 6.0 of 10

    A web-context-derived dataset and a 'leaky modality mix' of captions with Q&A pairs improve vision-language model fine-tuning on several benchmarks.

  5. Toward Long-Tailed Online Anomaly Detection through Class-Agnostic Concepts

    cs.CV 2025-07 conditional novelty 5.0 of 10

    The paper proposes the long-tailed online anomaly detection (LTOAD) benchmark and a class-agnostic concept-based framework that outperforms class-aware baselines in most offline settings and in the online setting.

  6. SkySense V2: A Unified Foundation Model for Multi-modal Remote Sensing

    cs.CV 2025-07 conditional novelty 5.0 of 10

    A unified multi-modal remote sensing foundation model with adaptive patch merging, modality prompt tokens, mixture of experts, and query-based semantic aggregation contrastive learning outperforms SkySense by 1.8 poin...

  7. OmniVec2 -- A Novel Transformer based Network for Large Scale Multimodal and Multitask Learning

    cs.CV 2025-07 conditional novelty 5.0 of 10

    A shared-backbone transformer with pairwise modality training reports top results across 25 datasets spanning 12 modalities.

  8. CoNav: Collaborative Cross-Modal Reasoning for Embodied Navigation

    cs.CV 2025-05 conditional novelty 5.0 of 10

    CoNav lets a frozen 3D-text model pass spatial text hints to a lightly fine-tuned image-text navigation agent, improving path efficiency on several VLN benchmarks, though not all claimed state-of-the-art results hold.

  9. Fine-tuning Multimodal Transformers on Edge: A Parallel Split Learning Approach

    cs.DC 2025-02 reject novelty 5.0 of 10

    MPSL applies parallel split learning with a single aggregated server backward pass to fine-tune multimodal transformers, matching federated learning accuracy while reducing client compute by about 250x.

  10. Grounding Intelligence in Movement

    cs.AI 2025-07 conditional novelty 4.0 of 10

    Movement should be treated as a first-class AI modeling modality, and a unified, biomechanically grounded movement foundation model built from aggregated data across species and sensors is the proposed path forward.

  11. Multimodal Representation Alignment for Cross-modal Information Retrieval

    cs.IR 2025-06 conditional novelty 4.0 of 10

    Across CLIP, BLIP, Meta-Transformer, and three combined unimodal models on IMDB, Flickr30K, and MS-COCO, cosine similarity gives the best cross-modal retrieval for contrastively trained models, while learned MLP align...

  12. Vision Generalist Model: A Survey

    cs.CV 2025-06 conditional novelty 3.0 of 10

    A structured review of vision generalist models, classifying them into encoding-based and sequence-to-sequence frameworks and summarizing datasets, benchmarks, techniques, and open problems.

  13. A Comprehensive Survey on Video Scene Parsing:Advances, Challenges, and Prospects

    cs.CV 2025-06 conditional novelty 2.0 of 10

    A comprehensive survey of video scene parsing that organizes methods, datasets, metrics, and benchmark results across VSS, VIS, VPS, VTS, and OVVS.

Pith tools