REVIEW 6 cited by
Enhancing Multimodal Unified Representations for Cross Modal Generalization
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
To enhance the interpretability of multimodal unified representations, many studies have focused on discrete unified representations. These efforts typically start with contrastive learning and gradually extend to the disentanglement of modal information, achieving solid multimodal discrete unified representations. However, existing research often overlooks two critical issues: 1) The use of Euclidean distance for quantization in discrete representations often overlooks the important distinctions among different dimensions of features, resulting in redundant representations after quantization; 2) Different modalities have unique characteristics, and a uniform alignment approach does not fully exploit these traits. To address these issues, we propose Training-free Optimization of Codebook (TOC) and Fine and Coarse cross-modal Information Disentangling (FCID). These methods refine the unified discrete representations from pretraining and perform fine- and coarse-grained information disentanglement tailored to the specific characteristics of each modality, achieving significant performance improvements over previous state-of-the-art models. The code is available at https://github.com/haihuangcode/CMG.
Forward citations
Cited by 6 Pith papers
-
TAP: Parameter-efficient Task-Aware Prompting for Adverse Weather Removal
A two-stage prompt-tuning method with low-rank and contrastive prompt enhancement claims all-in-one adverse weather removal at 2.75M parameters.
-
Bridging Domain Generalization to Multimodal Domain Generalization via Unified Representations
By aligning category-level information across video, audio, and flow into one unified representation while keeping modality-specific details separate, this paper shows that standard domain generalization methods impro...
-
IRBridge: Solving Image Restoration Bridge with Pre-trained Generative Diffusion Models
A Gaussian-path transition equation lets a pretrained Stable Diffusion model serve as the denoiser inside image restoration bridges, cutting per-task training to a lightweight ControlNet.
-
Offline Map Matching Based on Localization Error Distribution Modeling
LNSP models city-wide GPS error distributions from fixed bus routes and uses them, plus detour detection, to match sparse trajectories more accurately than three existing map matching methods.
-
Open-set Cross Modal Generalization via Multimodal Unified Representation
The authors propose OSCMG, an open-set version of Cross Modal Generalization, and show their MICU method with masked contrastive learning and unified jigsaw puzzles outperforms prior methods.
-
MANTA: Cross-Modal Semantic Alignment and Information-Theoretic Optimization for Long-form Multimodal Understanding
A multimodal retrieval pipeline that projects video and audio into text and claims near-optimal context selection, with reported gains of up to 22.6% on Video-MME that rest on circular theory and unreleased data.
Discussion (0). Continue with ORCID to comment.