Pith. sign in

REVIEW 5 cited by

GeminiFusion: Efficient Pixel-wise Multimodal Fusion for Vision Transformer

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.01210 v2 pith:HPYUXF6I submitted 2024-06-03 cs.CV

Ding Jia , Jianyuan Guo , Kai Han , Han Wu , Chao Zhang , Chang Xu , Xinghao Chen This is my paper · ORCID
classification cs.CV
keywords geminifusionfusionmultimodalacrosscomputationalcross-modaldemonstrateexchange
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Cross-modal transformers have demonstrated superiority in various vision tasks by effectively integrating different modalities. This paper first critiques prior token exchange methods which replace less informative tokens with inter-modal features, and demonstrate exchange based methods underperform cross-attention mechanisms, while the computational demand of the latter inevitably restricts its use with longer sequences. To surmount the computational challenges, we propose GeminiFusion, a pixel-wise fusion approach that capitalizes on aligned cross-modal representations. GeminiFusion elegantly combines intra-modal and inter-modal attentions, dynamically integrating complementary information across modalities. We employ a layer-adaptive noise to adaptively control their interplay on a per-layer basis, thereby achieving a harmonized fusion process. Notably, GeminiFusion maintains linear complexity with respect to the number of input tokens, ensuring this multimodal framework operates with efficiency comparable to unimodal networks. Comprehensive evaluations across multimodal image-to-image translation, 3D object detection and arbitrary-modal semantic segmentation tasks, including RGB, depth, LiDAR, event data, etc. demonstrate the superior performance of our GeminiFusion against leading-edge techniques. The PyTorch code is available at https://github.com/JiaDingCN/GeminiFusion

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Weaving Light and Time: Unified Harmonic-Geometric Representation Learning for Dense RGB-Event Parsing

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Evita, a unified RGB-Event backbone with geometric rectification, spectral resonance, and transient routing, plus N-ImageNetV2 pretraining, reports SOTA dense parsing with better accuracy-latency trade-offs.

  2. URNet: A Unified Reparameterized Network for Efficient RGB-D Semantic Segmentation

    cs.CV 2026-08 conditional novelty 4.0 of 10

    A unified, reparameterized single-encoder RGB-D segmentation network with linear gated attention and a pyramid merging decoder achieves strong accuracy-efficiency trade-offs on NYUDepth V2 and SUN-RGBD.

  3. Efficient Segment Anything with Depth-Aware Fusion and Limited Training Data

    cs.CV 2026-02 reject novelty 4.0 of 10

    Adding monocular depth to EfficientViT-SAM improves point-prompted segmentation at 3 and 5 clicks after fine-tuning on 11.2k images, but universal gains and data-efficiency are not established.

  4. DCIRNet: Depth Completion with Iterative Refinement for Dexterous Grasping of Transparent and Reflective Objects

    cs.RO 2025-06 conditional novelty 4.0 of 10

    DCIRNet fuses RGB and sparse depth via a dual-branch Swin Transformer plus iterative spatial propagation, improving depth completion on DREDS/TransCG and raising DexGraspNetV2 grasp success from 38% to 82% on ten tran...

  5. Foundation Models for Clean Energy Forecasting: A Comprehensive Review

    eess.SY 2025-07 conditional novelty 3.0 of 10

    A survey of foundation model methods, data, and open problems for renewable energy forecasting, built from roughly 218 cited works.

Pith tools