Pith. sign in

REVIEW 7 major objections 6 minor 50 references

Synergistic Prompting for Robust Visual Recognition with Missing Modalities

T0 review · 7 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SyP combines static and dynamic prompts to keep visual recognition accurate when image or text inputs are missing, and reports consistent gains over five prompt-based baselines on three datasets.

desk verdict Solid-looking gains, but the claimed per-modality re-weighting mechanism is not in the equations: Eq. (3) applies one shared scaling vector to both modalities. read the letter →

arxiv 2507.07802 v2 pith:YJMUXZNT submitted 2025-07-10 cs.CV

Zhihui Zhang , Luanyuan Dai , Qika Lin , Yunfeng Diao , Guangyin Jin , Yufei Guo , Jing Zhang , Xiaoshuai Hao This is my paper · ORCID
classification cs.CV
keywords missingmodalitiesvisualrecognitionpromptlearningdynamicadaptersynergisticpromptsvision-languagemodelsCLIProbustness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that prompt-based methods for missing-modality visual recognition fail because their prompts are static: the same learnable embeddings are used no matter which input is absent or how much is missing. It proposes SyP (Synergistic Prompting), in which a dynamic adapter reads the available image and text features and outputs a scaling factor that modulates base prompts, while a parallel static prompt supplies stable cross-modal cues. On MM-IMDb, UPMC Food-101, and Hateful Memes, SyP is reported to beat five prior prompt methods at missing rates of 50%, 70%, and 90%, including settings where both modalities are incomplete. A sympathetic reader would take away that adapting prompts to each input's missing pattern is a cheap, effective way to keep a frozen CLIP backbone useful under incomplete data.

What carries the argument

The load-bearing object is the dynamic adapter, a small MLP that maps the concatenated image and text feature vector to a scalar scaling factor through a ReLU bottleneck and a sigmoid, so the factor lies in [0,1]. The adapter's output is applied element-wise to modality-specific base prompts, yielding dynamic prompts that strengthen the available modality and weaken the absent one; these are summed with projected static prompts to form the final synergistic prompts, and a layer-wise propagation re-applies the combination at each of six transformer layers. This mechanism lets a frozen CLIP backbone behave as a recognizer whose prompt content changes per input and per missing pattern without retraining the backbone.

What would settle it

Train SyP on the three datasets with the missing modality's tokens removed from the attention computation instead of zero-filled, holding all other settings fixed; if the masked variant matches or exceeds SyP, the zero-fill assumption is not load-bearing, and if it falls short, the adapter is exploiting zero patterns rather than merely re-weighting available features.

Watch

Extended reading notes

Core claim

The central claim is that a missing modality is best handled by generating prompts on the fly rather than fixing them at training time. SyP builds a concatenated feature vector from the image and text encoders, passes it through an MLP with ReLU and sigmoid activations to obtain a scaling factor in [0,1], and multiplies that factor into learned base prompts to create dynamic prompts. These dynamic prompts are added element-wise to static prompts projected into each modality, and the combined synergistic prompts are prepended to input tokens at multiple transformer layers while only the prompts and a final fully connected layer are updated. The paper reports that this design outperforms CoOp, MMP, MaPLe, DePT, and DCP on all three datasets, with the largest margins at high missing rates and in settings where both modalities are missing.

Load-bearing premise

The results assume that a missing modality can be represented by a zero-filled tensor and that the dynamic adapter, which sees those zeros as input, still computes meaningful scaling factors; this choice is never tested against masking or imputation of the absent input.

Editorial extensions

If this is right

  • If the reported results hold, SyP establishes that input-dependent prompt scaling is enough to close most of the accuracy gap that missing modalities open, with no generative imputation and no joint-learning objective.
  • Static prompt baselines (CoOp, MMP, MaPLe, DePT) and the correlated prompt baseline DCP are all beaten on the three datasets, so the improvement is not confined to one task or one missing type.
  • On Hateful Memes the margin over DCP is largest at a 90% missing rate, suggesting the dynamic adapter matters most precisely when the signal is scarcest.
  • Because only the prompts and the final classifier are trained, the gains are obtained with a small parameter overhead on top of standard prompt tuning.
  • Since absent modalities are replaced by zero-filled tensors, the pipeline needs no special imputation module; the missing input contributes no signal and the scaling factor is left to compensate.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same adapter-plus-scaling recipe should extend to three or more modalities with no architectural change beyond lengthening the concatenated feature vector, since the paper's per-modality base prompts share one scaling factor; testing this extension is a direct check of the claimed generalizability.
  • An ablation that freezes the scaling factor at 1 while keeping the two-prompt structure would isolate the adapter's contribution; the paper's ablations remove whole components instead.
  • Comparing zero-filled missing tensors against masking missing tokens out of attention would show whether the dynamic adapter relies on the zero pattern or simply re-weights available features.
  • The large margins at high missing rates on Hateful Memes hint that this style of prompting could serve as a lightweight robustness wrapper for safety-sensitive multimodal classifiers, though the paper does not test that deployment setting.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

7 major / 6 minor

Summary. The paper proposes Synergistic Prompting (SyP), a prompt-tuning framework for CLIP-based visual recognition under missing-modality conditions. SyP combines static prompts with dynamic prompts generated by a Dynamic Adapter that computes a scaling factor from concatenated image and text features; prompts are propagated layer-wise, and only the prompts and a downstream classifier are trained. The method is evaluated on MM-IMDb, UPMC Food-101, and Hateful Memes under three missing-rate settings (50%, 70%, 90%) and three missing conditions (text missing, image missing, and a condition the paper calls 'both missing'), reporting consistent improvements over CoOp, MMP, MaPLe, DePT, and DCP, together with ablations and generalization analyses.

Significance. If the stated mechanism were correctly implemented, the contribution would be a practical, parameter-efficient way to adapt frozen vision-language models to incomplete inputs, with a useful separation of static and input-dependent prompt components. The paper has commendable breadth: it evaluates on three external benchmarks, includes ablations of the prompt components and the dynamic adapter, reports a hyperparameter sensitivity table, and provides generalization experiments across missing rates. However, the central claimed mechanism of adaptive per-modality re-weighting is not present in the equations as written, and the training details are internally contradictory. These issues affect the interpretation of the empirical results and must be resolved before the paper can be accepted.

major comments (7)
  1. [Sec. 3.2-3.3, Eqs. (2)-(3)] The paper's core claim is that SyP 'dynamically re-weights modality contributions' and that 'when a modality is absent, the scaling factor increases the corresponding prompt's weight.' However, Eq. (2) produces a single scaling vector S_d from the concatenated features X_C = [X_I, X_T], and Eq. (3) applies this same S_d to both base prompts: P_I,D = P_I,B ⊙ S_d and P_T,D = P_T,B ⊙ S_d. A shared vector cannot increase one modality's prompt weight relative to the other; it scales both prompt sets identically, so the relative weights are fixed by the base prompts. The described per-modality adaptive re-weighting is therefore not implemented by the equations. Either the implementation contains separate per-modality scaling factors that are missing from Eq. (3), in which case the equations must be corrected, or the dynamic adapter's benefit must be attributed to some other mechanism, in which case the central narrative should be revised.
  2. [Sec. 4.1 vs. Appendix B] The training configurations are contradictory. Section 4.1 states AdamW with learning rate 1e-3, 20 epochs, batch size 32, and learnable prompts of length Lp=36 added to M=6 layers, while Appendix B reports batch size 256 for all datasets, learning rates of 1e-2 (Hateful Memes, Food101) and 1e-4 (MM-IMDb), and 20/200/100 epochs for Hateful Memes, Food101, and MM-IMDb respectively, with the prompt depth described as 36. These differences prevent reproduction and make it unclear which configuration produced Table 1. The authors must reconcile these statements and report the exact settings used for the main results.
  3. [Sec. 4.1, definition of 'missing both'] The 'missing both' condition is defined as η/2 percent text-only and η/2 percent image-only instances, with the remainder complete. This is not a condition in which both modalities are absent from any sample; it is a mixture of the two single-modality missing cases. Therefore statements such as 'even when both modalities are absent' (Fig. 3 caption and Sec. 5) and the claim of robustness 'when both modalities are missing' are not supported by the experimental protocol. The terminology should be corrected or the protocol should be changed to one where truly modality-absent inputs are handled.
  4. [Sec. 4.1 and Tab. 5] Missing modalities are replaced with zero-filled tensors, and the dynamic adapter receives these zero-filled features as input. The paper never validates this choice against alternative imputation strategies or masking the missing modality entirely. Since the adapter's input includes a zero vector for the missing modality, the computed scaling factor may be responding primarily to the presence/absence indicator rather than to the content of the available modality. A diagnostic comparing zero-filling, learned imputation, and input masking is needed to support the claim that the adapter 'computes adaptive scaling factors based on available modality features.'
  5. [Tab. 5 and Table 1] The reduction ratio r is tuned separately per dataset (r=5 for Food101, r=10 for MM-IMDb, r=16 for Hateful Memes) without stating a held-out selection protocol, and no error bars or multiple-seed results are reported anywhere in the paper. Given that the main-table gains are sometimes a few points and that the adapter's effect is the central contribution, the authors should specify how r was chosen (e.g., validation-based selection), report performance of the final configuration across at least three seeds, and show that the reported gains are not an artifact of per-dataset tuning.
  6. [Sec. 4.2 and Fig. 5] The paper claims that SyP 'even improved performance in text-missing and both-missing cases at higher missing rates.' This is a counterintuitive result that deserves explicit explanation and verification. In Table 1, several SyP numbers indeed rise with η (e.g., Hateful Memes text-missing AUROC 68.25 at η=50, 68.94 at η=70, 69.70 at η=90), but the paper does not explain why a higher fraction of missing data should improve performance. Without a mechanism or additional analysis, this claim appears fragile and may reflect dataset-specific label leakage or evaluation noise.
  7. [Sec. 4.3, Tab. 2 and Tab. 3] The ablation tables report only the best configuration and do not indicate whether the comparisons between variants are statistically significant. Since the differences between SyP (w/ Synergistic Prompts) and SyP (w/ only Dynamic Prompt) are small on Food101 and MM-IMDb (e.g., 86.17 vs. 82.90 and 54.72 vs. 51.21), the claim that the synergistic combination is responsible for the gains would be strengthened by reporting variance across seeds or a paired test.
minor comments (6)
  1. [Sec. 4.1] The heading 'Setating of Missing Pattern' contains a typo; it should read 'Setting of Missing Pattern.'
  2. [Fig. 5 caption] The caption says 'compared with sate-of-the-art baselines'; 'sate-of-the-art' should be 'state-of-the-art.'
  3. [Sec. 3.5, Eq. (9)] Equation (9) defines the total loss as a sum over N samples, but the text describes a per-sample loss; this should be clarified as an average over a minibatch or a sum over training samples with appropriate normalization.
  4. [Sec. 3.5, Eq. (10)] Equation (10) defines P_final as the fully connected output of concatenated prompts P_I^{Ri-1} and P_T^{Ri-1}, whereas the surrounding text and Fig. 2 indicate that the task-related tokens from both encoders are concatenated for prediction. These two descriptions of the classifier input are inconsistent and should be reconciled.
  5. [Sec. 4.1 and Appendix B] The notation for prompt configuration is inconsistent: the main text says 'prompts of length Lp = 36 added to features from M = 6 layers,' while Appendix B says 'prompt depth as 36.' The authors should clarify whether 36 is the prompt token length, the number of layers, or both, and align the terminology throughout.
  6. [Appendix D] The t-SNE visualization is presented without a quantitative clustering metric (e.g., silhouette score or nearest-neighbor accuracy). As an illustrative figure it is acceptable, but the associated claims about 'accurate recognition of similar samples' should be backed by numbers if used as evidence.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the method is an empirical architecture tested on external benchmarks, and the noted Eq. (3) issue is an internal-consistency concern, not a circular derivation.

full rationale

The paper proposes a trainable dynamic adapter that computes a scaling factor S_d from concatenated image/text features and applies it to base prompts. No quantity in Eqs. (2)-(5) is defined in terms of the target metric, and the training objective in Eq. (9) is a standard supervised loss evaluated on external datasets (MM-IMDb, UPMC Food-101, Hateful Memes). The only fitted quantities are prompt parameters and hyperparameters such as the reduction ratio r; the reduction-ratio study in Table 5 is presented as hyperparameter tuning, not as a prediction from first principles. Comparisons are made against external baselines (CoOp, MMP, MaPLe, DePT, DCP), so the central performance claim is not reduced to a self-citation or to the method's own construction. The authors' prior work appears only in related-work citations and is not used to justify the core mechanism. The skeptic's concern about Eq. (3) is a legitimate internal-consistency question: a single shared scaling vector cannot independently enlarge one modality's prompt relative to the other, so the stated 'adaptive modality re-weighting' is not realized exactly as written unless the implementation uses separate per-modality factors omitted from the equations. However, that is a correctness/consistency issue, not circularity; it does not make the empirical result equivalent to the input by construction. No step in the paper's derivation chain reduces to its own inputs or to an author-imported uniqueness theorem.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The paper rests on the empirical assumption that prompt scaling and concatenation of available features can compensate for missing inputs. It introduces no theoretical mechanism, and its auxiliary hyperparameters are chosen per dataset.

free parameters (5)
  • Reduction ratio r = 16 (Hateful Memes), 5 (Food101), 10 (MM-IMDb)
    Hyperparameter controlling the bottleneck width of the dynamic adapter MLP (Eq. 2); chosen per dataset in Tab. 5, with no held-out selection protocol.
  • Prompt length Lp = 36
    Number of prompt tokens per layer, chosen by hand in Sec. 4.1; the appendix calls it 'prompt depth', so the role is ambiguous.
  • Number of prompt layers M = 6
    Transformer layers that receive prompts, chosen by hand in Sec. 4.1.
  • Learning rate = 1e-3 (main), 1e-2 or 1e-4 (appendix)
    Training hyperparameter reported inconsistently across Sec. 4.1 and Appendix B, affecting reproducibility.
  • Batch size = 32 (main), 256 (appendix)
    Training hyperparameter reported inconsistently across Sec. 4.1 and Appendix B, affecting reproducibility.
assumptions (4)
  • domain assumption A missing modality can be represented by a zero-filled tensor and this remains a useful input to the frozen CLIP encoder and dynamic adapter.
    Sec. 4.1 states 'missing modalities are replaced with zero-filled tensors'. No ablation validates this choice against masks, learned placeholders, or imputation.
  • domain assumption The concatenation of image and text features (including zeros for the absent modality) carries enough signal for the adapter to compute useful scaling factors.
    Eqs. (1)-(2) define the scaling factor from concatenated features; if zeros dominate, the scaling factor could be constant or uninformative.
  • ad hoc to paper The 'missing both' protocol accurately reflects the intended missing-modality scenario.
    Sec. 4.1 defines 'missing both' as either text-only or image-only at equal rates, not both absent. The figures and text refer to it as 'both missing', which is a misleading re-labeling.
  • domain assumption Prompt tokens prepended to frozen CLIP layers are sufficient to elicit robust missing-modality behavior without any adaptation of the backbone parameters.
    Sec. 3.1 freezes CLIP; all adaptation comes from prompts and a final FC layer.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Synergistic Prompting for Robust Visual Recognition with Missing Modalities." pith.science (2026). https://pith.science/paper/YJMUXZNT

@misc{pith2026250707802,
  author       = {Pith},
  title        = {Pith review of: Synergistic Prompting for Robust Visual Recognition with Missing Modalities},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YJMUXZNT}},
  note         = {Machine review of arXiv:2507.07802}
}
read the original abstract

Large-scale multi-modal models have demonstrated remarkable performance across various visual recognition tasks by leveraging extensive paired multi-modal training data. However, in real-world applications, the presence of missing or incomplete modality inputs often leads to significant performance degradation. Recent research has focused on prompt-based strategies to tackle this issue; however, existing methods are hindered by two major limitations: (1) static prompts lack the flexibility to adapt to varying missing-data conditions, and (2) basic prompt-tuning methods struggle to ensure reliable performance when critical modalities are missing.To address these challenges, we propose a novel Synergistic Prompting (SyP) framework for robust visual recognition with missing modalities. The proposed SyP introduces two key innovations: (I) a Dynamic Adapter, which computes adaptive scaling factors to dynamically generate prompts, replacing static parameters for flexible multi-modal adaptation, and (II) a Synergistic Prompting Strategy, which combines static and dynamic prompts to balance information across modalities, ensuring robust reasoning even when key modalities are missing. The proposed SyP achieves significant performance improvements over existing approaches across three widely-used visual recognition datasets, demonstrating robustness under diverse missing rates and conditions. Extensive experiments and ablation studies validate its effectiveness in handling missing modalities, highlighting its superior adaptability and reliability.

Figures

Figures reproduced from arXiv: 2507.07802 by the authors.

Figure 1
Figure 1. Comparison of Prior Static Prompt Methods and Our SyP for Incomplete Multi-Modal Learning. SyP integrates static and dynamic prompts to balance information across modalities, ensuring robust reasoning even under missing key modalities. learning has made remarkable progress [12, 48] in various fields such as cross-modal retrieval [10, 11, 16], caption￾ing [5, 22, 37], and visual question answering [27, 29, 36], large… view at source ↗
Figure 2
Figure 2. Overall Framework of Synergistic Prompting (SyP). For each input, modality-specific prompts P I m and P T m are generated, where m ∈ {c, m1, m2} denotes the missing modality type. The prompts consist of dynamic prompts PI,D(PT,D), generated via a dynamic adapter, and static prompts PI,S(PT,S), integrated through a synergistic strategy. These prompts are prepended to input tokens xm1 and xm2 , forming a unified seque… view at source ↗
Figure 3
Figure 3. Generalization analysis on the Hateful Memes dataset across various missing rates in terms of AUROC [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Generalizability Analysis of Our Method to Different Missing Rates on MM-IMDB dataset. (a) Models are trained on missing-both or missing-text cases, and evaluated on missing-text cases with different missing rates. (b) Models are trained on missing-both or missing-imag…
Figure 5
Figure 5. Figure 5: Analysis of Robustness to Different Missing Rates compared with sate-of-the-art baselines on Hateful Memes dataset. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Visualization of the samples in three dataset. [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Generalization analysis on the Food101 dataset across various missing rates in terms of Accuracy. [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Generalization analysis on the MM-IMDb dataset across various missing rates in terms of F1_Macro. [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: Generalizability Analysis of Our Method to Different Missing Rates on Hate Memes dataset. (a) Models are trained on missing-both or missing-text cases, and evaluated on missing-text cases with different missing rates. (b) Models are trained on missing-both or missing-i…
Figure 10
Figure 10. Figure 10: Generalizability Analysis of Our Method to Different Missing Rates on Food101 dataset. (a) Models are trained on missing-both or missing-text cases, and evaluated on missing-text cases with different missing rates. (b) Models are trained on missing-both or missing-ima…
Figure 11
Figure 11. Figure 11: t-SNE visualization of our model and other baselines on the MM-IMDb dataset under a 50% both missing rate. [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

50 extracted references · 39 canonical work pages

  1. [1]

    Multimodal federated learning in aiot systems: Existing solutions, applications, and challenges

    Christos Anagnostopoulos, Alexandros Gkillas, Christos Mavrokefalidis, Erion-Vasilis Pikoulis, Nikos Piperigkos, and Aris S Lalos. Multimodal federated learning in aiot systems: Existing solutions, applications, and challenges. IEEE Access,

    work page

  2. [2]

    Gated multimodal units for information fusion

    John Arevalo, Thamar Solorio, Manuel Montes-y Gómez, and Fabio A González. Gated multimodal units for information fusion. arXiv preprint arXiv:1702.01992, 2017. 5, 6, 12

    arXiv 2017

  3. [3]

    Vivit: A video vision transformer

    Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Luˇci´c, and Cordelia Schmid. Vivit: A video vision transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6836–6846, 2021. 1

    work page 2021

  4. [4]

    Layer normalization

    Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016. 5

    arXiv 2016

  5. [5]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. In European Conference on Computer Vision, pages 370–387,

    work page

  6. [6]

    A fine-grained self- adapting prompt learning approach for few-shot learning with pre-trained language models

    Xiaojun Chen, Ting Liu, Philippe Fournier-Viger, Bowen Zhang, Guodong Long, and Qin Zhang. A fine-grained self- adapting prompt learning approach for few-shot learning with pre-trained language models. Knowledge-Based Systems, 299: 111968, 2024. 2

    work page 2024

  7. [7]

    Retrieval-augmented hypergraph for multimodal social media popularity prediction

    Zhangtao Cheng, Jienan Zhang, Xovee Xu, Goce Trajcevski, Ting Zhong, and Fan Zhou. Retrieval-augmented hypergraph for multimodal social media popularity prediction. In Pro- ceedings of the 30th ACM SIGKDD conference on knowledge discovery and data mining, pages 445–455, 2024. 1

    work page 2024

  8. [8]

    An image is worth 16x16 words: Trans- formers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Weissenborn, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. 5

    arXiv 2010

Show all 50 references
  1. [9]

    Mfhod: Multi-modal image fusion method based on the higher-order degradation model

    Jinxin Guo, Weida Zhan, Yichun Jiang, Wei Ge, Yu Chen, Xiaoyu Xu, Jin Li, and Yanyan Liu. Mfhod: Multi-modal image fusion method based on the higher-order degradation model. Expert Systems with Applications, 249:123731, 2024. 1, 2

    2024

  2. [10]

    Uncertainty-aware align- ment network for cross-domain video-text retrieval.Advances in Neural Information Processing Systems, 36:38284–38296,

    Xiaoshuai Hao and Wanqian Zhang. Uncertainty-aware align- ment network for cross-domain video-text retrieval.Advances in Neural Information Processing Systems, 36:38284–38296,

  3. [11]

    Dual alignment unsupervised domain adaptation for video- text retrieval

    Xiaoshuai Hao, Wanqian Zhang, Dayan Wu, Fei Zhu, and Bo Li. Dual alignment unsupervised domain adaptation for video- text retrieval. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18962–18972,

  4. [12]

    Mixgen: A new multi- modal data augmentation

    Xiaoshuai Hao, Yi Zhu, Srikar Appalaraju, Aston Zhang, Wanqian Zhang, Bo Li, and Mu Li. Mixgen: A new multi- modal data augmentation. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 379–389, 2023. 1

    2023

  5. [13]

    Mapfusion: A novel bev feature fusion network for multi-modal map construction

    Xiaoshuai Hao, Yunfeng Diao, Mengchuan Wei, Yifan Yang, Peng Hao, Rong Yin, Hui Zhang, Weiming Li, Shu Zhao, and Yu Liu. Mapfusion: A novel bev feature fusion network for multi-modal map construction. Information Fusion, 119: 103018, 2025. 2

    2025

  6. [14]

    Gaussian error linear units (gelus)

    Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016. 5

    2016 arXiv

  7. [15]

    Robobrain: A unified brain model for robotic manipulation from abstract to concrete

    Yuheng Ji, Huajie Tan, Jiayu Shi, Xiaoshuai Hao, Yuan Zhang, Hengyuan Zhang, Pengwei Wang, Mengdi Zhao, Yao Mu, Pengju An, et al. Robobrain: A unified brain model for robotic manipulation from abstract to concrete. In Proceedings of the Computer Vision and Pattern Recognition ...

    2025

  8. [16]

    Cross-modal implicit relation rea- soning and aligning for text-to-image person retrieval

    Ding Jiang and Mang Ye. Cross-modal implicit relation rea- soning and aligning for text-to-image person retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 2787–2797, 2023. 1

    2023

  9. [17]

    En- hancing industrial prognostic accuracy in noisy and missing data context: Assessing multimodal learning performance

    Sagar Jose, Khanh TP Nguyen, and Kamal Medjaher. En- hancing industrial prognostic accuracy in noisy and missing data context: Assessing multimodal learning performance. Journal of Intelligent Manufacturing, pages 1–25, 2024. 1

    2024

  10. [18]

    Exploring contactless tech- niques in multimodal emotion recognition: insights into di- verse applications, challenges, solutions, and prospects

    Umair Ali Khan, Qianru Xu, Yang Liu, Altti Lagstedt, Ari Alamäki, and Janne Kauttonen. Exploring contactless tech- niques in multimodal emotion recognition: insights into di- verse applications, challenges, solutions, and prospects. Mul- timedia Systems, 30(3):115, 2024. 1

    2024

  11. [19]

    Maple: Multi- modal prompt learning

    Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi- modal prompt learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 19113–19122, 2023. 2, 5, 6, 14

    2023

  12. [20]

    The hateful memes challenge: Detecting hate speech in multimodal memes

    Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. The hateful memes challenge: Detecting hate speech in multimodal memes. Advances in neural information processing systems, pages 2611–2624, 2020. 1, 5, 6, 12

    2020

  13. [21]

    Missing modality predic- tion for unpaired multimodal learning via joint embedding of unimodal models

    Donggeun Kim and Taesup Kim. Missing modality predic- tion for unpaired multimodal learning via joint embedding of unimodal models. In European Conference on Computer Vision, pages 171–187, 2024. 1, 2

    2024

  14. [22]

    Do you remember? dense video caption- ing with cross-modal memory retrieval

    Minkuk Kim, Hyeon Bae Kim, Jinyoung Moon, Jinwoo Choi, and Seong Tae Kim. Do you remember? dense video caption- ing with cross-modal memory retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13894–13904, 2024. 1

    2024

  15. [23]

    Retrieval-augmented dynamic prompt tuning for incomplete multimodal learning

    Jian Lang, Zhangtao Cheng, Ting Zhong, and Fan Zhou. Retrieval-augmented dynamic prompt tuning for incomplete multimodal learning. arXiv preprint arXiv:2501.01120, 2025. 2

    2025 arXiv

  16. [24]

    Cross-modal prototype based multimodal federated learning under severely missing modality

    Huy Q Le, Chu Myaet Thwal, Yu Qiao, Ye Lin Tun, Minh NH Nguyen, and Choong Seon Hong. Cross-modal prototype based multimodal federated learning under severely missing modality. arXiv preprint arXiv:2401.13898, 2024. 1, 2

    2024 arXiv

  17. [25]

    Multimodal prompting with missing modalities for vi- sual recognition

    Yi-Lun Lee, Yi-Hsuan Tsai, Wei-Chen Chiu, and Chen-Yu Lee. Multimodal prompting with missing modalities for vi- sual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14943– 14952, 2023. 2, 5, 6, 14

    2023

  18. [26]

    What foundation models can bring for robot learning in manipulation: A survey

    Dingzhe Li, Yixiang Jin, Yuhao Sun, Hongze Yu, Jun Shi, Xiaoshuai Hao, Peng Hao, Huaping Liu, Fuchun Sun, Jian- wei Zhang, et al. What foundation models can bring for robot learning in manipulation: A survey. arXiv preprint arXiv:2404.18201, 2024. 2

    2024

  19. [27]

    How to configure good in-context sequence for visual question answering

    Li Li, Jiawei Peng, Huiyi Chen, Chongyang Gao, and Xu Yang. How to configure good in-context sequence for visual question answering. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 26710–26720, 2024. 1

    2024

  20. [28]

    Tackling real-world complex- ity: Hierarchical modeling and dynamic prompting for multi- modal long document classification

    Tengfei Liu, Yongli Hu, Mingjie Li, Junfei Yi, Xiaojun Chang, Junbin Gao, and Baocai Yin. Tackling real-world complex- ity: Hierarchical modeling and dynamic prompting for multi- modal long document classification. IEEE Transactions on Circuits and Systems for Video Technology...

    2025

  21. [29]

    Cross-modal causal relational reasoning for event-level visual question answer- ing

    Yang Liu, Guanbin Li, and Liang Lin. Cross-modal causal relational reasoning for event-level visual question answer- ing. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(10):11624–11641, 2023. 1

    2023

  22. [30]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021. 1

    2021

  23. [31]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 5, 11

    2017 arXiv

  24. [32]

    Multi-modal modality-masked diffusion network for brain mri synthesis with random modality missing

    Xiangxi Meng, Kaicong Sun, Jun Xu, Xuming He, and Ding- gang Shen. Multi-modal modality-masked diffusion network for brain mri synthesis with random modality missing. IEEE Transactions on Medical Imaging, 2024. 1, 2

    2024

  25. [33]

    Semantically conditioned prompts for visual recognition under missing modality scenarios

    Vittorio Pipoli, Federico Bolelli, Sara Sarto, Marcella Cornia, Lorenzo Baraldi, Costantino Grana, Rita Cucchiara, Elisa Ficarra, et al. Semantically conditioned prompts for visual recognition under missing modality scenarios. In Proceed- ings of the IEEE/CVF Winter Conference...

    2025

  26. [34]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  27. [35]

    Deep correlated prompting for visual recognition with miss- ing modalities

    Tongkai Shi, Wei Feng, Fanhua Shang, Liang Wan, et al. Deep correlated prompting for visual recognition with miss- ing modalities. Advances in Neural Information Processing Systems, pages 67446–67466, 2025. 2, 5, 6, 14

    2025

  28. [36]

    Drivelm: Driving with graph visual question answering

    Chonghao Sima, Katrin Renz, Kashyap Chitta, Li Chen, Hanxue Zhang, Chengen Xie, Jens Beißwenger, Ping Luo, Andreas Geiger, and Hongyang Li. Drivelm: Driving with graph visual question answering. In European Conference on Computer Vision, pages 256–274, 2024. 1

    2024

  29. [37]

    Emotional video captioning with vision-based emo- tion interpretation network

    Peipei Song, Dan Guo, Xun Yang, Shengeng Tang, and Meng Wang. Emotional video captioning with vision-based emo- tion interpretation network. IEEE Transactions on Image Processing, pages 1122–1135, 2024. 1

    2024

  30. [38]

    Strong and weak prompt engineer- ing for remote sensing image-text cross-modal retrieval.IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2025

    Tianci Sun, Chengyu Zheng, Xiu Li, Yanli Gao, Jie Nie, Lei Huang, and Zhiqiang Wei. Strong and weak prompt engineer- ing for remote sensing image-text cross-modal retrieval.IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2025. 2

    2025

  31. [39]

    Reason- rft: Reinforcement fine-tuning for visual reasoning

    Huajie Tan, Yuheng Ji, Xiaoshuai Hao, Minglan Lin, Pengwei Wang, Zhongyuan Wang, and Shanghang Zhang. Reason- rft: Reinforcement fine-tuning for visual reasoning. arXiv preprint arXiv:2503.20752, 2025. 2

    2025

  32. [40]

    Visualizing data using t-sne

    Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11),

  33. [41]

    Recipe recognition with large mul- timodal food dataset

    Xin Wang, Devinder Kumar, Nicolas Thome, Matthieu Cord, and Frederic Precioso. Recipe recognition with large mul- timodal food dataset. In IEEE International Conference on Multimedia & Expo Workshops, pages 1–6, 2015. 1, 5, 6, 12

    2015

  34. [42]

    Deep multimodal learning with missing modal- ity: A survey

    Renjie Wu, Hu Wang, Hsiang-Ting Chen, and Gustavo Carneiro. Deep multimodal learning with missing modal- ity: A survey. arXiv preprint arXiv:2409.07825, 2024. 1

    2024 arXiv

  35. [43]

    Consistent prompt tuning for generalized category discovery

    Muli Yang, Jie Yin, Yanan Gu, Cheng Deng, Hanwang Zhang, and Hongyuan Zhu. Consistent prompt tuning for generalized category discovery. International Journal of Computer Vision, pages 1–28, 2025. 2

    2025

  36. [44]

    Cpt: Colorful prompt tuning for pre-trained vision-language models

    Yuan Yao, Ao Zhang, Zhengyan Zhang, Zhiyuan Liu, Tat- Seng Chua, and Maosong Sun. Cpt: Colorful prompt tuning for pre-trained vision-language models. AI Open, 5:30–38,

  37. [45]

    Dept: Decoupled prompt tuning

    Ji Zhang, Shihan Wu, Lianli Gao, Heng Tao Shen, and Jingkuan Song. Dept: Decoupled prompt tuning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12924–12933, 2024. 2, 5, 6, 14

    2024

  38. [46]

    Unified multi-modal diagnostic framework with reconstruc- tion pre-training and heterogeneity-combat tuning

    Yupei Zhang, Li Pan, Qiushi Yang, Tan Li, and Zhen Chen. Unified multi-modal diagnostic framework with reconstruc- tion pre-training and heterogeneity-combat tuning. IEEE Journal of Biomedical and Health Informatics, 2024. 2

    2024

  39. [47]

    Missing modality imagination network for emotion recognition with uncertain missing modalities

    Jinming Zhao, Ruichen Li, and Qin Jin. Missing modality imagination network for emotion recognition with uncertain missing modalities. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference on Na...

    2021

  40. [48]

    A survey on safe multi-modal learning systems

    Tianyi Zhao, Liangliang Zhang, Yao Ma, and Lu Cheng. A survey on safe multi-modal learning systems. In Proceed- ings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 6655–6665, 2024. 1

    2024

  41. [49]

    Learning to prompt for vision-language models

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. Interna- tional Journal of Computer Vision, pages 2337–2348, 2022. 2, 5, 6, 14

    2022

  42. [50]

    Dynamic adapter meets prompt tuning: Parameter-efficient transfer learning for point cloud analysis

    Xin Zhou, Dingkang Liang, Wei Xu, Xingkui Zhu, Yihan Xu, Zhikang Zou, and Xiang Bai. Dynamic adapter meets prompt tuning: Parameter-efficient transfer learning for point cloud analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages...

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.