REVIEW 4 major objections 5 minor 24 references
EPIC: Efficient Prompt Interaction for Text-Image Classification
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read EPIC shows that gated prompt exchange between frozen vision and text encoders beats prior prompt-based methods on food and visual-entailment classification while training only 2.0M parameters.
desk verdict Solid incremental prompt-interaction method with real gains, but the 'minimal memory/parameters' claim doesn't survive its own Table I and test-set tuning. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The Interaction Hub is the module that carries the argument. Given the temporal prompts $\hat{p}^l_m$ and $\hat{p}^l_{m'}$ of the two modalities on an interaction layer, it projects the other modality's prompt into the current one's space, computes intra-modality and inter-modality similarity maps with cosine similarity under ReLU, softmaxes each map into a gate, and outputs $p^{l+1}_m = z^l_m \cdot \hat{p}^l_m + (1-z^l_m)\cdot r^l_m \cdot \tilde{p}^l_{m'}$. The gates decide how much context to keep and how much cross-modal content to accept token by token. Because the prompt and projection parameters are shared across all interaction layers, adding more interaction layers does not increase the trainable parameter count.
What would settle it
Split UPMC-Food101 (or SNLI-VE) into training and validation, tune interaction-layer indices, similarity type, and prompt length only on the validation split, then evaluate the chosen configuration once on the untouched test set; if EPIC's margin over PMF shrinks to noise or reverses, the reported SOTA is an artifact of test-set-based selection.
Extended reading notes
Core claim
EPIC's central claim is that prompt-level interaction alone is sufficient for effective vision-language fusion: temporal prompts that take part in each encoder layer's self-attention can simultaneously act as a communication channel between the modalities. The Interaction Hub measures cosine similarity within and across the two prompt streams, turns these maps into softmax gates, and forms the next layer's prompt as a gated blend that keeps the modality's own context and admits the other modality's information only where correspondences are strong. With this mechanism the paper reports the best accuracy among prompt-based methods that share the same CLIP ViT-L/14 foundation model: 93.95% on UPMC-Food101, 73.45% on SNLI-VE, and 65.9/56.3 F1-micro/macro on MM-IMDB, using a 2.0M-parameter trainable set and minimal GPU memory during training.
Load-bearing premise
The performance claim rests on hyperparameters being chosen honestly: the interaction layers (4,5,6), the cosine similarity metric, and the prompt length were selected after inspecting results on the UPMC-Food101 test set, and the same test set is the headline benchmark, with no held-out validation split reported.
Editorial extensions
If this is right
- If the reported results hold, frozen-backbone prompt interaction can replace heavy fine-tuning and explicit fusion modules for text-image classification, cutting training memory to a small fraction of full fine-tuning.
- The same gated prompt-exchange pattern transfers to other multimodal classification tasks built on frozen encoders, including visual entailment and multi-label sentiment analysis.
- Because prompt and projection parameters are shared across interaction layers, increasing the number of interaction layers can improve accuracy without increasing the trainable parameter count.
- The single-interaction-layer variant, EPIC-small, already beats several prompt baselines on all three datasets, indicating the mechanism does not depend on stacking many interaction layers.
Reading between the lines
- A testable extension the authors do not run is to replace the cosine-similarity gates with a scalar learned mixing weight; if accuracy drops, token-level correspondence gating is doing the work, and if it does not, a simpler mechanism suffices.
- Because the interaction-layer choice (4,5,6) came from the UPMC-Food101 sweep in Figure 4, re-tuning per dataset on a true validation split could shift the ranking on SNLI-VE and MM-IMDB.
- The Interaction Hub is effectively a content-addressable exchange between prompts, so a natural probe is whether high gate values coincide with human-judged alignment between image and text.
- Since only prompts and the projection MLP are trained, EPIC could be a substrate for continual learning, where new tasks reuse the frozen backbone and exchange only the small prompt set.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes EPIC, a prompt-based interaction method for text-image classification using a frozen CLIP ViT-L/14 backbone. Temporal prompts are inserted on selected intermediate layers of the vision and text branches and are exchanged through a lightweight 'Interaction Hub' that computes intra- and inter-modality similarity gates. The method reports 93.95% accuracy on UPMC-Food101, 73.45% on SNLI-VE, and 65.9/56.3 F1-micro/macro on MM-IMDB with 2.0M trainable parameters. The paper claims state-of-the-art results among prompt-based methods with the same foundation model at minimal GPU memory usage and trainable parameters, and includes an ablation study of the components and a sensitivity analysis of interaction layer choices and similarity functions.
Significance. The interaction mechanism is conceptually simple, and the ablation study in Table II shows monotonic gains from adding temporal prompts, prompt interaction, and the similarity-based gating, which gives internal support to the design. The reported accuracy wins over re-implemented prompt-based baselines on two of three datasets, and the parameter count of EPIC-base (2.0M) is far below full fine-tuning. However, the paper's central efficiency claim is contradicted by its own Table I, and the headline UPMC-Food101 accuracy is obtained after selecting interaction layers and similarity type on the same test set that is used for the main result. If these protocol issues are corrected, the method could be a useful parameter-efficient multimodal fusion baseline; as written, the strength of the claimed contribution is not fully established.
major comments (4)
- [Table I, Abstract, Conclusions] The central efficiency claim is not supported by the paper's own measurements. Table I reports EPIC-base with 17.63 GB training memory, which is higher than PMF's 15.29 GB, and 2.0M trainable parameters, which is higher than MaPLe* (0.8M), P-CLIP, PromptFuse, and BlindPrompt (all 0.1M). The abstract and conclusion state that EPIC achieves 'minimal GPU memory usage and trainable parameters', and Figure 3(c) repeats this. Since no measurement protocol (batch size, sequence length, hardware, gradient checkpointing) is provided, the claim as stated is directly contradicted by the reported numbers. The authors should either provide controlled comparisons and correct the claim to 'competitive', or supply additional evidence for a minimal-resource interpretation.
- [Section IV-B and Table I] The interaction layers (4,5,6) and cosine similarity are selected by inspecting results on UPMC-Food101, as shown in Figure 4, and the same dataset is then used as the headline benchmark in Table I. No held-out validation split is used. This means the reported 93.95% accuracy is the maximum over configurations evaluated on the test set, so the improvement over PMF* (93.95 vs 92.54) may be inflated by selection. The authors should either use a separate validation split for model selection or report results under a protocol that avoids test-set-based selection, for example by pre-registering the configuration or by reporting post-selection corrected estimates.
- [Section II-E, Eq (8)] The training loss for uni-label classification is written as binary cross-entropy with y_i in {0,1}, but the prediction is a softmax over K classes in Eq (7). For a K-class task such as UPMC-Food101, the correct loss is the negative log-likelihood of the true class, not the binary form in Eq (8). If a sigmoid activation and binary labels are used instead, Eq (7) should be changed accordingly. This inconsistency must be resolved for the training procedure to be reproducible.
- [Table I and Section III-B] All results are reported as single numbers with no error bars, confidence intervals, or significance tests. The margins over PMF* are 1.01-1.41 accuracy points on two datasets; without repeated runs (e.g., 3-5 seeds) and variance reporting, it is not possible to assess whether these differences are reliable. The authors should report mean and standard deviation over multiple runs.
minor comments (5)
- [Section II-D, Eqs (4)-(6)] The dimensions of the similarity matrices and the axis over which SoftMax is applied are not specified. Please clarify whether softmax is over prompt tokens, feature units, or both.
- [Table I] The row for EPIC-base has values run together without visible separators, making the table difficult to read. Please reformat the table so each metric is clearly aligned.
- [Section I and Section II-D] The introduction says the interaction happens 'without participation of modality features', but Eq (2) concatenates prompts with modality features before the encoder layer. Please rephrase to avoid this apparent contradiction.
- [Section II-C] The first temporal prompt for text is randomly initialized with a Gaussian and then used to produce the vision prompt via a linear layer. Please clarify whether the text prompt itself is also trained and how the initialization relates to the shared trainable parameters across layers.
- [Abstract] The abstract says 'fewer trainable parameters (about 1% of the foundation model)' but gives no calculation; please specify the foundation model parameter count used for the percentage.
Circularity Check
EPIC's headline UPMC-Food101 accuracy is partly selected by test-set hyperparameter tuning; the method itself has no definitional circularity.
-
fitted input called prediction
[Section IV-B / Figure 4 and Section III-A]
"The experimental results are shown in Figure 4. The results indicate that the model reaches its highest performance at interaction layers (4,5,6) with cosine similarity. ... The results are on UPMC-Food101. ... We leverage 3 interaction layers empirically for the base EPIC."
The configuration yielding the reported 93.95% UPMC-Food101 accuracy (interaction layers 4-6, cosine similarity) is chosen by inspecting accuracy on the UPMC-Food101 test set in Figure 4, and the same test set is then used to report the headline result in Table I. No held-out validation split is described. The headline accuracy is therefore the maximum of a configuration sweep over the evaluation set rather than a prediction from a fixed method, so the comparison against PMF (93.95 vs 92.54) is not a clean out-of-sample measurement. This is a selected input being presented as the method's reported performance.
full rationale
The method's equations are self-contained: Eqs. (1)-(6) define prompt updates solely from input features and trainable parameters, with no target labels or reported accuracies embedded, so there is no self-definitional circularity. Baselines are external (MaPLe*, PMF*, etc.) and the paper re-implements them; no load-bearing self-citation or imported uniqueness theorem is used. The only circular element is the evaluation protocol for UPMC-Food101: the interaction-layer and similarity-type choices are selected by inspecting results on that dataset's test set, and then the same test set is used for the headline accuracy. This partially inflates the central performance claim but does not affect the architectural derivation. The 'minimal GPU memory' claim is contradicted by Table I (EPIC-base 17.63 GB vs PMF 15.29 GB), but that is an unsupported efficiency claim, not a circular derivation. Overall score 4: one moderate evaluation-circularity step in an otherwise independent empirical method.
Assumptions & free parameters
free parameters (6)
- Prompt length =
3
- Interaction layers =
layers 4, 5, 6 for EPIC-base; layer 4 for EPIC-small
- Similarity type =
cosine similarity
- Number of interaction layers =
3 for base, 1 for small
- Text prompt Gaussian std =
0.02
- Temperature tau in Eq (7) =
not reported
assumptions (4)
- domain assumption Frozen CLIP ViT-L/14 features are sufficient backbones for classification after prompt tuning.
- domain assumption Temporal prompts can store contextual information and transfer it across layers and modalities.
- ad hoc to paper Similarity-based gating (Eqs 4-6) is a valid and sufficient way to fuse prompt information.
- standard math Standard softmax, cosine similarity, and MLP projection behave as expected in high-dimensional CLIP embedding space.
Cite this review
Pith. "Pith review of EPIC: Efficient Prompt Interaction for Text-Image Classification." pith.science (2026). https://pith.science/paper/UEQWJQPJ
@misc{pith2026250707415,
author = {Pith},
title = {Pith review of: EPIC: Efficient Prompt Interaction for Text-Image Classification},
year = {2026},
howpublished = {\url{https://pith.science/paper/UEQWJQPJ}},
note = {Machine review of arXiv:2507.07415}
}
read the original abstract
In recent years, large-scale pre-trained multimodal models (LMMs) generally emerge to integrate the vision and language modalities, achieving considerable success in multimodal tasks, such as text-image classification. The growing size of LMMs, however, results in a significant computational cost for fine-tuning these models for downstream tasks. Hence, prompt-based interaction strategy is studied to align modalities more efficiently. In this context, we propose a novel efficient prompt-based multimodal interaction strategy, namely Efficient Prompt Interaction for text-image Classification (EPIC). Specifically, we utilize temporal prompts on intermediate layers, and integrate different modalities with similarity-based prompt interaction, to leverage sufficient information exchange between modalities. Utilizing this approach, our method achieves reduced computational resource consumption and fewer trainable parameters (about 1\% of the foundation model) compared to other fine-tuning strategies. Furthermore, it demonstrates superior performance on the UPMC-Food101 and SNLI-VE datasets, while achieving comparable performance on the MM-IMDB dataset.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
Cma-clip: Cross- modality attention clip for text-image classification,
Jinmiao Fu, Shaoyuan Xu, Huidong Liu, Yang Liu, Ning Xie, Chien- Chih Wang, Jia Liu, Yi Sun, and Bryan Wang, “Cma-clip: Cross- modality attention clip for text-image classification,” in2022 IEEE International Conference on Image Processing (ICIP). IEEE, 2022, pp. 2846–2850. 1, 4
work page 2022
-
[2]
Future-aware diverse trends framework for recommendation,
Yujie Lu, Shengyu Zhang, Yingxuan Huang, Luyao Wang, Xinyao Yu, Zhou Zhao, and Fei Wu, “Future-aware diverse trends framework for recommendation,” inProceedings of the Web Conference 2021, 2021, pp. 2992–3001. 1
work page 2021
-
[3]
Dense fusion network with multimodal residual for sentiment classification,
Huan Deng, Peipei Kang, Zhenguo Yang, Tianyong Hao, Qing Li, and Wenyin Liu, “Dense fusion network with multimodal residual for sentiment classification,” in2021 IEEE International Conference on Multimedia and Expo (ICME). IEEE, 2021, pp. 1–6. 1
work page 2021
-
[4]
Tensor fusion network for multimodal sentiment analysis,
Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis- Philippe Morency, “Tensor fusion network for multimodal sentiment analysis,”arXiv preprint arXiv:1707.07250, 2017. 1
arXiv 2017
-
[5]
Memory fusion network for multi-view sequential learning,
Amir Zadeh, Paul Pu Liang, Navonil Mazumder, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency, “Memory fusion network for multi-view sequential learning,” inProceedings of the AAAI conference on artificial intelligence, 2018, vol. 32. 1
work page 2018
-
[6]
Misa: Modality-invariant and-specific representations for multimodal sentiment analysis,
Devamanyu Hazarika, Roger Zimmermann, and Soujanya Poria, “Misa: Modality-invariant and-specific representations for multimodal sentiment analysis,” inProceedings of the 28th ACM international conference on multimedia, 2020, pp. 1122–1131. 1
work page 2020
-
[7]
Centralnet: a multilayer approach for multimodal fusion,
Valentin Vielzeuf, Alexis Lechervy, St ´ephane Pateux, and Fr ´ed´eric Jurie, “Centralnet: a multilayer approach for multimodal fusion,” in Proceedings of the European Conference on Computer Vision (ECCV) Workshops, 2018, pp. 0–0. 1
work page 2018
-
[8]
Modular and Parameter-Efficient Multimodal Fusion with Prompting
Sheng Liang, Mengjie Zhao, and Hinrich Sch ¨utze, “Modular and parameter-efficient multimodal fusion with prompting,”arXiv preprint arXiv:2203.08055, 2022. 1, 2, 4
work page Pith review arXiv 2022
Show all 24 references
-
[9]
Efficient multimodal fusion via interactive prompting,
Yaowei Li, Ruijie Quan, Linchao Zhu, and Yi Yang, “Efficient multimodal fusion via interactive prompting,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 2604–2613. 1, 2, 4
2023
-
[10]
Supervised multimodal bitransformers for classifying images and text,
Douwe Kiela, Suvrat Bhooshan, Hamed Firooz, Ethan Perez, and Davide Testuggine, “Supervised multimodal bitransformers for classifying images and text,”arXiv preprint arXiv:1909.02950, 2019. 1, 4
1909 arXiv
-
[11]
Visual prompt tuning,
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim, “Visual prompt tuning,” inEuropean Conference on Computer Vision. Springer, 2022, pp. 709–
2022
-
[12]
Learning to prompt for vision-language models,
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu, “Learning to prompt for vision-language models,”International Journal of Computer Vision, vol. 130, no. 9, pp. 2337–2348, 2022. 1
2022
-
[13]
Con- ditional prompt learning for vision-language models,
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu, “Con- ditional prompt learning for vision-language models,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 16816–16825. 1
2022
-
[14]
Maple: Multi-modal prompt learning,
Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan, “Maple: Multi-modal prompt learning,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 19113–19122. 1, 2, 4
2023
-
[15]
Huse: Hierarchical universal semantic embeddings,
Pradyumna Narayana, Aniket Pednekar, Abishek Krishnamoorthy, Ka- zoo Sone, and Sugato Basu, “Huse: Hierarchical universal semantic embeddings,”arXiv preprint arXiv:1911.05978, 2019. 4
1911 arXiv
-
[16]
Multibench: Multiscale benchmarks for multimodal representation learning,
Paul Pu Liang, Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslie Chen, Peter Wu, Michelle A Lee, Yuke Zhu, et al., “Multibench: Multiscale benchmarks for multimodal representation learning,”arXiv preprint arXiv:2107.07502, 2021. 4
2021 arXiv
-
[17]
Dynamic multimodal fusion,
Zihui Xue and Radu Marculescu, “Dynamic multimodal fusion,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 2574–2583. 4
2023
-
[18]
Unit: Multimodal multitask learning with a unified transformer,
Ronghang Hu and Amanpreet Singh, “Unit: Multimodal multitask learning with a unified transformer,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 1439–1449. 4
2021
-
[19]
Vilt: Vision-and-language transformer without convolution or region supervision,
Wonjae Kim, Bokyung Son, and Ildoo Kim, “Vilt: Vision-and-language transformer without convolution or region supervision,” inProceedings of the 38th International Conference on Machine Learning, Marina Meila and Tong Zhang, Eds. 18–24 Jul 2021, vol. 139 ofProceedings of Machin...
2021
-
[20]
Recipe recognition with large multimodal food dataset,
Xin Wang, Devinder Kumar, Nicolas Thome, Matthieu Cord, and Frederic Precioso, “Recipe recognition with large multimodal food dataset,” in2015 IEEE International Conference on Multimedia & Expo Workshops (ICMEW). IEEE, 2015, pp. 1–6. 4
2015
-
[21]
Gated multimodal units for information fusion,
John Arevalo, Thamar Solorio, Manuel Montes-y G ´omez, and Fabio A Gonz´alez, “Gated multimodal units for information fusion,”arXiv preprint arXiv:1702.01992, 2017. 4
2017 arXiv
-
[22]
Visual en- tailment task for visually-grounded language learning,
Ning Xie, Farley Lai, Derek Doran, and Asim Kadav, “Visual en- tailment task for visually-grounded language learning,”arXiv preprint arXiv:1811.10582, 2018. 4
2018 arXiv
-
[23]
Learning transferable visual models from natural language supervision,
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al., “Learning transferable visual models from natural language supervision,” inInternational conference on machine learning. ...
2021
-
[24]
Visualbert: A simple and performant baseline for vision and language. arxiv 2019,
Liunian Harold Li, Mark Yatskar, D Yin, CJ Hsieh, and KW Chang, “Visualbert: A simple and performant baseline for vision and language. arxiv 2019,”arXiv preprint arXiv:1908.03557. 4
2019 arXiv
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.