Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Expertized Caption Auto-Enhancement for Video-Text Retrieval

T0 review · 4 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Generating diverse captions on the video side and learning to select the best per query narrows the modality gap, letting ExCae achieve state-of-the-art retrieval accuracy without extra data.

desk verdict A genuinely new video-side caption augmentation pipeline for video-text retrieval, with a load-bearing but unvalidated caption-scoring proxy and single-run numbers that need scrutiny before the SOTA claims are trusted. read the letter →

arxiv 2502.02885 v3 pith:5ACV665A submitted 2025-02-05 cs.CV cs.AIcs.LG

Baoyao Yang , Junxiang Chen , Wanyun Li , Wenbin Yao , Yang Zhou This is my paper · ORCID
classification cs.CVcs.AIcs.LG
keywords video-textretrievalcaptionaugmentationpromptoptimizationmultimodallargelanguagemodelmixtureofexpertsmodalitygapself-improvement
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Video-text retrieval usually fails because the text describing a video is short and one-sided, while the video itself is rich, so the two modalities sit far apart. This paper argues that the better fix is not to rewrite the text, but to add automatically generated, multi-perspective captions on the video side, bringing video representations closer to text. To do that it introduces ExCae, a pipeline with two parts: Caption Self-Improvement, which iteratively refines the prompt of a multimodal language model to make captions both faithful to the original text and diverse, and Expertized Caption Selection, a mixture-of-experts router that chooses which captions best match each query. The authors report state-of-the-art Top-1 recall (68.5% on MSR-VTT, 68.1% on MSVD, 62.0% on DiDeMo) using no extra training data, and show the same modules can be plugged into existing retrieval models to raise their accuracy.

What carries the argument

The two load-bearing pieces are the Caption Self-Improvement module (CSI) and the Expertized Caption Selection module (ECS). CSI runs a loop: a multimodal LLM (the Captioner) generates $K$ captions per video under a query prompt; a separate Prompt Engineer rewrites the prompt; and the rewrite is kept only if the caption score rises. The score is $Score_t = \mathbb{E}_{i,k}[\mathrm{sim}(\psi(c^t_{i,k}), \psi(t_i))] + \mathrm{div}(\psi(c^t_i))$, where $\psi$ is a semantic embedding extractor, the first term rewards semantic consistency with the original text, and the second term (a mean pairwise dissimilarity over the $K$ captions) rewards diversity. ECS takes the $N$ sampled frames plus the $K$ generated captions as $N{+}K$ video-side expressions and routes them through $M$ learnable experts with top-$R$ gating: $\tilde{e}_v = \frac{1}{M}\sum_m f_m(r_m(e_v)\otimes e_v) + e_v$. The router weights are learned under the same contrastive loss as the encoders, so the model personalizes which caption viewpoints are used for each video-text pair.

What would settle it

Take a set of candidate prompts, run the CSI scoring function on each, then fine-tune the retrieval model with each prompt's captions and measure R@1 on a held-out validation split; if the ranking of prompts by Score_t does not match the ranking by R@1, the central mechanism fails.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that directional augmentation — moving from video to text rather than text to video — is what actually expands the representation space enough to shrink the modality gap. The authors show that generated captions alone help little, and original videos alone help little; the large gain appears only when the two are combined and filtered through learnable experts. The prompt optimizer is not chasing a hand-written style: it maximizes a score that rewards semantic agreement with the original caption and diversity among the ten generated captions, and the learned prompt transfers across datasets. With a ViT-G/14 backbone the method reaches R@1 of 67.7% text-to-video and 69.3% video-to-text on MSR-VTT, and similar records on MSVD and DiDeMo, surpassing methods that use extra pretraining data. The paper also reports that the same caption-auto-enhancement and expert-selection modules, attached to existing retrieval encoders, improve their Top-1 recall by roughly five to nine points.

Load-bearing premise

The caption-scoring function in Equation (1) is assumed to be a valid proxy for retrieval accuracy, but the paper does not demonstrate that a higher score actually predicts higher recall; if the proxy is miscalibrated, the self-improvement loop could be optimizing the wrong target.

Editorial extensions

If this is right

  • Retrieval accuracy no longer depends on expensive extra corpora: ExCae on a ViT-B/16 backbone already beats ViT-based methods that use additional data, and the ViT-G/14 variant surpasses pretrained foundation models on MSR-VTT.
  • Existing video-text retrieval models can be upgraded without redesign: the plug-in experiments raise Top-1 recall by 4.8, 9.4, and 7.7 points for three base methods, indicating a general add-on.
  • The learned caption prompt is reusable: because prompt optimization is a one-off process and the resulting prompt transfers across datasets, the cost is amortized over many retrieval tasks.
  • The method keeps improving with model scale: R@1 grows roughly seven points from the ViT-L/14 to ViT-G/14 backbone, implying the best results are not yet saturated.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same CSI loop should transfer to image-text retrieval, where captions are equally short; the scoring function already uses a text embedding extractor, so only the Captioner would need to accept still images.
  • Editorial inference: the ECS mixture-of-experts router could be applied as a general cross-modal fusion layer — for example, to choose among multiple human-written captions or multiple text rewrites — not just among video-derived captions.
  • Editorial inference: the paper's convergence claim (CSI stabilizes in about 400 iterations) suggests the learned prompt has converged to a style rather than to dataset-specific details, which is consistent with the reported cross-dataset transfer; a direct test would be to initialize a different dataset with the prompt learned on MSR-VTT and compare against starting from scratch.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper proposes ExCae, a caption auto-enhancement method for video-text retrieval. The Caption Self-Improvement (CSI) module iteratively refines a prompt for an mLLM Captioner using a Prompt Engineer, guided by a caption score that combines M3-embedding-based semantic consistency and diversity (Eq. (1)). The Expertized Caption Selection (ECS) module then fuses sampled video frames with K generated captions through a mixture-of-experts layer before the standard contrastive alignment. Experiments on MSR-VTT, MSVD, and DiDeMo report strong R@1 gains over ViT-based baselines, ablations of the two modules, and plug-in improvements for DRL, TS2-Net, and Clip4Clip. The code is publicly available.

Significance. If the reported results are reproducible, the paper makes a useful practical contribution: it improves video-text retrieval without collecting extra training data, offers a plug-in formulation, and releases code. The core idea of generating captions from the video side and selecting them per sample is clearly motivated and reasonably presented. I agree with the reader's assessment that the method is not circular: prompt selection uses training data and a heuristic score, not test labels. The main weaknesses are the unvalidated caption-scoring proxy that drives CSI, the absence of variance estimates, and overstatements in the benchmark comparisons; these are fixable with additional experiments and tighter claims.

major comments (4)
  1. [Section III-A, Eq. (1)] The caption scoring function Score_t is the only feedback signal for the CSI prompt search, but the paper never shows that a higher Score_t corresponds to higher retrieval R@1. The text itself says "we hope" that the adjustment is positively correlated with VTR results. Since Score_t is computed in M3-embedding space while retrieval is performed in CLIP space, a correlation analysis is needed, for example by reporting Score_t and R@1 across CSI iterations or across a set of candidate prompts. Table IV only shows that the final learned prompt beats the initial prompt; it does not establish that hill-climbing on Score_t is optimizing the retrieval objective. Also, M3-embedding is a text-only model, so the score cannot detect whether a caption is faithful to the video content, only whether it resembles the original text.
  2. [Tables I and II] All reported numbers are single-run point estimates with no error bars, standard deviations, or significance tests. This is especially problematic because several head-to-head comparisons are close: on MSVD text-to-video with ViT-B/16, ExCae (51.3) is below Cap4Video++ (51.8), and on DiDeMo several ExCae entries are within a few points of existing methods. The claim of state-of-the-art performance cannot be evaluated from unreplicated point estimates; the authors should provide variance over multiple seeds or at least state that the reported runs are representative.
  3. [Table II (DiDeMo)] The abstract and Section IV-C claim state-of-the-art results on all three benchmarks, but Table II shows that on DiDeMo the best ExCae text-to-video R@1 (62.0 with ViT-G/14) is substantially below the pre-trained foundation models UMT-L (70.4) and VAST (72.0). The paper's claim is valid only within the "ViT-based" category, so the benchmark claim in the abstract should be revised or explicitly scoped to that category.
  4. [Algorithm 1 and Section IV-B] The CSI prompt-selection protocol is underspecified. The Prompt Engineer is given "the corresponding video-text pairs (x,t) involved in this iteration," but the paper does not state how many pairs are used, how they are sampled, or whether they come strictly from the training split. Because the selected prompt affects all downstream retrieval scores, the authors must document the data split and the stopping or iteration budget (e.g., the claim of convergence in 400 iterations) so that test-set leakage can be ruled out.
minor comments (6)
  1. [Abstract vs. Tables I and II] The abstract reports 68.5% on MSR-VTT and 68.1% on MSVD, which appear to be the means of the text-to-video and video-to-text R@1 entries in the tables rather than a single reported direction; the abstract should state exactly which metric is being reported.
  2. [Section IV-C] The statement that existing methods "could hardly transcend 50%" without extra data is contradicted by Cap4Video++, which reports 52.3 R@1 on MSR-VTT with no extra data in Table I.
  3. [Section IV-C] The claimed improvements of "7% on MSVD" and "13% on DiDeMo" are actually percentage-point differences and are not computed correctly: against the best ViT-based method in Table II, the differences are 7.4 points and 9.5 points, respectively.
  4. [Table IV] The row labels and checkmark columns of Table IV are ambiguous: several rows have no checkmark yet still report numbers, and it is not clear which checkmark indicates the CSI module versus the ECS module for each row; the table should be reformatted.
  5. [Section IV-B] The implementation details should state the exact GPT-4o version, sampling temperature, and number of stochastic runs, because caption generation from a closed LLM is non-deterministic and affects the reproducibility of all reported numbers.
  6. [Section IV-D] The method name "TSR-Net" in the plug-in experiment appears to be a typo for "TS2-Net"; please correct it.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the caption scoring proxy is a training-time supervision signal, not a renamed retrieval metric, and test labels never enter prompt selection.

full rationale

ExCae's two modules are trained on training video-text pairs and evaluated on held-out test splits. The CSI prompt optimizer maximizes Score_t (Eq. 1), a combination of M3-embedding similarity between generated captions and the original training text plus caption diversity; this is an auxiliary training objective, not a fitted value of the reported R@1 results. The paper explicitly frames it as a proxy ('We hope that the adjustment of the query prompt will be positively correlated with the increase in VTR results'), and even if the proxy is miscalibrated, that is a correctness or validity concern, not circularity. The ECS experts are learned under standard contrastive supervision from the same training texts, and no test labels are used to select prompts or experts. The reported R@1 numbers are measured on held-out MSR-VTT, MSVD, and DiDeMo test sets. No equation reduces a predicted quantity to a fitted parameter, no load-bearing argument rests on a self-citation, and no uniqueness theorem or ansatz is smuggled in via citation. The 'fully data-driven' claim is somewhat weakened by the hand-chosen initial prompt and the reliance on GPT-4o and M3-embedding, but that is a novelty or robustness limitation, not circular reasoning.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The core method rests on several hand-chosen hyperparameters (K, M, R, p0, T) and on unstated assumptions about the caption scoring proxy, GPT-4o familiarity with 8-frame video input, and the stability of the MoE module. These are the price of the claimed automaticity and are the main burden the reader must accept.

free parameters (5)
  • K: number of generated captions per video = 10
    Sensitivity experiment in Fig. 6 shows performance improves with caption count and stabilizes above 7; K=10 is used in all main experiments.
  • M: total number of ECS experts = 16
    Sixteen experts are used throughout, but no sensitivity analysis for M is reported; it is a hand-chosen hyperparameter.
  • R: number of activated experts per sample = 2
    Fig. 7 shows R=2 is the best activated-count on MSR-VTT, and this value is then applied to all datasets.
  • p0: initial prompt for the Captioner = Generate 10 captions from different perspectives about this video.
    The hand-written initial prompt anchors the entire prompt search; sensitivity to p0 is not studied.
  • T: maximum CSI iterations = 400
    The paper states the CSI module tends to converge in 400 iterations based on experimental expertise; no formal stopping criterion is given.
assumptions (4)
  • domain assumption CLIP pretrained encoders provide a suitable shared embedding space for video frames and text, and the standard contrastive loss aligns them.
    The method starts from CLIP weights (Section IV-B) and relies on this alignment for video-text matching.
  • ad hoc to paper M3-embedding (psi) measures semantic consistency and diversity in a way that is a valid proxy for retrieval quality.
    Equation (1) uses psi to score captions for prompt selection, but the paper does not demonstrate a correlation between this score and R@1.
  • domain assumption GPT-4o can generate complete and reasonable video captions from 8 sampled frames plus a textual prompt.
    The Captioner and Prompt Engineer are implemented by GPT-4o (Section IV-B); if GPT-4o cannot faithfully describe videos from 8 frames, the whole video-side augmentation collapses.
  • domain assumption The ECS MoE with 16 experts and top-2 routing can be trained stably with the standard contrastive loss without expert collapse.
    No analysis of router collapse or expert specialization is provided in Section III-B or the experiments.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Expertized Caption Auto-Enhancement for Video-Text Retrieval." pith.science (2026). https://pith.science/paper/5ACV665A

@misc{pith2026250202885,
  author       = {Pith},
  title        = {Pith review of: Expertized Caption Auto-Enhancement for Video-Text Retrieval},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5ACV665A}},
  note         = {Machine review of arXiv:2502.02885}
}
read the original abstract

Video-text retrieval has been stuck in the information mismatch caused by personalized and inadequate textual descriptions of videos. The substantial information gap between the two modalities hinders an effective cross-modal representation alignment, resulting in ambiguous retrieval results. Although text rewriting methods have been proposed to broaden text expressions, the modality gap remains significant, as the text representation space is hardly expanded with insufficient semantic enrichment.Instead, this paper turns to enhancing visual presentation, bridging video expression closer to textual representation via caption generation and thereby facilitating video-text matching.While multimodal large language models (mLLM) have shown a powerful capability to convert video content into text, carefully crafted prompts are essential to ensure the reasonableness and completeness of the generated captions. Therefore, this paper proposes an automatic caption enhancement method that improves expression quality and mitigates empiricism in augmented captions through self-learning.Additionally, an expertized caption selection mechanism is designed and introduced to customize augmented captions for each video, further exploring the utilization potential of caption augmentation.Our method is entirely data-driven, which not only dispenses with heavy data collection and computation workload but also improves self-adaptability by circumventing lexicon dependence and introducing personalized matching. The superiority of our method is validated by state-of-the-art results on various benchmarks, specifically achieving Top-1 recall accuracy of 68.5% on MSR-VTT, 68.1% on MSVD, and 62.0% on DiDeMo. Our code is publicly available at https://github.com/CaryXiang/ECA4VTR.

Figures

Figures reproduced from arXiv: 2502.02885 by the authors.

Figure 1
Figure 1. Comprehension gap between videos and texts [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Comparison of our method to the traditional [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Illustration of the CSI module remain deficient in the rewritten texts through these text￾to-text literal restatements. Generating captions from videos as auxiliaries is a brighter approach for text enhancement. These methods form new pairs using videos and derived captions to promote cross-modal representation learning [15]. However, carefully designed prompts are necessary, and later works manually craft various q… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Examples of text-to-video retrieval results on MSR-VTT dataset. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Examples of video-to-text retrieval results on MSR-VTT dataset. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Sensitivity to the number of video-derived captions [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: We also show the results obtained without the ECS [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Modality gap and Convergence on MSR-VTT B. Convergence Since the generated captions are changeless once the opti￾mal prompt is determined, the cross-modal post-training of our method owns the same convergence as existing ViT￾based methods. Regarding learning the CSI mo…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MemVerse: Multimodal Memory for Lifelong Learning Agents

    cs.AI 2025-12 reject novelty 4.0 of 10

    MemVerse reports large gains on multimodal benchmarks by adding a hierarchical knowledge-graph memory plus fine-tuned parametric recall, but its strongest video-retrieval result uses ground-truth caption-video pairs i...

Reference graph

Works this paper leans on

39 extracted references · 31 canonical work pages · cited by 1 Pith paper

  1. [1]

    Un- masked teacher: Towards training-efficient video foundation models,

    K. Li, Y . Wang, Y . Li, Y . Wang, Y . He, L. Wang, and Y . Qiao, “Un- masked teacher: Towards training-efficient video foundation models,” in IEEE/CVF International Conference on Computer Vision , 2023, pp. 19 948–19 960

    work page 2023

  2. [2]

    Vast: A vision-audio-subtitle-text omni-modality foundation model and dataset,

    S. Chen, H. Li, Q. Wang, Z. Zhao, M. Sun, X. Zhu, and J. Liu, “Vast: A vision-audio-subtitle-text omni-modality foundation model and dataset,” Advances in Neural Information Processing Systems , vol. 36, 2024

    work page 2024

  3. [3]

    Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning,

    H. Luo, L. Ji, M. Zhong, Y . Chen et al. , “Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning,” Neurocomputing, vol. 508, pp. 293–304, 2022

    work page 2022

  4. [4]

    Clip-vip: Adapting pre-trained image-text model to video-language representation alignment,

    H. Xue, Y . Sun, B. Liu, J. Fu, R. Song, H. Li, and J. Luo, “Clip-vip: Adapting pre-trained image-text model to video-language representation alignment,” in International Conference on Learning Representation , 2023. 9

    work page 2023

  5. [5]

    Long-form video- language pre-training with multimodal temporal contrastive learning,

    Y . Sun, H. Xue, R. Song, B. Liu, H. Yang, and J. Fu, “Long-form video- language pre-training with multimodal temporal contrastive learning,” in Advances in Neural Information Processing Systems , 2022

    work page 2022

  6. [6]

    Litevl: Efficient video-language learning with enhanced spatial-temporal mod- eling,

    D. Chen, C. Tao, L. Hou, L. Shang, X. Jiang, and Q. Liu, “Litevl: Efficient video-language learning with enhanced spatial-temporal mod- eling,” in Conference on Empirical Methods in Natural Language Processing, 2022

    work page 2022

  7. [7]

    Clip2tv: An empirical study on transformer-based methods for video-text retrieval,

    Z. Gao, J. Liu, S. Chen, D. Chang et al., “Clip2tv: An empirical study on transformer-based methods for video-text retrieval,” arXiv preprint arXiv:2111.05610, vol. 1, no. 2, p. 6, 2021

    arXiv 2021

  8. [8]

    Less is more: Clipbert for video-and-language learning via sparse sampling,

    J. Lei, L. Li, L. Zhou, Z. Gan, T. L. Berg, M. Bansal, and J. Liu, “Less is more: Clipbert for video-and-language learning via sparse sampling,” in IEEE/CVF conference on computer vision and pattern recognition , 2021, pp. 7331–7341

    work page 2021

Show all 39 references
  1. [9]

    X-clip: End-to-end multi-grained con- trastive learning for video-text retrieval,

    Y . Ma, G. Xu, X. Sun et al. , “X-clip: End-to-end multi-grained con- trastive learning for video-text retrieval,” in ACM International Confer- ence on Multimedia , 2022, pp. 638–647

    2022

  2. [10]

    Advancing high-resolution video-language representation with large-scale video transcriptions,

    H. Xue, T. Hang, Y . Zeng, Y . Sun, B. Liu, H. Yang, J. Fu, and B. Guo, “Advancing high-resolution video-language representation with large-scale video transcriptions,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 5036–5045

    2022

  3. [11]

    Bidirectional cross-modal knowledge exploration for video recognition with pre-trained vision-language models,

    W. Wu, X. Wang, H. Luo, J. Wang, Y . Yang, and W. Ouyang, “Bidirectional cross-modal knowledge exploration for video recognition with pre-trained vision-language models,” in IEEE/CVF conference on computer vision and pattern recognition , 2023, pp. 6620–6630

    2023

  4. [12]

    She-net: Syntax-hierarchy-enhanced text-video retrieval,

    X. Yu, C. Jiang, X. Dong, T. Gan, M. Yang, and Q. Guo, “She-net: Syntax-hierarchy-enhanced text-video retrieval,” IEEE Transactions on Circuits and Systems for Video Technology , 2025

    2025

  5. [13]

    Teachtext: Crossmodal generalized distillation for text-video retrieval,

    I. Croitoru, S.-V . Bogolin, M. Leordeanu, H. Jin, A. Zisserman, S. Al- banie, and Y . Liu, “Teachtext: Crossmodal generalized distillation for text-video retrieval,” in Proceedings of the IEEE/CVF international conference on computer vision , 2021, pp. 11 583–11 593

    2021

  6. [14]

    Text is mass: Modeling as stochastic embedding for text-video retrieval,

    J. Wang, G. Sun, P. Wang, D. Liu, S. Dianat, M. Rabbani, R. Rao, and Z. Tao, “Text is mass: Modeling as stochastic embedding for text-video retrieval,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2024, pp. 16 551–16 560

    2024

  7. [15]

    Cap4video++: Enhancing video understanding with auxiliary captions,

    W. Wu, X. Wang, H. Luo, J. Wang, Y . Yang, and W. Ouyang, “Cap4video++: Enhancing video understanding with auxiliary captions,” TPAMI, pp. 1–15, 2024

    2024

  8. [16]

    The neglected tails in vision-language models,

    S. Parashar, Z. Lin, T. Liu et al., “The neglected tails in vision-language models,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 12 988–12 997

    2024

  9. [17]

    Verbs in action: Improving verb understanding in video-language models,

    L. Momeni, M. Caron, A. Nagrani, A. Zisserman, and C. Schmid, “Verbs in action: Improving verb understanding in video-language models,” in IEEE/CVF International Conference on Computer Vision , 2023, pp. 15 579–15 591

    2023

  10. [18]

    Videocon: Robust video-language alignment via contrast captions,

    H. Bansal, Y . Bitton, I. Szpektor, K.-W. Chang, and A. Grover, “Videocon: Robust video-language alignment via contrast captions,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 13 927–13 937

    2024

  11. [19]

    Havtr: Improving video-text retrieval through augmentation using large foundation models,

    Y . Wang, S. Yuan, X. Jian et al., “Havtr: Improving video-text retrieval through augmentation using large foundation models,” arXiv preprint arXiv:2404.05083, 2024

    2024 arXiv

  12. [20]

    Automatic prompt optimization with

    R. Pryzant, D. Iter, J. Li, Y . T. Lee, C. Zhu, and M. Zeng, “Automatic prompt optimization with” gradient descent” and beam search,” Confer- ence on Empirical Methods in Natural Language Processing , 2023

    2023

  13. [21]

    Dynamic prompt learning: Addressing cross-attention leakage for text-based image edit- ing,

    F. Yang, S. Yang, M. A. Butt, J. van de Weijer et al., “Dynamic prompt learning: Addressing cross-attention leakage for text-based image edit- ing,” Advances in Neural Information Processing Systems , vol. 36, pp. 26 291–26 303, 2023

    2023

  14. [22]

    Internvideo: General video foundation models via generative and discriminative learning,

    Y . Wang, K. Li, Y . Li et al. , “Internvideo: General video foundation models via generative and discriminative learning,” arXiv preprint arXiv:2212.03191, 2022

    2022 arXiv

  15. [23]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal et al. , “Learning transferable visual models from natural language supervision,” in International conference on machine learning , 2021, pp. 8748–8763

    2021

  16. [24]

    Ts2-net: Token shift and selection trans- former for text-video retrieval,

    Y . Liu, P. Xiong, L. Xu et al., “Ts2-net: Token shift and selection trans- former for text-video retrieval,” in European Conference on Computer Vision, 2022, pp. 319–335

    2022

  17. [25]

    Mv-adapter: Multimodal video transfer learning for video text retrieval,

    X. Jin, B. Zhang, W. Gong, K. Xu, X. Deng, P. Wang, Z. Zhang, X. Shen, and J. Feng, “Mv-adapter: Multimodal video transfer learning for video text retrieval,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 27 144–27 153

    2024

  18. [26]

    Holistic features are almost sufficient for text-to-video retrieval,

    K. Tian, R. Zhao, Z. Xin, B. Lan, and X. Li, “Holistic features are almost sufficient for text-to-video retrieval,” in CVPR, 2024, pp. 17 138–17 147

    2024

  19. [27]

    Disentan- gled representation learning for text-video retrieval,

    Q. Wang, Y . Zhang, Y . Zheng, P. Pan, and X.-S. Hua, “Disentan- gled representation learning for text-video retrieval,” arXiv preprint arXiv:2203.07111, 2022

    2022 arXiv

  20. [28]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar et al. , “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  21. [29]

    Reversed in time: A novel temporal- emphasized benchmark for cross-modal video-text retrieval,

    Y . Du, Y . Liu, and Q. Jin, “Reversed in time: A novel temporal- emphasized benchmark for cross-modal video-text retrieval,” in Pro- ceedings of the 32nd ACM International Conference on Multimedia , 2024, pp. 5260–5269

    2024

  22. [30]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,” Advances in neural information processing systems, vol. 36, pp. 34 892– 34 916, 2023

    2023

  23. [31]

    Improving clip training with language rewrites,

    L. Fan, D. Krishnan, P. Isola, D. Katabi, and Y . Tian, “Improving clip training with language rewrites,” Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  24. [32]

    mplug- 2: A modularized multi-modal foundation model across text, image and video,

    H. Xu, Q. Ye, M. Yan, Y . Shi, J. Ye, Y . Xu, C. Li, B. Bi et al., “mplug- 2: A modularized multi-modal foundation model across text, image and video,” in International Conference on Machine Learning , 2023, pp. 38 728–38 748

    2023

  25. [33]

    Dual-modal attention-enhanced text- video retrieval with triplet partial margin contrastive learning,

    C. Jiang, H. Liu, X. Yu et al. , “Dual-modal attention-enhanced text- video retrieval with triplet partial margin contrastive learning,” in ACM International Conference on Multimedia , 2023, pp. 4626–4636

    2023

  26. [34]

    Msr-vtt: A large video description dataset for bridging video and language,

    J. Xu, T. Mei, T. Yao, and Y . Rui, “Msr-vtt: A large video description dataset for bridging video and language,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2016, pp. 5288–5296

    2016

  27. [35]

    Use what you have: Video retrieval using representations from collaborative experts,

    Y . Liu, S. Albanie, A. Nagrani, and A. Zisserman, “Use what you have: Video retrieval using representations from collaborative experts,” ArXiv, vol. abs/1907.13487, 2019

    1907 arXiv

  28. [36]

    Collecting highly parallel data for paraphrase evaluation,

    D. Chen and W. B. Dolan, “Collecting highly parallel data for paraphrase evaluation,” in Aannual meeting of the association for computational linguistics: human language technologies , 2011, pp. 190–200

    2011

  29. [37]

    Localizing moments in video with natural language,

    L. Anne Hendricks, O. Wang, E. Shechtman, J. Sivic, T. Darrell, and B. Russell, “Localizing moments in video with natural language,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2017, pp. 5803–5812

    2017

  30. [38]

    Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self- knowledge distillation,

    J. Chen, S. Xiao, P. Zhang et al. , “Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self- knowledge distillation,” arXiv preprint arXiv:2402.03216 , 2024

    2024 arXiv

  31. [39]

    Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning,

    V . W. Liang, Y . Zhang, Y . Kwon et al., “Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning,”Ad- vances in Neural Information Processing Systems , vol. 35, pp. 17 612– 17 625, 2022

    2022

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.