Pith. sign in

REVIEW 3 major objections 4 minor 44 references

DIR: Retrieval-Augmented Image Captioning with Comprehensive Understanding

T0 review · 3 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that training an image-captioning encoder with a frozen diffusion denoising objective and retrieving text from a database decomposed into objects, actions, and environments improves out-of-domain captioning at no extra…

desk verdict Diffusion-guided retrieval features and an attribute-rich database give small but consistent captioning gains; the feature-space mismatch and missing error bars need fixing before the claims hold. read the letter →

arxiv 2412.01115 v1 pith:LZCWOGDD submitted 2024-12-02 cs.CV

Hao Wu , Zhihang Zhong , Xiao Sun This is my paper · ORCID
classification cs.CV
keywords retrieval-augmentedimagecaptioningdiffusionguidanceout-of-domaingeneralizationimage-textretrievalattributedecompositionfrozenmodelfrequency-basedfilteringzero-shot
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that retrieval-augmented image captioning can generalize to new domains without any inference-time cost if the image features used for retrieval encode the whole image rather than only the ground-truth caption's viewpoint. DIR does this in two steps: a frozen text-to-image diffusion model is used during training to force the image encoder to reconstruct noisy images from its feature vector, and the retrieval database stores captions parsed into objects, actions, and environments instead of raw sentences or object lists. Trained only on COCO, the model improves Flickr30k and NoCaps out-of-domain scores relative to the lightweight EVCap baseline while keeping in-domain COCO and NoCaps performance competitive, with the same learnable parameter count. A sympathetic reader would care because the result suggests that a training-only auxiliary objective plus a richer key-value store can buy generalization for deployed captioning systems at zero added latency.

What carries the argument

The mechanism that carries the argument is the joint training objective $L_{\text{total}} = L_{\text{caption}} + \lambda L_{\text{denoise}}$ (Equation 5). In $L_{\text{denoise}}$, the image encoder's feature vector $z$ conditions a frozen latent diffusion model that must predict the Gaussian noise added to a noised image, so $z$ has to retain reconstructable holistic content; $\lambda$ balances this against next-token caption prediction. On the retrieval side, the database stores raw EVA-CLIP image features as keys, while each training image's captions are parsed into object, action, and environment terms and filtered by a top-$n$ frequency rule; the surviving attributes are assembled into a soft prompt such as "[objects] and [actions] in [scenes]" and encoded by BERT. A Text Q-Former then fuses these retrieved text features with the image features, and Vicuna-13B generates the caption from the concatenation. All novel machinery except the database construction is active only during training, so inference adds no cost.

What would settle it

Measure retrieval quality directly: take held-out COCO images, compute their diffusion-trained query features, retrieve from the EVA-CLIP database, and check whether the top-k retrieved captions come from the same image at a rate clearly above chance. If retrieval accuracy is near random, or if substituting plain EVA-CLIP features as queries gives the same or better captioning scores, then the reported out-of-domain gains cannot be attributed to the diffusion-guided features.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that supervision from a denoising objective transfers to retrieval: conditioning a frozen Stable Diffusion UNet on the image encoder's feature vector and asking it to predict the injected noise, jointly with the captioning loss, makes the encoder keep fine-grained, whole-scene information that caption-only training discards. The second discovery is that retrieved text helps most when it is semantically decomposed: a database keyed by EVA-CLIP image features and valued with frequency-filtered objects, actions, and environments provides context that generalizes to out-of-domain images better than raw captions or object-only memories. Experimentally, DIR trained solely on COCO surpasses lightweight methods on Flickr30k and NoCaps out-of-domain and overall splits and stays competitive in-domain, using the same number of learnable parameters as the EVCap framework it extends; because the diffusion model is frozen and used only in training, the inference pipeline is identical in cost.

Load-bearing premise

The argument leans on an unstated compatibility: features produced by the diffusion-trained BLIP2-style encoder and the EVA-CLIP features stored in the database must live in a space where nearest-neighbor cosine matching genuinely picks relevant captions, and the paper describes no projection or learned alignment between these two spaces.

Editorial extensions

If this is right

  • With only COCO training data, DIR improves CIDEr and SPICE on Flickr30k and the out-of-domain and overall NoCaps validation sets compared with the lightweight EVCap baseline.
  • Diffusion-guided feature learning reduces the annotator-bias problem by making retrieval features answer to the image itself, not just to one human-written caption.
  • A retrieval database decomposed into objects, actions, and environments, with frequency-based top-n filtering, supplies more transferable context than raw captions or LVIS-only object names.
  • The diffusion loss weight $\lambda$ is a practical control: larger values help out-of-domain generalization until denoising begins to dominate the caption objective.
  • Keeping retrieval features as raw image features, rather than fusing them with retrieved text before matching, works better for both in-domain and out-of-domain captioning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the query and key feature spaces are genuinely compatible, a testable extension is to store diffusion-trained features in the database as well; that would remove the need for a separate EVA-CLIP feature extractor and may improve matching further.
  • The same training-only diffusion guidance could transfer to other retrieval-augmented multimodal tasks, such as video captioning or visual question answering, wherever GT annotations cover only one perspective of the input.
  • The biggest unresolved contribution split is whether the gains come from the better query features, the richer database, or their interaction; ablating the database with diffusion guidance held fixed, and vice versa, would isolate the two effects.
  • The 'no inference cost' claim presumes a static offline database; a dynamic retrieval corpus that must be re-indexed would incur maintenance overhead that the paper does not account for.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper presents DIR, a retrieval-augmented image captioning model built on EVCap. DIR has two main components: a diffusion-guided training objective in which a frozen Stable Diffusion model reconstructs noisy images conditioned on the image encoder's features, and a retrieval database that stores parsed object/action/environment attributes with frequency-based top-n filtering. The model is trained on COCO and evaluated on COCO, Flickr30k, and NoCaps. The authors report that DIR maintains or slightly improves in-domain CIDEr/SPICE compared with EVCap and improves out-of-domain metrics, with no additional inference cost because diffusion guidance is used only during training.

Significance. If the results hold, the paper would provide a simple, training-only mechanism for improving retrieval-augmented captioning out of distribution while keeping inference unchanged. The paper's strengths include the clear architectural presentation, the component-wise ablations in Tables 3-6, the use of the same parameter count and training data as EVCap in the main comparison, and the qualitative illustrations of retrieval effects. However, the significance is currently limited by three issues: the retrieval space compatibility of the query and database features is not established, the reported gains are small and are not accompanied by any uncertainty quantification or a clean hyperparameter-selection protocol, and the claimed value of the environment attribute is not isolated by ablation.

major comments (3)
  1. [§3.2–§3.3, Fig. 2] The query and database features are never placed in a common space. Section 3.3 says the database stores 'raw features extracted by EV A-CLIP' and explicitly prioritizes them over the diffusion-trained encoder's features, while Section 3.2 and Figure 2 use the diffusion-trained BLIP2 image/Q-Former output as the query for matching. No projection, normalization, or alignment is described, and the contrastive-alignment justification for EVA-CLIP does not automatically transfer to a Q-Former output trained with captioning plus denoising losses. If the two spaces are not comparable, the match step in Figure 2 is not meaningful nearest-neighbor retrieval, and the ablations in Tables 3–4 cannot be attributed to 'better retrieval.' Please report a direct retrieval-quality evaluation (e.g., recall@k or precision of retrieved attributes for diffusion-guided versus EVA-CLIP queries) or introduce and validate an alignment/projection between the query and database spaces.
  2. [§4.4, Table 7, Suppl. Fig. 5] The reported improvements over EVCap are small (Table 1: Flickr30k CIDEr 85.7 vs 84.4; NoCaps out-of-domain 118.7 vs 116.5; Table 4: Flickr30k 85.7 vs 83.9), but no error bars, multiple seeds, or significance tests are reported in Tables 1–9. In addition, the diffusion loss weight λ is selected on the evaluation datasets themselves (Table 7), and top-n is selected on the same datasets (Supplementary Figure 5); for instance, λ=5 gives the best Flickr30k CIDEr while λ=7 is the reported configuration based on NoCaps. This makes the abstract's 'significantly improves' claim unsupported. Please add variance estimates, multiple-seed runs, or a properly separated validation protocol for hyperparameters.
  3. [§3.3, §4.4 (Table 3)] The contribution of the environment attribute is not isolated. Table 3 compares 'Ours' against 'EVCap's' database, but the two differ simultaneously in attribute categories (objects, actions, environments), vocabulary coverage, and the frequency-filtering mechanism. To support the claim in Section 3.3 that 'the environment... provides valuable context,' the paper should include an ablation with objects and actions only, and a second variant with environments added, while keeping the rest of the pipeline fixed.
minor comments (4)
  1. [Figures 3 and 4] Figures 3 and 4 contain unfinished annotations: Chinese text in Figure 3 ('feature done 480、1296 可以作为额外的分析，可以发现玉米，即使没有检索到') and English fragments in Figure 4 ('Rt database done', 'Setting: 277:use lvis retrieval database', 'Setting: 169: best', and empty 'LVIS () Ours: ()' labels). These must be cleaned or translated before resubmission.
  2. [§3.3 / Suppl. §8] The main text never states that the frequency-based filter uses top-n=3; the value appears only in the supplementary. Move this choice into Section 3.3 or the experimental settings.
  3. [References] Reference formatting is inconsistent: several entries are arXiv identifiers without venues (e.g., [8], [24]), and some entries mix conference and page-range styles (e.g., [3]). Normalize to the journal style.
  4. [Supplementary Figure 5] Supplementary Figure 5 has no axis labels or legend; add them so the top-n trade-off can be read directly.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: DIR is an empirical comparison with external benchmarks and no derivation reduces to its inputs.

full rationale

DIR is an empirical method-comparison paper, not a derivation. The diffusion loss (Eq. 2) conditions a frozen Stable Diffusion model on image features z = E(x0) to reconstruct noisy images; this is a self-supervised auxiliary loss, and the caption loss (Eq. 4) is standard next-token prediction on COCO. No predicted quantity is defined in terms of the fitted constants: the retrieval database stores EV A-CLIP features (Sec. 3.3) and retrieval is an internal mechanism, not an output. The final CIDEr/SPICE numbers on Flickr30k and NoCaps are independent external benchmarks, and the cited prior work (EVCap, DIVA, Diffusion-TTA) is not from the present authors. Hyperparameters lambda and top-n are selected by ablations on the same evaluation sets (Tables 7 and Fig. 5), which is a test-set-overfitting concern, but that is selection bias, not a constructional reduction of the reported metric to the fit; the central claim does not reduce to a fitted parameter renamed as a prediction. Therefore no circular step is present.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central method relies on standard pretrained models and two design assumptions about feature compatibility and text parsing; it introduces no new physical or mathematical entities. The two free parameters (λ and top-n) are tuned on the evaluation datasets, which weakens the independence of the reported numbers.

free parameters (2)
  • diffusion loss weight λ = 7
    Selected by scanning λ ∈ {3,5,7,9} on the evaluation datasets (Table 7); final performance depends on this choice.
  • top-n retrieved attributes per category = 3
    Selected based on CIDEr across COCO, Flickr30k, and NoCaps validation sets (Figure 5); top-n=3 is reported as the best balance.
assumptions (3)
  • domain assumption Pretrained components (BLIP2, Stable Diffusion v1.4, Vicuna-13B, BERT, EV A-CLIP) transfer appropriately to the captioning task when used as fixed or lightly trained backbones.
    The entire method rests on these pretrained models; no re-training or adaptation of them is described beyond the image encoder and Q-Formers.
  • ad hoc to paper Query features from the diffusion-guided BLIP2 encoder are compatible with EV A-CLIP database features for retrieval matching.
    Section 3.3 stores EV A-CLIP features in the database while Section 3.2 and Figure 2 use the diffusion-trained encoder output as the retrieval query; no projection or alignment is described.
  • domain assumption spaCy-based parsing into objects, actions, and environments, followed by top-n frequency filtering, preserves semantically useful retrieval context.
    The whole high-quality retrieval database relies on this NLP pipeline; its benefit is only measured indirectly through final caption metrics.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DIR: Retrieval-Augmented Image Captioning with Comprehensive Understanding." pith.science (2026). https://pith.science/paper/LZCWOGDD

@misc{pith2026241201115,
  author       = {Pith},
  title        = {Pith review of: DIR: Retrieval-Augmented Image Captioning with Comprehensive Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LZCWOGDD}},
  note         = {Machine review of arXiv:2412.01115}
}
read the original abstract

Image captioning models often suffer from performance degradation when applied to novel datasets, as they are typically trained on domain-specific data. To enhance generalization in out-of-domain scenarios, retrieval-augmented approaches have garnered increasing attention. However, current methods face two key challenges: (1) image features used for retrieval are often optimized based on ground-truth (GT) captions, which represent the image from a specific perspective and are influenced by annotator biases, and (2) they underutilize the full potential of retrieved text, typically relying on raw captions or parsed objects, which fail to capture the full semantic richness of the data. In this paper, we propose Dive Into Retrieval (DIR), a method designed to enhance both the image-to-text retrieval process and the utilization of retrieved text to achieve a more comprehensive understanding of the visual content. Our approach introduces two key innovations: (1) diffusion-guided retrieval enhancement, where a pretrained diffusion model guides image feature learning by reconstructing noisy images, allowing the model to capture more comprehensive and fine-grained visual information beyond standard annotated captions; and (2) a high-quality retrieval database, which provides comprehensive semantic information to enhance caption generation, especially in out-of-domain scenarios. Extensive experiments demonstrate that DIR not only maintains competitive in-domain performance but also significantly improves out-of-domain generalization, all without increasing inference costs.

Figures

Figures reproduced from arXiv: 2412.01115 by the authors.

Figure 1
Figure 1. Comparison of our method with previous retrieval [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The architecture of our retrieval-augmented image captioning framework employs an image encoder and Q-Former from BLIP2, [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. Qualitative comparison of model predictions using EV [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Comparison of model performance with varying top- [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Comparison of word clouds illustrating differences [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Qualitative comparison of captions generated by SmallCap, EVCap, and our proposed model on COCO, Flickr30k, and NoCaps [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

44 extracted references · 31 canonical work pages

  1. [1]

    nocaps: novel object caption- ing at scale

    Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Ste- fan Lee, and Peter Anderson. nocaps: novel object caption- ing at scale. International Conference on Computer Vision, pages 8947–8956, 2019. 6

    work page 2019

  2. [2]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Men- sch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Se- bastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sa- hand Sharifzadeh, Mikolaj ...

    work page 2022

  3. [3]

    Spice: Semantic propositional image cap- tion evaluation

    Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image cap- tion evaluation. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, Octo- ber 11-14, 2016, Proceedings, Part V 14 , pages 382–398. Springer, 2016. 6

    2016

  4. [4]

    Bottom-up and top-down attention for image captioning and visual question answering

    Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6077– 6086, 2017. 2

    work page 2018

  5. [5]

    Self-RAG: Learning to retrieve, generate, and critique through self-reflection

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Han- naneh Hajishirzi. Self-RAG: Learning to retrieve, generate, and critique through self-reflection. In The Twelfth Interna- tional Conference on Learning Representations, 2024. 2, 3

    work page 2024

  6. [6]

    Meteor: An automatic metric for mt evaluation with improved correlation with hu- man judgments

    Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with hu- man judgments. In Proceedings of the acl workshop on in- trinsic and extrinsic evaluation measures for machine trans- lation and/or summarization, pages 65–72, 2005. 6

    2005

  7. [7]

    Label-efficient se- mantic segmentation with diffusion models

    Dmitry Baranchuk, Andrey V oynov, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Label-efficient se- mantic segmentation with diffusion models. In International Conference on Learning Representations, 2022. 3

    2022

  8. [8]

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023) , 2(3):6,

    2023

Show all 44 references
  1. [9]

    Bert: Pre-training of deep bidirectional trans- formers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional trans- formers for language understanding. In North American Chapter of the Association for Computational Linguistics ,

  2. [10]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. 6

    2010 arXiv

  3. [11]

    Eva: Exploring the limits of masked visual representation learning at scale

    Yuxin Fang, Wen Wang, Binhui Xie, Quan-Sen Sun, Ledell Yu Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages ...

    2023

  4. [12]

    Transferable decoding with visual entities for zero-shot image captioning

    Junjie Fei, Teng Wang, Jinrui Zhang, Zhenyu He, Chengjie Wang, and Feng Zheng. Transferable decoding with visual entities for zero-shot image captioning. 2023 IEEE/CVF In- ternational Conference on Computer Vision (ICCV) , pages 3113–3123, 2023. 3, 6

    2023

  5. [13]

    Retrieval augmented language model pre- training

    Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre- training. In International conference on machine learning , pages 3929–3938. PMLR, 2020. 2, 3

    2020

  6. [14]

    Deep visual-semantic align- ments for generating image descriptions

    Andrej Karpathy and Li Fei-Fei. Deep visual-semantic align- ments for generating image descriptions. 2015 IEEE Confer- ence on Computer Vision and Pattern Recognition (CVPR) , pages 3128–3137, 2014. 2, 6

    2015

  7. [15]

    Do you remember? dense video captioning with cross-modal memory retrieval

    Minkuk Kim, Hyeon Bae Kim, Jinyoung Moon, Jinwoo Choi, and Seong Tae Kim. Do you remember? dense video captioning with cross-modal memory retrieval. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13894–13904, 2024. 3

    2024

  8. [16]

    Retrieval-augmented generation for knowledge-intensive nlp tasks

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K¨uttler, Mike Lewis, Wen-tau Yih, Tim Rockt ¨aschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Syst...

    2020

  9. [17]

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In In- ternational Conference on Machine Learning, 2022. 6

    2022

  10. [18]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International Conference on Machine Learning, 2023. 1, 3, 6

    2023

  11. [19]

    Evcap: Retrieval-augmented image captioning 9 with external visual-name memory for open-world compre- hension

    Jiaxuan Li, Duc Minh V o, Akihiro Sugimoto, and Hideki Nakayama. Evcap: Retrieval-augmented image captioning 9 with external visual-name memory for open-world compre- hension. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13733–13742, 2023. ...

    2024

  12. [20]

    Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll ´ar, and C

    Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll ´ar, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision, 2014. 6, 11

    2014

  13. [21]

    Prefix-diffusion: A lightweight diffusion model for diverse image captioning

    Guisheng Liu, Yi Li, Zhengcong Fei, Haiyan Fu, Xiangyang Luo, and Yanqing Guo. Prefix-diffusion: A lightweight diffusion model for diverse image captioning. ArXiv, abs/2309.04965, 2023. 3

    2023 arXiv

  14. [22]

    Diffusion hyperfeatures: Searching through time and space for semantic correspondence

    Grace Luo, Lisa Dunlap, Dong Huk Park, Aleksander Holyn- ski, and Trevor Darrell. Diffusion hyperfeatures: Searching through time and space for semantic correspondence. Ad- vances in Neural Information Processing Systems, 36, 2024. 3

    2024

  15. [23]

    Semantic-conditional dif- fusion networks for image captioning*

    Jianjie Luo, Yehao Li, Yingwei Pan, Ting Yao, Jianlin Feng, Hongyang Chao, and Tao Mei. Semantic-conditional dif- fusion networks for image captioning*. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 23359–23368, 2022. 3

    2023

  16. [24]

    Clipcap: Clip prefix for image captioning

    Ron Mokady and Amir Hertz. Clipcap: Clip prefix for image captioning. ArXiv, abs/2111.09734, 2021. 6

    2021 arXiv

  17. [25]

    Junting Pan, Ziyi Lin, Yuying Ge, Xiatian Zhu, Ren- rui Zhang, Yi Wang, Yu Jiao Qiao, and Hongsheng Li. Retrieving-to-answer: Zero-shot video question answering with frozen large language models.2023 IEEE/CVF Interna- tional Conference on Computer Vision Workshops (ICCVW), pag...

    2023

  18. [26]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Annual Meeting of the Association for Com- putational Linguistics, 2002. 6

    2002

  19. [27]

    Plummer, Liwei Wang, Christopher M

    Bryan A. Plummer, Liwei Wang, Christopher M. Cervantes, Juan C. Caicedo, J. Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspon- dences for richer image-to-sentence models. International Journal of Computer Vision, 123:74 – 93, 2015. 6

    2015

  20. [28]

    Li, Deepak Pathak, and Katerina Fragkiadaki

    Mihir Prabhudesai, Tsung-Wei Ke, Alexander C. Li, Deepak Pathak, and Katerina Fragkiadaki. Diffusion-tta: Test-time adaptation of discriminative models via generative feedback. ArXiv, abs/2311.16102, 2023. 3

    2023 arXiv

  21. [29]

    Smallcap: Lightweight image captioning prompted with retrieval augmentation

    Rita Parada Ramos, Bruno Martins, Desmond Elliott, and Yova Kementchedjhieva. Smallcap: Lightweight image captioning prompted with retrieval augmentation. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2840–2849, 2022. 2, 3, 6

    2023

  22. [30]

    Retrieval-augmented image captioning

    Rita Parada Ramos, Desmond Elliott, and Bruno Martins. Retrieval-augmented image captioning. In Conference of the European Chapter of the Association for Computational Lin- guistics, 2023. 3

    2023

  23. [31]

    Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer

    Robin Rombach, A. Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10674–10685, 2021. 4, 6, 11

    2022

  24. [32]

    Ilya Sutskever, James Martens, and Geoffrey E. Hinton. Gen- erating text with recurrent neural networks. In International Conference on Machine Learning, 2011. 2

    2011

  25. [33]

    Attention is all you need

    A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017. 2

    2017

  26. [34]

    Lawrence Zitnick, and Devi Parikh

    Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evalua- tion. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4566–4575, 2014. 6

    2015

  27. [35]

    Diffusion feedback helps clip see better

    Wenxuan Wang, Quan Sun, Fan Zhang, Yepeng Tang, Jing Liu, and Xinlong Wang. Diffusion feedback helps clip see better. arXiv preprint arXiv:2407.20171, 2024. 3

    2024 arXiv

  28. [36]

    Language models with im- age descriptors are strong few-shot video-language learners

    Zhenhailong Wang, Manling Li, Ruochen Xu, Luowei Zhou, Jie Lei, Xudong Lin, Shuohang Wang, Ziyi Yang, Chen- guang Zhu, Derek Hoiem, et al. Language models with im- age descriptors are strong few-shot video-language learners. arXiv preprint arXiv:2205.10747, 2022. 5

    2022 arXiv

  29. [37]

    Re-vilm: Retrieval-augmented visual lan- guage model for zero and few-shot image captioning

    Zhuolin Yang, Wei Ping, Zihan Liu, Vijay Anand Kor- thikanti, Weili Nie, De-An Huang, Linxi (Jim) Fan, Zhid- ing Yu, Shiyi Lan, Bo Li, Mingyan Liu, Yuke Zhu, Mo- hammad Shoeybi, Bryan Catanzaro, Chaowei Xiao, and An- ima Anandkumar. Re-vilm: Retrieval-augmented visual lan- gua...

    2023

  30. [38]

    Retrieval-augmented mul- timodal language modeling

    Michihiro Yasunaga, Armen Aghajanyan, Weijia Shi, Rich James, Jure Leskovec, Percy Liang, Mike Lewis, Luke Zettlemoyer, and Wen tau Yih. Retrieval-augmented mul- timodal language modeling. ArXiv, abs/2211.12561, 2023. 3

    2023 arXiv

  31. [39]

    Meacap: Memory-augmented zero- shot image captioning

    Zequn Zeng, Yan Xie, Hao Zhang, Chiyu Chen, Zhengjue Wang, and Boli Chen. Meacap: Memory-augmented zero- shot image captioning. 2024 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 14100– 14110, 2024. 3, 5

    2024

  32. [40]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. ArXiv, abs/2304.10592, 2023. 1, 3, 6 10 DIR: Retrieval-Augmented Image Captioning with Comprehensive Understanding Sup...

    2023 arXiv

  33. [41]

    This approach significantly reduces memory and processing requirements while preserving critical visual information

    Implementation Details of Diffusion Guid- ance We adopt the latent diffusion model Stable Diffusion 1- 4 [31] to implement diffusion guidance, enabling computa- tionally efficient denoising by operating on latent features rather than the original image resolution. This approac...

  34. [42]

    NOUN” or “PROPN

    Construction of Retrieval Database The retrieval database is constructed using the training set of the COCO [20] dataset. For each image, we extract image features using EV A-CLIP [11], which maps images and text to a shared embedding space. To enrich contextual infor- mation,...

  35. [43]

    While the SPICE scores remain similar across different top- n values, the CIDEr scores exhibit more noticeable variation

    Additional Ablation Analysis Effect of the Number of Retrieved Terms (top- n) To in- vestigate the impact of the number of retrieved terms (top- n) on model performance, we visualize the CIDEr metric across three datasets: COCO, Flickr30k, and the out-of- domain subset of NoCa...

  36. [44]

    Visualization Results Comparison of Retrieval Databases We analyze the word diversity of the retrieval databases used in our model and EVCap by comparing their respective word clouds, as illustrated in Figure 6. The word cloud generated from our retrieval database demonstrates...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.