Pith. sign in

REVIEW 5 major objections 5 minor 67 references

Beam-Guided Knowledge Replay for Knowledge-Rich Image Captioning using Vision-Language Model

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A fine-tuned vision-language captioning model that uses beam-search replay, patch self-attention, and cosine scheduling raises recognition of unseen real-world knowledge categories from 45.8% to 57.7% while keeping caption quality.

desk verdict A plausible beam-search effect on knowledge recognition is buried in a paper that never evaluates the combined model it claims to propose, and the bibliography has credibility problems. read the letter →

arxiv 2505.23358 v1 pith:PFXZKQTW submitted 2025-05-29 cs.CV

Reem AlJunaid , Muzammil Behzad This is my paper
classification cs.CV
keywords imagecaptioningknowledge-richknowledgereplayvision-languagemodelscatastrophicforgettingbeamsearchdecodingOFAKnowCap
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

KRCapVLM is a knowledge-replay method for image captioning that starts from the K-Replay framework and changes three things: pseudo-captions for replay samples are generated with beam search instead of greedy decoding, the image encoder gets an extra self-attention layer over patch embeddings, and training uses a cosine annealing scheduler. The paper's central claim is that these changes make a fine-tuned OFA-Large model produce captions that include more real-world knowledge keywords, raising recognition accuracy on the 120 unseen KnowCap categories from 45.80% to 57.69% while keeping or slightly improving standard captioning scores such as CIDEr. The significance, if true, is that a VLM can be fine-tuned for a downstream captioning task without forgetting the concept knowledge it learned in pre-training, and without needing external knowledge sources at inference time.

What carries the argument

The load-bearing mechanism is the replay loop itself: during fine-tuning on COCO caption pairs, the model also sees image-keyword pairs from the Replay CC12M subset, where a teacher OFA-Large produces pseudo-captions by beam search (width 5), and the student is trained to cover the keyword tokens in its output (coverage loss), avoid repeating them (repetition penalty), and match the teacher's softened logits (KL distillation). On top of that, patch self-attention is inserted in the image encoder and a cosine scheduler controls the combined loss, which the paper argues stabilizes the multi-loss training. This is what carries the claim: replay reinforces pretrained knowledge while COCO fine-tuning adds captioning style.

What would settle it

Compare the 120 unseen category labels against the 122 keywords used to build the Replay CC12M subset and against the keyword coverage loss; if any overlap exists, re-evaluate on a truly disjoint split to see whether the 57.69% recognition gain survives. Alternatively, recompute recognition accuracy while masking all keyword tokens that appeared in replay pseudo-captions.

Watch

Extended reading notes

Core claim

The paper claims that extending the K-Replay knowledge-replay recipe with three modifications—beam-search decoding for replay pseudo-captions, an extra self-attention layer over image patches in the OFA-Large encoder, and a cosine annealing learning-rate scheduler—makes a fine-tuned vision-language model generate captions that are more knowledge-rich without being less fluent. Concretely, on the KnowCap benchmark the scheduler-plus-beam variant raises recognition accuracy on the 120 unseen categories from 45.80% (K-Replay baseline) to 57.69%, and on the full KnowCap test the same configuration reaches 63.30%, while CIDEr stays at 92.6 versus the baseline's 90.3. The full model with all three components reaches 55.0% unseen recognition and 58.9% on the full test, which the paper reads as a balanced improvement. The conclusion's headline number is the 57.69% unseen-category improvement.

Load-bearing premise

The 120 unseen knowledge categories really are absent from the replay training data and from the keyword list that guides the knowledge coverage loss, so the 57.69% figure measures generalization to new concepts rather than memorization of seen keywords.

Editorial extensions

If this is right

  • If the central claim holds, knowledge-rich captioning can be achieved by fine-tuning a VLM alone, with no external object detectors, knowledge graphs, or metadata at inference time.
  • The unseen-category gain implies the replay mechanism does more than memorize the 122 replayed keywords; it preserves the model's ability to express concepts it was pre-trained on but never saw in replay.
  • The approach transfers to the full KnowCap test and keeps COCO captioning scores essentially unchanged, so users do not have to trade general caption quality for knowledge specificity.
  • Because the scheduler alone raised unseen recognition from 45.8% to 55.6%, the paper implies that optimization stability is itself a knowledge-retention mechanism, not just a training convenience.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next step, not tested in the paper, is swapping beam search for sampling-based decoding (e.g., top-p or nucleus sampling) with the same replay losses to see whether the recognition gain is due to beam search's higher-probability decoding or to the replay curriculum itself.
  • The paper's additive ablation leaves the interaction between the three components untested; multi-seed runs on the unseen split would show whether the 2.7-point gap between the full model (55.0%) and the beam-only configuration (57.69%) is stable or noise.
  • If the mechanism transfers, similar replay schemes could be used to keep other pretrained VLMs from forgetting domain knowledge during downstream fine-tuning in medical or robotic captioning, where rare named entities matter.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes KRCapVLM, an extension of the K-Replay framework for knowledge-rich image captioning. Three modifications are introduced: beam-search decoding for generating replay pseudo-captions, an additional self-attention layer over image patch embeddings in the OFA-Large encoder, and a cosine annealing learning-rate scheduler. The model is evaluated on the KnowCap and COCO datasets, with emphasis on knowledge recognition accuracy (Rec) on both the full test set and an 'unseen' subset of 120 categories that are not present in the replay keyword list. The conclusion claims that KRCapVLM substantially improves recognition accuracy, from 45.80% to 57.69% on unseen categories, while maintaining or improving caption quality.

Significance. If the claimed results were backed by experiments on the full KRCapVLM model, the paper would provide a useful empirical demonstration that simple decoding and training-stabilization techniques can improve knowledge retention in fine-tuned vision-language models, an area of current interest. The work builds directly on an established baseline (K-Replay) and reports comparisons against several catastrophic-forgetting methods. However, the central quantitative claim is currently not supported by the reported tables, because no row corresponds to the combined model. The paper also does not ship code or error bars, so the reliability of the component-wise improvements is unclear. As presented, the significance is contingent on the missing experiment.

major comments (5)
  1. [Section 6, Tables 1-3] The conclusion attributes the unseen-category recognition improvement from 45.80% to 57.69% to the proposed KRCapVLM model, but Tables 1 and 2 contain no row for the full KRCapVLM combination described in Section 3.1 and Algorithm 1. In Table 2, the 57.69% figure belongs to the '+Beam' row, and 45.80% is the '+K-Replay' row; the '+Scheduler' and '+Attention' rows give 55.60% and 55.00%, respectively. Since the proposed model is defined as the combination of beam search, patch self-attention, and cosine annealing, the central claim that this combination yields the reported gain is not directly tested. Please add a row for the full model, or, if 'KRCapVLM' is intended to refer to one of the ablated variants, state so explicitly and revise the conclusion accordingly.
  2. [Section 5.1, Tables 1-2] The text describes '+Scheduler', '+Beam', and '+Attention' as individual additions, but it is not stated whether these rows are cumulative over '+K-Replay' or are independent one-component ablations. If they are independent, the combined effect of all three components is unknown; if they are cumulative, the final configuration (and its metrics) is never reported. Either interpretation is inconsistent with the conclusion's attribution of the headline numbers to the assembled KRCapVLM.
  3. [Section 4.1-4.2] The definition of the unseen split and the Rec metric needs clarification to rule out metric optimisation. The paper states that the unseen 120 knowledge categories are 'not present in the predefined keyword list' (the 122 keywords used to filter the Replay CC12M subset). If Rec on the unseen set is computed using a different list of keywords, the paper should specify that list and confirm that those keywords do not appear in the replay data or in the knowledge prediction loss; otherwise, the reported 57.69% could reflect overlap between the scoring keyword list and the training signal rather than generalisation to novel concepts.
  4. [Section 5.1, Table 1] The claim that '+Beam' causes a CIDEr drop on KnowCap is contradicted by Table 1: '+K-Replay' has CIDEr 90.3 while '+Beam' has 92.6, which is an increase. The same paragraph also says beam search 'significantly increases recognition accuracy' and then discusses 'a trade-off between linguistic diversity and precision'; please reconcile the text with the reported numbers and clarify which comparison is being made.
  5. [Section 4.3] All results are from a single run with the checkpoint selected on validation performance, and no variance or significance estimates are reported. Given that several Rec differences between configurations on the unseen set are small (e.g., 55.60% vs 55.00%), multi-seed runs or confidence intervals are needed to support the claim that the individual components provide reliable gains.
minor comments (5)
  1. [Section 4.1] The manuscript contains the leftover editing instruction 'convert this to para as follows' immediately before the dataset description; this should be removed.
  2. [Section 2.2 / References] Reference [46] is cited for Vaswani et al.'s introduction of self-attention, but [46] is 'Guiding the long-short term memory model for image caption generation' by Jia et al.; the intended citation appears to be 'Attention is all you need' (Vaswani et al., 2017).
  3. [Section 5.1] The sentence 'This indicates a trade-off between linguistic diversity and precision' is unclear in context, since the cited numbers show an increase in CIDEr; please rephrase or clarify the intended comparison.
  4. [References] References [30] and [50] both cite the same CLIP paper; the duplicate should be removed.
  5. [Section 4.1] The subsampling of COCO to 27,000 pairs and Replay CC12M to 5,000 pairs is mentioned without justification; a sentence explaining why these sizes were chosen would help.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: KRCapVLM is an empirical captioning study whose reported gains are benchmark comparisons against external baselines; the central claim about unseen categories is not built into the training objective.

full rationale

This paper contains no derivation chain whose conclusions are equivalent to its inputs. The method is an empirical extension of the K-Replay framework: beam-search pseudo-caption generation, a patch self-attention layer, and cosine scheduling, evaluated on KnowCap and COCO. The loss functions (cross-entropy, knowledge prediction, and distillation) are training objectives, not derived predictions. The main generalization claim—unseen KnowCap recognition accuracy rising from 45.80% to 57.69%—concerns 120 categories explicitly excluded from the 122-keyword replay list, so the reported unseen gain is not forced by the keyword coverage loss. The full-set recognition metric does overlap with the Lkpred objective for the seen categories, but that is a standard train/eval alignment rather than a circular derivation. The more serious issue is that the conclusion attributes the 57.69% figure to the combined KRCapVLM model, whereas Tables 1 and 2 show only single-component ablations; this is an attribution and experimental-support problem, not circularity. No load-bearing self-citations or imported uniqueness theorems are present, and the empirical comparisons are anchored to external datasets and baselines.

Assumptions & free parameters 6 free parameters · 7 assumptions · 0 invented entities

The framework depends on assumptions about the teacher model, the replay data, and the evaluation protocol, plus several hand-set hyperparameters. The paper's own contribution is an architecture and decoding modification, not a derivation, so no invented entities are introduced and no formal proof is supplied.

free parameters (6)
  • lambda_k (knowledge prediction loss weight) = not reported
    Weights the knowledge coverage/repetition loss in Algorithm 1 line 15 and Eq. 4; its value is not given, and all reported results depend on it.
  • lambda_d (distillation loss weight) = not reported
    Weights the KL distillation loss in Algorithm 1 line 15 and Eq. 5; not reported.
  • Distillation temperature T = 16
    Used in Eq. 5 to soften teacher logits; chosen by hand in Section 4.3.
  • Beam size for replay decoding = 5
    Used in Algorithm 1 line 8; set by hand and changes the Rec/CIDEr trade-off.
  • Replay buffer composition = 5,000 CC12M pairs + 27,000 COCO pairs
    Section 4.1 selects these counts; the balance between current-task and replay samples is a hand-picked hyperparameter affecting the loss mixture.
  • Added self-attention layers = not specified
    The number and location of extra attention layers in the image encoder are not specified, despite being the core of the +Attention variant.
assumptions (7)
  • domain assumption OFA-Large's pretrained multimodal knowledge is accessible through CC12M image-keyword replay and can be retained via the three loss terms.
    Sections 3.2 and 3.3 build the whole framework on this assumption; no evidence is given that the replay subset is representative.
  • domain assumption A fine-tuned OFA-Large model (M_ref) is a faithful teacher whose logits preserve knowledge without adding hallucination.
    Used in Eq. 5 for distillation; the teacher is itself a fine-tuned model and may share the biases the paper wants to avoid.
  • domain assumption The 120 unseen knowledge categories are truly absent from the replay and training data and are scoreable by the same recognition metric.
    Section 4.1 describes the unseen split; the paper does not audit keyword or image-level overlap with the training and replay corpora.
  • domain assumption Knowledge Recognition Accuracy based on KnowCap keywords is a valid proxy for knowledge-richness.
    Section 4.2 defines Rec as the percentage of captions containing valid keywords; this is a keyword-presence metric, not human judgment of knowledge quality.
  • standard math Softmax with temperature and the KL divergence in Eq. 5 are the standard distillation objective.
    Standard definitions; not controversial.
  • domain assumption Cosine annealing improves convergence for the multi-loss objective, and the reported training budget (10 epochs, batch size 8) is sufficient.
    Section 3.1 and Section 4.3; no convergence analysis or loss curves are shown.
  • ad hoc to paper The repetition penalty (1 - p(w_k^i))^2 in Eq. 3 is a valid regularizer for the keyword coverage objective.
    Eq. 3 is presented without citation or derivation; it is a hand-designed penalty whose shape is not justified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beam-Guided Knowledge Replay for Knowledge-Rich Image Captioning using Vision-Language Model." pith.science (2026). https://pith.science/paper/PFXZKQTW

@misc{pith2026250523358,
  author       = {Pith},
  title        = {Pith review of: Beam-Guided Knowledge Replay for Knowledge-Rich Image Captioning using Vision-Language Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PFXZKQTW}},
  note         = {Machine review of arXiv:2505.23358}
}
read the original abstract

Generating informative and knowledge-rich image captions remains a challenge for many existing captioning models, which often produce generic descriptions that lack specificity and contextual depth. To address this limitation, we propose KRCapVLM, a knowledge replay-based novel image captioning framework using vision-language model. We incorporate beam search decoding to generate more diverse and coherent captions. We also integrate attention-based modules into the image encoder to enhance feature representation. Finally, we employ training schedulers to improve stability and ensure smoother convergence during training. These proposals accelerate substantial gains in both caption quality and knowledge recognition. Our proposed model demonstrates clear improvements in both the accuracy of knowledge recognition and the overall quality of generated captions. It shows a stronger ability to generalize to previously unseen knowledge concepts, producing more informative and contextually relevant descriptions. These results indicate the effectiveness of our approach in enhancing the model's capacity to generate meaningful, knowledge-grounded captions across a range of scenarios.

Figures

Figures reproduced from arXiv: 2505.23358 by the authors.

Figure 1
Figure 1. Illustration of our KRCapVLM framework. Knowledge replay is performed in parallel with downstream task fine [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Custom OFA-Large model with Patch Self-Attention for Image Captioning. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Comparison of captions generated by K-Replay [38] and KRCapVLM (Ours) across diverse image examples. [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

67 extracted references · 51 canonical work pages

  1. [46]

    X. Jia, E. Gavves, B. Fernando, T. Tuytelaars, Guiding the long-short term memory model for im- age caption generation, in: Proceedings of the IEEE International Conference on Computer Vision (ICCV), IEEE, 2015, pp. 2407–2415

    work page 2015

  2. [1]

    Anderson, X

    P. Anderson, X. He, C. Buehler, D. Teney, M. Johnson, S. Gould, L. Zhang, Bottom-up and top-down attention for image captioning and visual question answering, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 6077–6086

    2018

  3. [2]

    Cornia, M

    M. Cornia, M. Stefanini, L. Baraldi, R. Cucchiara, Meshed-memory transformer for image captioning, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 10578–10587

    work page 2020

  4. [3]

    Vinyals, A

    O. Vinyals, A. Toshev, S. Bengio, D. Erhan, Show and tell: A neural image caption generator, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 3156–3164

    work page 2015

  5. [4]

    Y. Qu, H. Wang, X. Li, Visually-aware context modeling for news image captioning, in: Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2024

    work page 2024

  6. [5]

    Y. Liu, R. Zhang, M. Chen, Image captioning in news report scenario, Academic Journal of Science and Technology 12 (2) (2024) 45–58

    work page 2024

  7. [6]

    Stefanini, M

    M. Stefanini, M. Cornia, L. Baraldi, S. Cascianelli, G. Fiameni, R. Cucchiara, From show to tell: A survey on deep learning-based image captioning, IEEE transactions on pattern analysis and machine intelligence 45 (1) (2022) 539–559

    work page 2022

  8. [7]

    L. Yu, M. Nikandrou, J. Jin, V. Rieser, Quality-agnostic image captioning to safely assist people with vision impairment, in: Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence (IJCAI), 2023, pp. 6281–6289.doi:10.24963/ijcai.2023/697. URLhttps://www.ijcai.org/proceedings/2023/697

    work page doi:10.24963/ijcai.2023/697 2023

Show all 67 references
  1. [8]

    Ahsan, N

    H. Ahsan, N. Bhalla, D. Bhatt, K. Shah, Multi-modal image captioning for the visually impaired, arXiv preprint arXiv:2105.08106 (2021). URLhttps://arxiv.org/abs/2105.08106

    2021 arXiv

  2. [9]

    Dognin, I

    P. Dognin, I. Melnyk, Y. Mroueh, I. Padhi, M. Rigotti, J. Ross, Y. Schiff, R. A. Young, B. Belgo- dere, Image captioning as an assistive technology: Lessons learned from vizwiz 2020 challenge, arXiv preprint arXiv:2012.11696 (2020). URLhttps://arxiv.org/abs/2012.11696

    2020 arXiv

  3. [10]

    Gurari, Y

    D. Gurari, Y. Zhao, M. Zhang, N. Bhattacharya, Captioning images taken by people who are blind, arXiv preprint arXiv:2002.08565 (2020). URLhttps://arxiv.org/abs/2002.08565

    2020 arXiv

  4. [11]

    Faurina, A

    R. Faurina, A. Jelita, A. Vatresia, I. Agustian, Image captioning to aid blind and visually impaired outdoor navigation, IAES International Journal of Artificial Intelligence 12 (3) (2023) 1104–1117. doi:10.11591/ijai.v12.i3.pp1104-1117. URLhttps://ijai.iaescore.com/index.php/...

    2023 doi

  5. [12]

    J. Jia, X. Ding, S. Pang, X. Gao, X. Xin, R. Hu, J. Nie, Image captioning based on scene graphs: A survey, Expert Systems with Applications 231 (2023) 120698. 13

    2023

  6. [13]

    Nikiforova, T

    S. Nikiforova, T. Deoskar, D. Paperno, V. Seggev, et al., Geo-aware image caption generation, in: Proceedings of the 28th International Conference on Computational Linguistics (COLING), 2020, pp. 3143–3154

    2020

  7. [14]

    S. Zhao, P. Sharma, T. Levinboim, R. Soricut, Informative image captioning with external sources of information, arXiv preprint arXiv:1906.08876 (2019)

    2019 arXiv

  8. [15]

    D. M. Vo, H. Chen, A. Sugimoto, H. Nakayama, Noc-rek: Novel object captioning with retrieved vocabulary from external knowledge, arXiv preprint arXiv:2203.14499 (2022)

    2022 arXiv

  9. [16]

    W. Zhao, Y. Hu, H. Wang, X. Wu, J. Luo, Boosting entity-aware image captioning with multi-modal knowledge graph, arXiv preprint arXiv:2107.11970 (2021)

    2021 arXiv

  10. [17]

    D. Lu, S. Whitehead, L. Huang, H. Ji, S.-F. Chang, Entity-aware image caption generation, arXiv preprint arXiv:1804.07889 (2018)

    2018 arXiv

  11. [18]

    J. Li, D. M. Vo, A. Sugimoto, H. Nakayama, Evcap: Retrieval-augmented image captioning with external visual-name memory for open-world comprehension, arXiv preprint arXiv:2311.15879 (2023)

    2023 arXiv

  12. [19]

    X. Chen, M. Jiang, Q. Zhao, Towards retrieval-augmented architectures for image captioning, ACM Transactions on Multimedia Computing, Communications, and Applications 19 (3) (2023) 1–23

    2023

  13. [20]

    Zhang, X

    Y. Zhang, X. Liu, Y. Wang, Fashion-oriented image captioning with external knowledge retrieval and fully attentive gates, Sensors 23 (3) (2023) 1286

    2023

  14. [21]

    Ayesha, J

    S. Ayesha, J. Smith, J. Doe, Generating image captions with external encyclopedic knowledge, Re- searchGate Preprint (2022)

    2022

  15. [22]

    J. Wu, Z. Hu, R. J. Mooney, Joint image captioning and question answering, arXiv preprint arXiv:1805.08389 (2018)

    2018 arXiv

  16. [23]

    J. Wu, Z. Hu, R. Mooney, Generating question relevant captions to aid visual question answering, in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019, pp. 3585–3594

    2019

  17. [24]

    Salaberria, G

    A. Salaberria, G. Azkune, O. Lopez de Lacalle, A. Soroa, E. Agirre, Image captioning for effective use of language models in knowledge-based visual question answering, Expert Systems with Applications 212 (2023) 118669

    2023

  18. [25]

    Ozdemir, E

    O. Ozdemir, E. Akagunduz, Enhancing visual question answering through question-driven image cap- tions as prompts, arXiv preprint arXiv:2404.08589 (2024)

    2024 arXiv

  19. [26]

    Q. Wu, C. Shen, A. van den Hengel, P. Wang, A. Dick, Image captioning and visual question answering based on attributes and external knowledge, arXiv preprint arXiv:1603.02814 (2016)

    2016 arXiv

  20. [27]

    K. Tran, X. He, L. Zhang, J. Sun, C. Carapcea, C. Thrasher, C. Buehler, C. Sienkiewicz, Rich image captioning in the wild, in: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, 2016, pp. 49–56

    2016

  21. [28]

    Whitehead, H

    S. Whitehead, H. Ji, M. Bansal, S.-F. Chang, C. Voss, Incorporating background knowledge into video description generation, in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018, pp. 3992–4001

    2018

  22. [29]

    G. Li, N. Duan, Y. Fang, M. Gong, D. Jiang, Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34, 2020, pp. 11336–11344. 14

    2020

  23. [30]

    Radford, J

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al., Learning transferable visual models from natural language supervision, in: Inter- national Conference on Machine Learning, Vol. 139 of Proceedings of Machine L...

    2021

  24. [31]

    J. Li, D. Li, C. Xiong, S. Hoi, Blip: Bootstrapping language-image pre-training for unified vision- language understanding and generation, in: International Conference on Machine Learning, Vol. 162 of Proceedings of Machine Learning Research, PMLR, 2022, pp. 12888–12900

    2022

  25. [32]

    P. Wang, A. Yang, R. Men, J. Lin, S. Bai, Z. Li, J. Ma, C. Zhou, J. Zhou, H. Yang, Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework, in: International Conference on Machine Learning, Vol. 162 of Proceedings of Machine ...

    2022

  26. [33]

    Z. Wang, J. Yu, A. W. Yu, Z. Dai, Y. Tsvetkov, Y. Cao, Simvlm: Simple visual language model pretraining with weak supervision, arXiv preprint arXiv:2108.10904 (2021)

    2021 arXiv

  27. [34]

    Zhang, X

    P. Zhang, X. Li, X. Hu, J. Yang, L. Zhang, L. Wang, Y. Choi, J. Gao, Vinvl: Revisiting visual representations in vision-language models, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 5579–5588

    2021

  28. [35]

    Cheng, W

    L. Cheng, W. Zhang, H. Xu, et al., Knowledge hallucination in visual-language pretraining, Journal of AI Research 58 (2023) 123–145.doi:10.1109/JAI.2023.1234567

    2023

  29. [36]

    X. Dai, L. Wang, X. Zhang, Object-level visual-language pretraining: A unified approach, Interna- tional Journal of Computer Vision 111 (2022) 200–219.doi:10.1109/IJCV.2022.1234568

    2022

  30. [37]

    J. Zhao, M. Li, Z. Chen, Mitigating generic bias in fine-tuning vlp models, IEEE Transactions on Neural Networks and Learning Systems 36 (2025) 50–60.doi:10.1109/TNNLS.2025.6789012

    2025

  31. [38]

    Cheng, W

    K. Cheng, W. Song, Z. Ma, W. Zhu, Z. Zhu, J. Zhang, Beyond generic: Enhancing image cap- tioning with real-world knowledge using vision-language pre-training model, in: Proceedings of the 31st ACM International Conference on Multimedia (MM ’23), ACM, 2023, pp. 5038–5047. doi:1...

    2023

  32. [39]

    X. Yang, X. Yao, M. Yang, F. Li, X. He, A survey on image captioning: Models, datasets, and evaluation metrics, Computer Vision and Image Understanding 193 (2020) 102906

    2020

  33. [40]

    Zhang, Z

    Y. Zhang, Z. Wang, J. Zhang, A survey on enhancing image captioning with advanced strategies and techniques, Computational and Mathematical Engineering Sciences 142 (3) (2023) 59756. URLhttps://www.techscience.com/CMES/v142n3/59756

    2023

  34. [41]

    Z. Li, X. Zhang, L. Zhang, X. Chen, H. Xu, Image captioning: A survey of models, evaluation metrics, and challenges, IEEE Access 9 (2021) 14311–14329

    2021

  35. [42]

    Z. Chao, Z. Zhang, J. Li, D. Xia, A survey on image captioning: Datasets, models, evaluation metrics, and challenges, IEEE Transactions on Multimedia 24 (2022) 2703–2723

    2022

  36. [43]

    K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhudinov, R. Zemel, Y. Bengio, Show, attend and tell: Neural image caption generation with visual attention, in: International conference on machine learning, PMLR, 2015, pp. 2048–2057

    2015

  37. [44]

    Huang, W

    L. Huang, W. Wang, J. Chen, X.-Y. Wei, Attention on attention for image captioning, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 4634–4643. 15

    2019

  38. [45]

    X. Xiao, L. Wang, K. Ding, S. Xiang, C. Pan, Deep hierarchical encoder–decoder network for image captioning, IEEE Transactions on Multimedia 21 (11) (2019) 2942–2956.doi:10.1109/TMM.2019. 2915033

    2019 doi

  39. [47]

    Devlin, M.-W

    J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, in: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, A...

    2019

  40. [48]

    F. Chen, R. Ji, J. Su, Y. Wu, Y. Wu, Structcap: Structured semantic embedding for image captioning, in: Proceedings of the 25th ACM International Conference on Multimedia, 2017, pp. 46–54

    2017

  41. [49]

    F. Chen, R. Ji, X. Sun, Y. Wu, J. Su, Groupcap: Group-based image captioning with structured relevance and diversity constraints, 2017

    2017

  42. [50]

    Radford, J

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al., Learning transferable visual models from natural language supervision, in: Interna- tional Conference on Machine Learning, Vol. 139, 2021, pp. 8748–8763

    2021

  43. [51]

    Mokady, A

    R. Mokady, A. Hertz, A. H. Bermano, Clipcap: Clip prefix for image captioning, arXiv preprint arXiv:2111.09734 (2021)

    2021 arXiv

  44. [52]

    X. Li, X. Yin, C. Li, P. Zhang, X. Hu, L. Zhang, L. Wang, H. Hu, L. Dong, F. Wei, et al., Os- car: Object-semantics aligned pre-training for vision-language tasks, in: European Conference on Computer Vision, Springer, 2020, pp. 121–137

    2020

  45. [53]

    J. Wang, Z. Yang, X. Hu, L. Li, K. Lin, Z. Gan, Z. Liu, C. Liu, L. Wang, Git: A generative image- to-text transformer for vision and language, arXiv preprint arXiv:2205.14100 (2022)

    2022 arXiv

  46. [54]

    A. F. Biten, L. Gomez, M. Rusinol, D. Karatzas, Good news, everyone! context driven entity-aware captioning for news images, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 12466–12475

    2019

  47. [55]

    13035–13045

    A.Tran, A.Mathews, L.Xie, Transformandtell: Entity-awarenewsimagecaptioning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 13035–13045

    2020

  48. [56]

    OFA-Sys, Ofa-large model,https://huggingface.co/OFA-Sys/ofa-large, accessed: 2025-05-19 (2022)

    2022

  49. [57]

    Changpinyo, P

    S. Changpinyo, P. Sharma, N. Ding, R. Soricut, Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 3558–3568.doi:10.1109/CVPR46437.2021. 00350

    2021

  50. [58]

    Kullback, R

    S. Kullback, R. A. Leibler, On information and sufficiency, Annals of Mathematical Statistics 22 (1) (1951) 79–86

    1951

  51. [59]

    X. Chen, H. Fang, T.-Y. Lin, R. Vedantam, S. Gupta, P. Dollár, C. L. Zitnick, Microsoft coco captions: Data collection and evaluation server, arXiv preprint arXiv:1504.00325 (2015). URLhttps://arxiv.org/abs/1504.00325 16

    2015 arXiv

  52. [60]

    Papineni, S

    K. Papineni, S. Roukos, T. Ward, W.-J. Zhu, Bleu: a method for automatic evaluation of machine translation, in: Proceedings of the 40th annual meeting of the Association for Computational Lin- guistics (ACL), 2002, pp. 311–318

    2002

  53. [61]

    Banerjee, A

    S. Banerjee, A. Lavie, Meteor: An automatic metric for mt evaluation with improved correlation with human judgments, in: Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for MT and/or Summarization, 2005, pp. 65–72

    2005

  54. [62]

    Lin, Rouge: A package for automatic evaluation of summaries, in: Text summarization branches out: Proceedings of the ACL-04 workshop, 2004, pp

    C.-Y. Lin, Rouge: A package for automatic evaluation of summaries, in: Text summarization branches out: Proceedings of the ACL-04 workshop, 2004, pp. 74–81

    2004

  55. [63]

    Vedantam, C

    R. Vedantam, C. Lawrence Zitnick, D. Parikh, Cider: Consensus-based image description evaluation, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 4566– 4575

    2015

  56. [64]

    Kirkpatrick, R

    J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, et al., Overcoming catastrophic forgetting in neural networks, Proceedings of the National Academy of Sciences 114 (13) (2017) 3521–3526

    2017

  57. [65]

    S. Chen, Y. Hou, Y. Cui, W. Che, T. Liu, X. Yu, Recall and learn: Fine-tuning deep pretrained language models with less forgetting, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics, 20...

    2020

  58. [66]

    R. Xu, F. Luo, Z. Zhang, C. Tan, B. Chang, S. Huang, F. Huang, Raise a child in large language model: Towards effective and generalizable fine-tuning, in: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021, pp. 9514–9528

    2021

  59. [67]

    P. Gao, S. Geng, R. Zhang, T. Ma, R. Fang, Y. Zhang, H. Li, Y. Qiao, Clip-adapter: Better vision- language models with feature adapters, arXiv preprint arXiv:2110.04544 (2021). 17

    2021 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.