Pith. sign in

REVIEW 3 major objections 5 minor 68 references

L-CLIPScore: a Lightweight Embedding-based Captioning Metric for Evaluating and Training

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A 99M-parameter distilled CLIP matches the 338M teacher as a caption evaluator, and its score, blended with CIDEr, improves captioning training.

desk verdict Judge-side compression results are credible, but the supervisor experiments have an internal inconsistency between the equation, Table V, and Table VI that must be fixed. read the letter →

arxiv 2507.08710 v1 pith:VH3AKZC5 submitted 2025-07-11 cs.CV

Li Li , Yingzhe Peng , Xu Yang , Ruoxi Cheng , Haiyang Xu , Ming Yan , Fei Huang This is my paper
classification cs.CV
keywords imagecaptioningcaptionevaluationmetricCLIPcompressionknowledgedistillationvision-languagealignmentself-criticalsequencetrainingSimilarityRegulatorlossembedding-based
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that CLIP's vision-language alignment knowledge is over-parameterized for caption evaluation. By compressing the dual encoders with weight multiplexing and matrix decomposition (338M down to 99M parameters) and distilling with a new multi-modal Similarity Regulator loss, the student L-CLIP reaches human-correlation and image-text matching scores comparable to the teacher CLIP. The authors then propose L-CLIPScore, computed from L-CLIP, as a lightweight judge of caption quality and show it can also serve as a reinforcement-learning training reward, provided it is mixed with an n-gram metric like CIDEr. With the mixed reward, captioning models beat pure-CIDEr training on both n-gram and embedding metrics. If correct, embedding-based caption evaluation no longer requires a large model, and caption training can use a cheap visual-alignment signal without sacrificing fluency.

What carries the argument

The load-bearing object is the Similarity Regulator (SR) loss, Eq. (2), which distills alignment by comparing teacher and student cosine similarities on matched and non-matched image-caption pairs in a batch: it pushes the student's similarity for a matched pair up to at least the teacher's value, and pushes non-matched similarities down to at most the teacher's value. This contrastive-style distillation transfers the teacher's multi-modal alignment without needing CLIP-scale data. The architecture also relies on weight multiplexing, which turns a few shared Transformer blocks into more blocks through learnable sub-networks, and matrix decomposition, which replaces a 49408x768 word embedding matrix with two smaller matrices.

What would settle it

Run L-CLIPScore on a fresh human-rated image-caption benchmark that was not used to design or tune the model and compare its Kendall correlation with the reported 53-56 range; if it falls near the n-gram baselines or below chance, the claimed alignment transfer fails.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that most of CLIP's vision-language alignment ability survives a 3.4x compression when the student is trained with a multi-modal Similarity Regulator loss. The resulting L-CLIP (99M parameters, about 100 ms faster than the 338M teacher on a 128-pair batch) yields L-CLIPScore values whose Kendall correlations with human ratings are comparable to CLIP's: with references, 53.8/36.0/56.5 versus 53.0/36.4/55.4 on Flickr8k-Expert, Flickr8k-CF, and Composite. It also slightly exceeds the teacher on Pascal-50S mean accuracy (83.3 versus 83.1) and FOIL accuracy (93.73 versus 93.43). The paper further claims that L-CLIPScore works as a sentence-level training reward only when mixed with an n-gram metric: pure L-CLIPScore raises the embedding score while collapsing language, whereas mixing it with CIDEr improves CIDEr from 114.5 to 118.7 on bottom-up features and from 117.1 to 121.2 on VinVL features.

Load-bearing premise

The load-bearing premise is that the teacher CLIP's similarity scores on the distillation batches are reliable soft targets; if those similarities are noisy or biased, the student inherits that bias and every downstream claim about L-CLIPScore as judge and supervisor is weakened.

Editorial extensions

If this is right

  • L-CLIPScore can serve as a reference-free caption evaluation metric in settings where the full CLIP is too large or too slow, since L-CLIP uses 99M parameters and about half the inference time of the teacher.
  • When human references are available, the reference-based RefL-CLIPScore generally outperforms n-gram metrics on human correlation, Pascal-50S sentence-pair accuracy, and FOIL robustness, according to the paper's tables.
  • Using L-CLIPScore alone as the self-critical training reward produces higher L-CLIPScore but broken grammar and repeated tokens, so embedding-based rewards need an n-gram constraint to remain linguistically valid.
  • The mixed reward, combining L-CLIPScore with CIDEr, improves captioning models on both n-gram and embedding metrics compared with pure CIDEr training.
  • The Similarity Regulator loss, not the compression alone, is what closes most of the gap to the teacher in the paper's ablations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same compress-and-distill recipe should transfer to other vision-language models: pick a large dual-encoder teacher, compress with weight sharing or low-rank embeddings, and distill alignment with a similarity-regulator loss; the ablations suggest the loss carries the alignment knowledge.
  • The language-collapse result is evidence for a general property: an embedding-only reward in self-critical training is hackable, so any embedding metric used as an RL reward likely needs a syntactic or n-gram constraint.
  • The paper selects its recommended mixing weight on the test split; choosing the weight on a held-out validation split would make the recommended ratio more portable to other captioning models and datasets.
  • Distilling several vision-language teachers into one lightweight student could mitigate the model-preference bias the paper identifies as a limitation of embedding-based metrics.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes L-CLIP, a 99M-parameter dual-encoder vision-language model obtained by compressing ViT-B/32 CLIP via weight multiplexing, matrix decomposition, and a new multi-modal Similarity Regulator (SR) distillation loss. L-CLIPScore is then defined as the cosine similarity between L-CLIP image and text embeddings, optionally combined with reference captions, and is proposed as both a caption-evaluation metric (judge) and a reward for self-critical sequence training (supervisor). The judge experiments report Kendall correlations, Pascal-50S accuracy, and FOIL accuracy showing L-CLIP is close to or slightly better than the teacher CLIP while being smaller and faster. The supervisor experiments report that training with pure L-CLIPScore causes language collapse, while a mixed reward of CIDEr and L-CLIPScore improves CIDEr from 114.5 to 118.7 (bottom-up features) and from 117.1 to 121.2 (VinVL features) over pure CIDEr, with an ablation over the mixing weight α recommending α=0.7.

Significance. If the results are correct, the paper makes two useful contributions. First, it demonstrates a concrete recipe for compressing a large CLIP model to roughly one-third of its size while retaining most of its multimodal alignment ability, and it validates the compressed model on standard caption-evaluation benchmarks with a novel distillation objective. Second, it provides evidence that embedding-based rewards should be mixed with n-gram rewards to avoid language collapse, a finding that could inform the design of future captioning training objectives. The judge-side evaluation is carefully benchmarked against several human-judgment datasets and the compressed model is competitive with CLIPScore, which is a practically valuable result for resource-constrained deployment. The supervisor-side claim, however, currently rests on an internally inconsistent set of tables, as detailed below, so the training contribution cannot yet be considered established.

major comments (3)
  1. [§IV-D, Tables V and VI] The mixed-reward configuration reported in Table V is inconsistent with the ablation in Table VI. The text states that SMix uses equal weights (both 0.5), and Table VI defines a row α=0.5 that 'is equivalent to SMix in Table V.' However, the SMix(BU) row in Table V (BLEU-4 35.4, METEOR 27.0, ROUGE-L 56.5, CIDEr 118.7, SPICE 20.6, L-CLIPScore 77.7) does not match the α=0.5 row in Table VI (34.4, 26.7, 55.8, 115.3, 20.4, 77.4) or any other row. Moreover, the displayed equation SMix = α·SCD + (1−α)·SL-C assigns α=0 to pure L-CLIPScore and α=1 to pure CIDEr, which is the opposite of the text in the paragraph immediately below the equation ('When α is 0, it is equivalent to SCD. When α is 1.0, it is equivalent to SL-C'). Because the headline supervisor result—that mixing L-CLIPScore with CIDEr improves over pure CIDEr—is the basis for one of the paper's two advertised use cases, the contradiction must be resolved by reporting the exact configuration used for Table V and re-running the ablation with a consistent definition.
  2. [§IV-D, §IV-B (Karpathy split)] The recommended mixing weight α=0.7 is selected from the same Karpathy test split on which the final results in Tables V and VI are reported. The text states that 'All hyperparameters are systematically validated on the validation set through controlled experiments, with final selections achieving optimal test performance,' but the α ablation in Table VI and the reported SMix improvements are not accompanied by any validation-set results. Selecting α on the test set and then reporting test-set numbers for that chosen α makes the quantitative recommendation vulnerable to overfitting to the test split. The authors should either report α selection on a validation split or explicitly acknowledge and bound the effect of test-set peeking.
  3. [Tables II–IV] Several claims that L-CLIP 'outperforms' or 'achieves higher scores than' CLIP rest on differences of 0.3–1.4 points (e.g., Table II, Ref Flickr8kτc 53.8 vs 53.0; Table IV, 1-ref mean 90.11 vs 88.72; Table III, mean 81.4 vs 80.7). No error bars, confidence intervals, or significance tests are provided, and the numbers are reported to only one decimal place. Since the paper's central claim is 'comparable' performance, these small advantages are not essential, but the paper also repeatedly reads them as evidence of improvement. The authors should either add variance estimates (e.g., bootstraps over captions or multiple runs) or temper the superiority claims to 'comparable' where the differences are within noise.
minor comments (5)
  1. [Throughout] There are several typos and inconsistent terms: 'paris' should be 'pairs' (Section III-C), 'disill' should be 'distill' (Section II-C), 'MEREOR' should be 'METEOR' (Table II), and 'ConceptNet' in the introduction appears to refer to Conceptual Captions (the dataset actually used in Section IV-A).
  2. [Section III-A and Table I] The paper alternately states the original CLIP has '338M parameters' and reports in Table I that CLIP has 151,277,312 parameters (338M size in fp32). Please clarify whether '338M' refers to parameter count or memory footprint, and use the units consistently.
  3. [Section IV-C, text before Figure 3] The sentence 'it can be found that L-CLIPScore achieves higher image-text matching ability than the other embedding-based metrics like ViLBERTScore even it uses a larger 1023M model' is grammatically incomplete and the comparison is not shown in the cited tables; either add the missing comparison or rephrase.
  4. [Section IV-D, Figure 4] The caption of Figure 4 refers to 'the captions trained by CIDEr (red), L-CLIPScore (orange), and mixed reward (green)' but the figure itself does not visibly label colors for the reader; adding explicit labels or a legend would improve clarity.
  5. [Conclusion] The limitation paragraph correctly acknowledges the circularity of evaluating a captioning model with the same embedding metric used for training. This is a strength, but the point should also be applied to the judge-side claims: the FOIL and Pascal tests use the same L-CLIP embeddings that were distilled from CLIP, so gains over CLIP on those tests could partly reflect distillation artifacts rather than true alignment quality. A discussion of this would be helpful.

Circularity Check

1 steps flagged · score 2.0 of 10

No load-bearing circularity: judge experiments are externally anchored; one acknowledged self-evaluation step appears in the supervisor experiments, and the reported Table V/VI mismatch is a reproducibility concern rather than circularity.

  1. fitted input called prediction [Section V (Conclusion and Limitation); Table V rows SL-C(BU) and SRefL-C(BU) versus SCD(BU)]
    "Another limitation of all the embedding-based metrics is that such metrics can not be used to measure the captioning model trained by the same embeddings. For example, if CLIP is used to evaluate the quality, then it will naturally return a high score to the captioning model that is also trained by using CLIP-based vision and language embeddings."

    In Section IV-D, the sentence-level reward S in Eq. (7) is set to L-CLIPScore, Eq. (3) for SL-C and Eq. (4) for SRefL-C. The L-CLIPScore column in Table V for those rows is therefore the exact objective being optimized by reinforcement learning, not an independent measure of caption quality. The high values (82.3 and 82.2 vs. 76.3 for CIDEr-trained SCD) are largely forced by construction. The paper explicitly acknowledges this in the quoted limitation, and the central 'mixing helps' claim is not derived from this self-scored column, so the circularity is real but non-load-bearing.

full rationale

The judge-side derivation is self-contained and externally anchored. L-CLIP is produced by compressing CLIP with weight multiplexing and matrix decomposition and distilling with Eqs. (1)-(2), and its quality as a judge is measured against external human judgments (Flickr8K-Expert, Flickr8k-CF, Composite, Pascal-50S, FOIL), so the claim of comparable alignment with CLIP is not equivalent to the training objective. No load-bearing self-citation chain or imported uniqueness theorem is present; the compression techniques and CLIPScore rescaling are attributed to external prior work. The only direct circularity is the acknowledged self-evaluation in the supervisor experiments: models trained with L-CLIPScore as the RL reward are then scored with L-CLIPScore, so the elevated L-CLIPScore values in Table V for SL-C and SRefL-C are expected by construction rather than independent evidence. This does not drive the paper's central conclusion, which is instead supported by the observed language collapse under pure L-CLIPScore and by n-gram metric improvements from the mixed reward. Separately, the supervisor evidence contains a serious internal inconsistency that is not circularity: Table V's SMix(BU) row (BLEU-4 35.4, CIDEr 118.7) does not match the supposedly equivalent alpha=0.5 row in Table VI (BLEU-4 34.4, CIDEr 115.3), and the displayed equation SMix = alpha*SCD + (1-alpha)*SL-C assigns alpha=0 to pure SL-C while the text and table assign alpha=0 to pure SCD. That contradiction undermines the reproducibility of the headline 'mixing helps' result but does not make it circular. Overall score reflects one minor, acknowledged, non-load-bearing self-evaluation circularity.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

No new physical or conceptual entities are introduced; L-CLIP is a compressed neural network, not a new theoretical object. The load-bearing assumptions are about the reliability of the teacher signal, the sufficiency of the distillation data, the validity of the evaluation benchmarks, and the transferability of the reward-mixing finding. The free parameters are the rescaling weight, the mixing weight, and the low-rank dimension, none of which are derived from first principles.

free parameters (3)
  • rescaling weight w = 2.5
    Adopted from CLIPScore [37] in Eq. (3); not re-tuned for the student model. Rank-based correlations are invariant to this scale, but the SCST reward magnitude depends on it.
  • mix weight alpha = 0.7 recommended, 0.5 used in main SMix
    In SMix = alpha * S_CD + (1 - alpha) * S_L-C, Table VI ablates alpha on the Karpathy test split and the text recommends 7:3. Selecting alpha on the test split is a post-hoc choice.
  • low-rank embedding dimension = 256
    The 49408x768 word embedding matrix is decomposed into 49408x256 and 256x768 matrices (Section III-A); the rank is a design choice not derived from theory.
assumptions (5)
  • domain assumption Teacher CLIP similarities are reliable soft targets for distillation
    The SR loss in Eq. (2) uses S_T values from the teacher as reference targets. If the teacher's alignment signal is biased or noisy on the distillation batches, the student inherits that bias.
  • domain assumption ImageNet-1M and Conceptual Captions suffice for multimodal distillation
    Section IV-A lists these datasets for distillation instead of the original 400M image-text pairs used to train CLIP. The paper's success depends on this smaller corpus transferring enough alignment knowledge.
  • domain assumption Human-annotated benchmarks are valid proxies for caption quality
    All judge evaluations in Tables II-IV use Flickr8K-Expert, Flickr8k-CF, Composite, Pascal-50S, and FOIL. If these benchmarks do not reflect true caption quality, the evaluation claims do not transfer.
  • domain assumption Weight multiplexing and low-rank decomposition preserve cross-modal alignment capacity
    The architecture in Section III-A assumes the compressed student retains enough capacity to imitate the teacher. The ablations test this assumption empirically, but it is not guaranteed by the compression techniques themselves.
  • domain assumption A single Transformer captioning model on MSCOCO is a sufficient testbed for reward design
    Supervisor experiments in Section IV-D use one architecture and one dataset. The conclusion that mixing with n-gram metrics is needed may not generalize to other captioning models or datasets.

how reviews work

0 comments
Cite this review

Pith. "Pith review of L-CLIPScore: a Lightweight Embedding-based Captioning Metric for Evaluating and Training." pith.science (2026). https://pith.science/paper/VH3AKZC5

@misc{pith2026250708710,
  author       = {Pith},
  title        = {Pith review of: L-CLIPScore: a Lightweight Embedding-based Captioning Metric for Evaluating and Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VH3AKZC5}},
  note         = {Machine review of arXiv:2507.08710}
}
read the original abstract

We propose a novel embedding-based captioning metric termed as L-CLIPScore that can be used for efficiently evaluating caption quality and training captioning model. L-CLIPScore is calculated from a lightweight CLIP (L-CLIP), which is a dual-encoder architecture compressed and distilled from CLIP. To compress, we apply two powerful techniques which are weight multiplexing and matrix decomposition for reducing the parameters of encoders and word embedding matrix, respectively. To distill, we design a novel multi-modal Similarity Regulator (SR) loss to transfer more vision-language alignment knowledge. Specifically, SR loss amplifies the multi-modal embedding similarity if the given image-text pair is matched and diminishes the similarity if the pair is non-matched. By compressing and distilling by this novel SR loss, our L-CLIP achieves comparable multi-modal alignment ability to the original CLIP while it requires fewer computation resources and running time. We carry out exhaustive experiments to validate the efficiency and effectiveness of L-CLIPScore when using it as the judge to evaluate caption quality. We also discover that when using L-CLIPScore as the supervisor to train the captioning model, it should be mixed up by an n-gram-based metric and meanwhile analyze why using L-CLIPScore only will cause fail training.

Figures

Figures reproduced from arXiv: 2507.08710 by the authors.

Figure 1
Figure 1. (a) An image-caption pair. (b) Although the top caption is more [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. (a) The architecture of L-CLIP. The vision/language encoders of L [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Two visualized examples compare CIDEr and our L-CLIPScore. In each sub-figure, the green caption denotes the less matched one, the red one [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: We show 4 images with the captions. In the middle block, from top to bottom are the captions trained by CIDEr (red), L-CLIPScore (orange), and [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

68 extracted references · 42 canonical work pages

  1. [1]

    Learning cnn-lstm architectures for image caption generation,

    M. Soh, “Learning cnn-lstm architectures for image caption generation,” Dept. Comput. Sci., Stanford Univ., Stanford, CA, USA, Tech. Rep, vol. 1, 2016

    work page 2016

  2. [2]

    Image captioning through image transformer,

    S. He, W. Liao, H. R. Tavakoli, M. Yang, B. Rosenhahn, and N. Pugeault, “Image captioning through image transformer,” in Proceedings of the Asian conference on computer vision , 2020

    work page 2020

  3. [3]

    Meshed-memory transformer for image captioning,

    M. Cornia, M. Stefanini, L. Baraldi, and R. Cucchiara, “Meshed-memory transformer for image captioning,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2020, pp. 10 578–10 587

    work page 2020

  4. [4]

    Bottom-up and top-down attention for image captioning and visual question answering,

    P. Anderson, X. He, C. Buehler, D. Teney, M. Johnson, S. Gould, and L. Zhang, “Bottom-up and top-down attention for image captioning and visual question answering,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2018, pp. 6077–6086

    work page 2018

  5. [5]

    Self- critical sequence training for image captioning,

    S. J. Rennie, E. Marcheret, Y . Mroueh, J. Ross, and V . Goel, “Self- critical sequence training for image captioning,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2017, pp. 7008–7024

    2017

  6. [6]

    Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image cap- tioning,

    P. Sharma, N. Ding, S. Goodman, and R. Soricut, “Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image cap- tioning,” in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2018, pp. 2556– 2565

    work page 2018

  7. [7]

    Microsoft coco: Common objects in context,

    T.-Y . Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll ´ar, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13 . Springer, 2014, pp. 740–755

    2014

  8. [8]

    Bleu: a method for automatic evaluation of machine translation,

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in Proceedings of the 40th annual meeting of the Association for Computational Linguistics , 2002, pp. 311–318

    2002

Show all 68 references
  1. [9]

    Cider: Consensus- based image description evaluation,

    R. Vedantam, C. Lawrence Zitnick, and D. Parikh, “Cider: Consensus- based image description evaluation,” in Proceedings of the IEEE confer- ence on computer vision and pattern recognition , 2015, pp. 4566–4575

    2015

  2. [10]

    Bertscore: Evaluating text generation with bert,

    T. Zhang, V . Kishore, F. Wu, K. Q. Weinberger, and Y . Artzi, “Bertscore: Evaluating text generation with bert,” in International Conference on Learning Representations

  3. [11]

    What you see is what you read? improv- ing text-image alignment evaluation,

    M. Yarom, Y . Bitton, S. Changpinyo, R. Aharoni, J. Herzig, O. Lang, E. Ofek, and I. Szpektor, “What you see is what you read? improv- ing text-image alignment evaluation,” Advances in Neural Information Processing Systems, vol. 36, pp. 1601–1619, 2023

    2023

  4. [12]

    Fast, accurate, and lightweight memory-enhanced embedding learning framework for image-text retrieval,

    Z. Li, L. Zhang, K. Zhang, Y . Zhang, and Z. Mao, “Fast, accurate, and lightweight memory-enhanced embedding learning framework for image-text retrieval,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 34, no. 7, pp. 6542–6558, 2024

    2024

  5. [13]

    Vilbertscore: Evaluating image caption using vision-and-language bert,

    H. Lee, S. Yoon, F. Dernoncourt, D. S. Kim, T. Bui, and K. Jung, “Vilbertscore: Evaluating image caption using vision-and-language bert,” in Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems, 2020, pp. 34–39

    2020

  6. [14]

    Image-text alignment and retrieval using light-weight transformer,

    W. Li and X. Fan, “Image-text alignment and retrieval using light-weight transformer,” in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2022, pp. 4758–4762

    2022

  7. [15]

    Umic: An unreferenced metric for image captioning via contrastive learning,

    H. Lee, S. Yoon, F. Dernoncourt, T. Bui, and K. Jung, “Umic: An unreferenced metric for image captioning via contrastive learning,”arXiv preprint arXiv:2106.14019, 2021

    2021 arXiv

  8. [16]

    Less is more: Clipbert for video-and-language learning via sparse sampling,

    J. Lei, L. Li, L. Zhou, Z. Gan, T. L. Berg, M. Bansal, and J. Liu, “Less is more: Clipbert for video-and-language learning via sparse sampling,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 7331–7341

    2021

  9. [17]

    Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks,

    J. Lu, D. Batra, D. Parikh, and S. Lee, “Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks,” 2019

    2019

  10. [18]

    Quality estimation for image captions based on large-scale human evaluations,

    T. Levinboim, A. V . Thapliyal, P. Sharma, and R. Soricut, “Quality estimation for image captions based on large-scale human evaluations,” in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techn...

    2021

  11. [19]

    Revealing the dark secrets of bert,

    O. Kovaleva, A. Romanov, A. Rogers, and A. Rumshisky, “Revealing the dark secrets of bert,” in Proceedings of the 2019 Conference on Empir- ical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , 20...

    2019

  12. [20]

    Minivit: Compressing vision transformers with weight multiplexing,

    J. Zhang, H. Peng, K. Wu, M. Liu, B. Xiao, J. Fu, and L. Yuan, “Minivit: Compressing vision transformers with weight multiplexing,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 12 145–12 154

    2022

  13. [21]

    Albert: A lite bert for self-supervised learning of language representa- tions,

    Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut, “Albert: A lite bert for self-supervised learning of language representa- tions,” arXiv preprint arXiv:1909.11942 , 2019

    1909 arXiv

  14. [22]

    Enabling multimodal generation on clip via vision-language knowledge distilla- tion,

    W. Dai, L. Hou, L. Shang, X. Jiang, Q. Liu, and P. Fung, “Enabling multimodal generation on clip via vision-language knowledge distilla- tion,” in Findings of the Association for Computational Linguistics: ACL 2022, 2022, pp. 2383–2395

    2022

  15. [23]

    Clip-td: Clip targeted distillation for vision-language tasks,

    Z. Wang, N. Codella, Y .-C. Chen, L. Zhou, J. Yang, X. Dai, B. Xiao, H. You, S.-F. Chang, and L. Yuan, “Clip-td: Clip targeted distillation for vision-language tasks,” arXiv preprint arXiv:2201.05729 , 2022

    2022 arXiv

  16. [24]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning . PMLR, 2021, pp. 8748–8763

    2021

  17. [25]

    Sca-cnn: Spatial and channel-wise attention in convolutional networks for image captioning,

    C. Long, H. Zhang, J. Xiao, L. Nie, and T. S. Chua, “Sca-cnn: Spatial and channel-wise attention in convolutional networks for image captioning,” IEEE, 2016

    2016

  18. [26]

    Knowing when to look: Adaptive attention via a visual sentinel for image captioning,

    J. Lu, C. Xiong, D. Parikh, and R. Socher, “Knowing when to look: Adaptive attention via a visual sentinel for image captioning,” IEEE, 2017

    2017

  19. [27]

    Know more say less: Image captioning based on scene graphs,

    X. Li and S. Jiang, “Know more say less: Image captioning based on scene graphs,” IEEE Transactions on Multimedia , vol. 21, no. 8, pp. 2117–2130, 2019

    2019

  20. [28]

    High-quality image cap- tioning with fine-grained and semantic-guided visual attention,

    Z. Zhang, Q. Wu, Y . Wang, and F. Chen, “High-quality image cap- tioning with fine-grained and semantic-guided visual attention,” IEEE Transactions on Multimedia , vol. 21, no. 7, pp. 1681–1693, 2018

    2018

  21. [29]

    Multimodal transformer with multi- view visual representation for image captioning,

    J. Yu, J. Li, Z. Yu, and Q. Huang, “Multimodal transformer with multi- view visual representation for image captioning,” IEEE transactions on circuits and systems for video technology , vol. 30, no. 12, pp. 4467– 4480, 2019

    2019

  22. [30]

    Compact bidirectional transformer for image captioning,

    Y . Zhou, Z. Hu, D. Liu, H. Ben, and M. Wang, “Compact bidirectional transformer for image captioning,” 2022

    2022

  23. [31]

    Task-adaptive attention for image captioning,

    C. Yan, Y . Hao, L. Li, J. Yin, A. Liu, Z. Mao, Z. Chen, and X. Gao, “Task-adaptive attention for image captioning,” IEEE Transactions on Circuits and Systems for Video technology , vol. 32, no. 1, pp. 43–51, 2021

    2021

  24. [32]

    Textual context-aware dense captioning with diverse words,

    Z. Shao, J. Han, K. Debattista, and Y . Pang, “Textual context-aware dense captioning with diverse words,”IEEE Transactions on Multimedia, 2023

    2023

  25. [33]

    Meteor: An automatic metric for mt evalua- tion with improved correlation with human judgments,

    S. Banerjee and A. Lavie, “Meteor: An automatic metric for mt evalua- tion with improved correlation with human judgments,” in Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization , 2005, pp. 65–72

    2005

  26. [34]

    Rouge: A package for automatic evaluation of summaries,

    C.-Y . Lin, “Rouge: A package for automatic evaluation of summaries,” in Text summarization branches out , 2004, pp. 74–81

    2004

  27. [35]

    Spice: Semantic propositional image caption evaluation,

    P. Anderson, B. Fernando, M. Johnson, and S. Gould, “Spice: Semantic propositional image caption evaluation,” in Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14 . Springer, 2016, pp. 382–398

    2016

  28. [36]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

  29. [37]

    Clipscore: A reference-free evaluation metric for image captioning,

    J. Hessel, A. Holtzman, M. Forbes, R. Le Bras, and Y . Choi, “Clipscore: A reference-free evaluation metric for image captioning,” in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021, pp. 7514–7528

    2021

  30. [38]

    Tiger: Text-to-image grounding for image caption evalua- tion,

    M. Jiang, Q. Huang, L. Zhang, X. Wang, P. Zhang, Z. Gan, J. Diesner, and J. Gao, “Tiger: Text-to-image grounding for image caption evalua- tion,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conferenc...

    2019

  31. [39]

    Em- score: Evaluating video captioning via coarse-grained and fine-grained embedding matching,

    Y . Shi, X. Yang, H. Xu, C. Yuan, B. Li, W. Hu, and Z.-J. Zha, “Em- score: Evaluating video captioning via coarse-grained and fine-grained embedding matching,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 17 929–17 938

    2022

  32. [40]

    Gpt-3: Its nature, scope, limits, and consequences,

    L. Floridi and M. Chiriatti, “Gpt-3: Its nature, scope, limits, and consequences,” Minds and Machines , vol. 30, pp. 681–694, 2020

    2020

  33. [41]

    Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehen- sion,

    M. Lewis, Y . Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V . Stoyanov, and L. Zettlemoyer, “Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehen- sion,” in Proceedings of the 58th Annual Meeting of the Associa...

    2020

  34. [42]

    Swin transformer: Hierarchical vision transformer using shifted windows,

    Z. Liu, Y . Lin, Y . Cao, H. Hu, Y . Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 10 012–10 022

    2021

  35. [44]

    Image as a foreign language: Beit pretraining for all vision and vision-language tasks,

    W. Wang, H. Bao, L. Dong, J. Bjorck, Z. Peng, Q. Liu, K. Aggarwal, O. K. Mohammed, S. Singhal, S. Som et al. , “Image as a foreign language: Beit pretraining for all vision and vision-language tasks,” arXiv preprint arXiv:2208.10442 , 2022

    2022 arXiv

  36. [45]

    Vl-bert: Pre-training of generic visual-linguistic representations,

    W. Su, X. Zhu, Y . Cao, B. Li, L. Lu, F. Wei, and J. Dai, “Vl-bert: Pre-training of generic visual-linguistic representations,” arXiv preprint arXiv:1908.08530, 2019

    1908 arXiv

  37. [46]

    Slip: Self-supervision meets language-image pre-training,

    N. Mu, A. Kirillov, D. Wagner, and S. Xie, “Slip: Self-supervision meets language-image pre-training,” in Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXVI. Springer, 2022, pp. 529–544

    2022

  38. [47]

    Pruning filters for efficient convnets,

    H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf, “Pruning filters for efficient convnets,” arXiv preprint arXiv:1608.08710 , 2016

    2016 arXiv

  39. [48]

    Vision transformer pruning,

    M. Zhu, Y . Tang, and K. Han, “Vision transformer pruning,” arXiv preprint arXiv:2104.08500, 2021

    2021 arXiv

  40. [49]

    Post-training quantization for vision transformer,

    Z. Liu, Y . Wang, K. Han, W. Zhang, S. Ma, and W. Gao, “Post-training quantization for vision transformer,” Advances in Neural Information Processing Systems, vol. 34, pp. 28 092–28 103, 2021

    2021

  41. [50]

    Tinyvit: Fast pretraining distillation for small vision transformers,

    K. Wu, J. Zhang, H. Peng, M. Liu, B. Xiao, J. Fu, and L. Yuan, “Tinyvit: Fast pretraining distillation for small vision transformers,” in Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXI , 2022, pp. 68–85

    2022

  42. [51]

    Tinybert: Distilling bert for natural language understanding,

    X. Jiao, Y . Yin, L. Shang, X. Jiang, X. Chen, L. Li, F. Wang, and Q. Liu, “Tinybert: Distilling bert for natural language understanding,” in Findings of the Association for Computational Linguistics: EMNLP 2020, 2020, pp. 4163–4174

    2020

  43. [52]

    Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter,

    V . Sanh, L. Debut, J. Chaumond, and T. Wolf, “Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter,” arXiv preprint arXiv:1910.01108, 2019

    1910 arXiv

  44. [53]

    Training data-efficient image transformers & distillation through attention,

    H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. J ´egou, “Training data-efficient image transformers & distillation through attention,” in International conference on machine learning . PMLR, 2021, pp. 10 347–10 357

    2021

  45. [54]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems , vol. 30, 2017

    2017

  46. [55]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929 , 2020

    2010 arXiv

  47. [56]

    Bag of tricks for image classification with convolutional neural networks,

    T. He, Z. Zhang, H. Zhang, Z. Zhang, J. Xie, and M. Li, “Bag of tricks for image classification with convolutional neural networks,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 558–567

    2019

  48. [57]

    Making convolutional networks shift-invariant again,

    R. Zhang, “Making convolutional networks shift-invariant again,” in International conference on machine learning. PMLR, 2019, pp. 7324– 7334

    2019

  49. [58]

    Discriminability objective for training descriptive captions,

    R. Luo, B. Price, S. Cohen, and G. Shakhnarovich, “Discriminability objective for training descriptive captions,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 6964– 6974

    2018

  50. [59]

    ImageNet Large Scale Visual Recognition Challenge,

    O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, “ImageNet Large Scale Visual Recognition Challenge,” International Journal of Computer Vision (IJCV) , vol. 115, no. 3, pp. 211–252, 2015

    2015

  51. [60]

    Framing image description as a ranking task: Data, models and evaluation metrics,

    M. Hodosh, P. Young, and J. Hockenmaier, “Framing image description as a ranking task: Data, models and evaluation metrics,” Journal of Artificial Intelligence Research , vol. 47, pp. 853–899, 2013

    2013

  52. [61]

    From im- ages to sentences through scene description graphs using commonsense reasoning and knowledge,

    S. Aditya, Y . Yang, C. Baral, C. Fermuller, and Y . Aloimonos, “From im- ages to sentences through scene description graphs using commonsense reasoning and knowledge,” arXiv preprint arXiv:1511.03292 , 2015

    2015 arXiv

  53. [62]

    Consensus-based image description evaluation,

    R. Vedantam, C. L. Zitnick, and D. Parikh, “Consensus-based image description evaluation,” in Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 4566–4575

  54. [63]

    Foil it! find one mismatch between image and language caption,

    R. Shekhar, S. Pezzelle, Y . Klimovich, A. Herbelot, M. Nabi, E. Sangineto, and R. Bernardi, “Foil it! find one mismatch between image and language caption,” in Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 201...

    2017

  55. [64]

    Deep visual-semantic alignments for gen- erating image descriptions,

    A. Karpathy and L. Fei-Fei, “Deep visual-semantic alignments for gen- erating image descriptions,” in Computer Vision & Pattern Recognition, 2015

    2015

  56. [65]

    Vinvl: Revisiting visual representations in vision-language models,

    J. Gao, J. Yang, L. Zhang, L. Wang, P. Zhang, X. Hu, X. Li, and Y . Choi, “Vinvl: Revisiting visual representations in vision-language models,” 2021

    2021

  57. [66]

    Adam: A method for stochastic optimization,

    D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in International Conference on Learning Representations , 2014

    2014

  58. [67]

    Improving image captioning evaluation by considering inter references variance,

    Y . Yi, H. Deng, and J. Hu, “Improving image captioning evaluation by considering inter references variance,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , 2020, pp. 985–994

    2020

  59. [68]

    Concrete problems in ai safety,

    D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Man ´e, “Concrete problems in ai safety,” arXiv preprint arXiv:1606.06565, 2016

    2016 arXiv

  60. [69]

    Spontaneous reward hacking in iterative self-refinement,

    J. Pan, H. He, S. R. Bowman, and S. Feng, “Spontaneous reward hacking in iterative self-refinement,” arXiv preprint arXiv:2407.04549 , 2024

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.