Pith. sign in

REVIEW 27 cited by

ClipCap: CLIP Prefix for Image Captioning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2111.09734 v1 pith:WE3DIJH5 submitted 2021-11-18 cs.CV

Ron Mokady , Amir Hertz , Amit H. Bermano This is my paper
classification cs.CV
keywords modelclipimagecaptioncaptioningcaptionslanguageprefix
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Image captioning is a fundamental task in vision-language understanding, where the model predicts a textual informative caption to a given input image. In this paper, we present a simple approach to address this task. We use CLIP encoding as a prefix to the caption, by employing a simple mapping network, and then fine-tunes a language model to generate the image captions. The recently proposed CLIP model contains rich semantic features which were trained with textual context, making it best for vision-language perception. Our key idea is that together with a pre-trained language model (GPT2), we obtain a wide understanding of both visual and textual data. Hence, our approach only requires rather quick training to produce a competent captioning model. Without additional annotations or pre-training, it efficiently generates meaningful captions for large-scale and diverse datasets. Surprisingly, our method works well even when only the mapping network is trained, while both CLIP and the language model remain frozen, allowing a lighter architecture with less trainable parameters. Through quantitative evaluation, we demonstrate our model achieves comparable results to state-of-the-art methods on the challenging Conceptual Captions and nocaps datasets, while it is simpler, faster, and lighter. Our code is available in https://github.com/rmokady/CLIP_prefix_caption.

Discussion (0). Sign in to comment.

Forward citations

Cited by 27 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Sample-efficient Integration of New Modalities into Large Language Models

    cs.CL 2025-09 conditional novelty 7.0 of 10

    A hypernetwork trained on image, audio, and video adapts a shared projector to new, low-resource modalities from as few as 32 examples.

  2. Adjudicated Captioning: Multi-Agent Alignment Scoring and Consensus-Distilled Beam Arbitration for Strict Zero-Shot Image Captioning

    cs.CV 2026-07 conditional novelty 6.0 of 10

    A multi-checkpoint alignment pipeline lifts zero-shot COCO captioning CIDEr from 108.0 to 117.6 by adding a cross-attention verifier and self-supervised beam rerankers to an unchanged IFCap captioner.

  3. REPREC: Representation Driven Parameter-Efficient Recommendation System

    cs.IR 2026-07 conditional novelty 6.0 of 10

    A tiny MLP injector maps a frozen recommender's user embedding into soft tokens, letting a frozen LLM match LoRA-based recommenders at lower training cost.

  4. LatentLens: Revealing Highly Interpretable Visual Tokens in LLMs

    cs.CV 2026-01 conditional novelty 6.0 of 10

    Visual tokens in VLMs look highly interpretable at every layer when decoded by nearest-neighbor retrieval against a corpus of contextualized text embeddings: 72% interpretable on average vs 23–30% for LogitLens and Em...

  5. Unpacking Hateful Memes: Presupposed Context and False Claims

    cs.CL 2025-10 conditional novelty 6.0 of 10

    A hateful-meme detector that combines presupposed-context fusion, LLM-based social perception, and cross-modal reference graphs outperforms prior models on three benchmarks and transfers to fake news.

  6. Sparse and Dense Retrievers Learn Better Together: Joint Sparse-Dense Optimization for Text-Image Retrieval

    cs.CL 2025-08 conditional novelty 6.0 of 10

    A shared integrated teacher improves both sparse and dense text-image retrieval, letting a sparse retriever match or beat dense baselines on MSCOCO and several Flickr30k settings.

  7. RAVID: Retrieval-Augmented Visual Detection: A Knowledge-Driven Approach for AI-Generated Image Identification

    cs.CV 2025-08 conditional novelty 6.0 of 10

    RAVID detects AI-generated images by retrieving similar images from a database and feeding them to a vision-language model, reporting 93.85% average accuracy on UniversalFakeDetect.

  8. Mitigating Information Loss under High Pruning Rates for Efficient Large Vision Language Models

    cs.CV 2025-08 conditional novelty 6.0 of 10

    ACCM recovers information lost in high-rate visual token pruning by generating a question-guided caption from discarded tokens and selecting the best candidate, improving pruned LVLM accuracy with fewer FLOPs.

  9. VL-CLIP: Enhancing Multimodal Recommendations via Visual Grounding and LLM-Augmented CLIP Embeddings

    cs.IR 2025-07 conditional novelty 6.0 of 10

    A production system that combines object-detection-based image cropping and LLM-based text rewriting with CLIP fine-tuning reports large gains in multimodal retrieval and online recommendation metrics at Walmart scale.

  10. HapticCap: A Multimodal Dataset and Task for Understanding User Experience of Vibration Haptic Signals

    cs.CL 2025-07 conditional novelty 6.0 of 10

    HapticCap is the first large human-annotated vibration-caption dataset, and a contrastive retrieval model using T5 and AST achieves the best caption-matching performance among the tested baselines.

  11. InverTune: Removing Backdoors from Multimodal Contrastive Learning Models via Trigger Inversion and Activation Tuning

    cs.CR 2025-06 conditional novelty 6.0 of 10

    InverTune removes backdoors from CLIP models by identifying the target label via adversarial perturbations, inverting the trigger, and selectively tuning backdoor-sensitive neurons, reducing attack success rates to ne...

  12. Fast or Slow? Integrating Fast Intuition and Deliberate Thinking for Enhancing Visual Question Answering

    cs.CL 2025-06 conditional novelty 6.0 of 10

    FOCUS improves VQA accuracy by routing easy questions through fast zero-shot answering and hard questions through question-conditioned image segmentation before the final answer.

  13. Light as Deception: GPT-driven Natural Relighting Against Vision-Language Pre-training Models

    cs.CV 2025-05 conditional novelty 6.0 of 10

    LightD creates natural adversarial relighting images with GPT-selected lighting parameters and gradient optimization, outperforming prior non-suspicious attacks on vision-language models.

  14. Optimizing fMRI Data Acquisition for Decoding Natural Speech with Limited Participants

    q-bio.NC 2025-05 conditional novelty 6.0 of 10

    In a small cohort, fMRI decoders improve with more data per participant, and multi-subject training or shared stimuli add no benefit, so deep phenotyping is the recommended acquisition strategy.

  15. Imperceptible and Reversible Adversarial Examples against Vision-Language Models for Privacy Protection

    cs.CV 2026-07 conditional novelty 5.5 of 10

    CloakDiff generates high-fidelity reversible adversarial images that suppress VLM text-query privacy leakage via diffusion attention editing plus invertible steganography.

  16. MIND: Multi-rationale INtegrated Discriminative Reasoning Framework for Multi-modal Large Models

    cs.AI 2025-12 conditional novelty 5.0 of 10

    MIND improves multimodal reasoning by training on diverse correct and deliberately wrong rationales with two-stage correction and contrastive alignment, reporting SOTA on ScienceQA, A-OKVQA, and M3CoT.

  17. Compression Beyond Pixels: Semantic Compression with Multimodal Foundation Models

    cs.CV 2025-09 conditional novelty 5.0 of 10

    A learned product-quantization VAE with a shared codebook compresses CLIP image features to about 2-3 x 10^-3 bits per pixel with little loss in downstream semantic task accuracy.

  18. GLAD: Generalizable Tuning for Vision-Language Models

    cs.CV 2025-07 conditional novelty 5.0 of 10

    GLAD improves few-shot CLIP generalization by fusing original and sharpness-aware gradients during LoRA tuning and adding image-conditional text alignment.

  19. On the rankability of visual embeddings

    cs.CV 2025-07 conditional novelty 5.0 of 10

    Visual embeddings from CLIP and other vision encoders encode ordinal attributes along linear directions, recoverable from as few as two extreme reference images, without full supervision.

  20. MoMa: Modulating Mamba for Adapting Image Foundation Models to Video Recognition

    cs.CV 2025-06 conditional novelty 5.0 of 10

    MoMa adapts frozen CLIP to video by injecting Mamba-computed scale and bias into each layer, improving accuracy and efficiency on multiple action recognition benchmarks.

  21. Extracting Multimodal Learngene in CLIP: Unveiling the Multimodal Generalizable Knowledge

    cs.CV 2025-06 conditional novelty 5.0 of 10

    MM-LG extracts a compact multimodal and unimodal block set from CLIP via distillation and uses it to initialize smaller vision-language and vision models, outperforming previous Learngene methods and sometimes pre-tra...

  22. CLIP-HandID: Vision-Language Model for Hand-Based Person Identification

    cs.CV 2025-06 conditional novelty 5.0 of 10

    CLIP-HandID learns a pseudo-token per hand identity and feeds it through CLIP's frozen text encoder, improving rank-1 hand identification accuracy on the 11k Hands and HD datasets relative to one prior baseline.

  23. CoLMbo: Speaker Language Model for Descriptive Profiling

    cs.CL 2025-06 reject novelty 5.0 of 10

    CoLMbo pairs a fixed speaker encoder with a small language model to write descriptive profiles from voice, reporting high zero-shot accuracy for age, gender, ethnicity, and dialect.

  24. Diffusion-based Cumulative Adversarial Purification for Vision Language Models

    cs.CV 2025-06 conditional novelty 5.0 of 10

    DiffCAP purifies adversarial images for vision-language models by injecting cumulative Gaussian noise until embeddings stabilize, then denoising, and outperforms prior defenses on captioning, VQA, and classification b...

  25. Enhanced Multimodal Aspect-Based Sentiment Analysis by LLM-Generated Rationales

    cs.CL 2025-05 reject novelty 5.0 of 10

    Feeding LLM-generated text and image rationales into fine-tuned small models with a dual cross-attention module improves multimodal aspect-based sentiment analysis by 1-2 F1 points on Twitter2015 and Twitter2017.

  26. From Image Captioning to Visual Storytelling

    cs.CL 2025-07 unverdicted novelty 4.0 of 10

    Visual storytelling improves by treating it as image captioning followed by language-to-language story generation, with a new 'ideality' metric to gauge distance from an oracle.

  27. Beam-Guided Knowledge Replay for Knowledge-Rich Image Captioning using Vision-Language Model

    cs.CV 2025-05 conditional novelty 3.0 of 10

    Applying beam search, patch self-attention, and cosine scheduling to the K-Replay captioning framework improves knowledge-keyword recognition on KnowCap, though the full combined model is not reported.

Pith tools