Pith. sign in

REVIEW 3 cited by

Text-Only Training for Image Captioning using Noise-Injected CLIP

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2211.00575 v1 pith:UJYIT7GO submitted 2022-11-01 cs.CV cs.AIcs.LG

David Nukrai , Ron Mokady , Amir Globerson This is my paper
classification cs.CVcs.AIcs.LG
keywords cliptextonlytrainingadditionalapproachcaptioningdata
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We consider the task of image-captioning using only the CLIP model and additional text data at training time, and no additional captioned images. Our approach relies on the fact that CLIP is trained to make visual and textual embeddings similar. Therefore, we only need to learn how to translate CLIP textual embeddings back into text, and we can learn how to do this by learning a decoder for the frozen CLIP text encoder using only text. We argue that this intuition is "almost correct" because of a gap between the embedding spaces, and propose to rectify this via noise injection during training. We demonstrate the effectiveness of our approach by showing SOTA zero-shot image captioning across four benchmarks, including style transfer. Code, data, and models are available on GitHub.

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Modality Gap-Driven Subspace Alignment Training Paradigm For Multimodal Large Language Models

    cs.CV 2026-02 unverdicted novelty 6.0 of 10

    ReAlign corrects the modality gap in unpaired data to let MLLMs learn visual distributions from text alone before instruction tuning, reducing dependence on expensive paired corpora.

  2. MoMa: Modulating Mamba for Adapting Image Foundation Models to Video Recognition

    cs.CV 2025-06 conditional novelty 5.0 of 10

    MoMa adapts frozen CLIP to video by injecting Mamba-computed scale and bias into each layer, improving accuracy and efficiency on multiple action recognition benchmarks.

  3. Text to Image for Multi-Label Image Recognition with Joint Prompt-Adapter Learning

    cs.CV 2025-06 conditional novelty 5.0 of 10

    Synthetic images from text captions, combined with a shared prompt-adapter, improve multi-label image recognition and reduce CLIP's modality gap.

Pith tools