REVIEW 2 major objections 5 minor 20 references
Captured by Captions: On Memorization and its Mitigation in CLIP Models
T0 review · 2 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read CLIPMem, a leave-one-out alignment gap, measures which image-text pairs a CLIP model memorizes; the paper reports that mis-captioned pairs are memorized most and that removing them improves downstream accuracy.
desk verdict CLIPMem is a useful and genuinely novel metric for CLIP memorization, but the per-point claims outrun the block-wise approximation and the abstract overstates the break with supervised learning. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machine is a leave-one-out alignment gap. For a given pair, the alignment score $A_{align}(f,I,T)$ is the cosine similarity between the image and text representations under model $f$, minus the average similarity to unrelated texts and unrelated images used to anchor the contrastive objective. CLIPMem subtracts the alignment score of a reference model $g$ trained without that pair, so simple pairs that align well in both models score near zero, while a pair that only the trained model aligns strongly is identified as memorized.
What would settle it
Train several independent pairs of CLIP models, placing the same image-text pair in the candidate set for one pair and in the shared set for another; if the pair does not reliably receive a high CLIPMem score when it is in the candidate set and a near-zero score when shared, the metric is not isolating per-pair memorization.
Extended reading notes
Core claim
The central claim is that memorization in a multimodal contrastive model can be measured directly on its output representations: for an image-text pair $(I,T)$, CLIPMem is the difference between the pair's alignment score under a model $f$ trained with the pair and under a model $g$ trained on identical data without it. A large positive gap means $f$ memorizes the pair. Across experiments, pairs with incorrect or imprecise captions receive the highest CLIPMem scores, followed by atypical samples, and the text encoder is responsible for more memorization than the image encoder. The paper also shows that text-side mitigations—multiple captions, generated captions, or Gaussian noise added to text embeddings during training—and removal of the most memorized samples reduce CLIPMem while improving linear probing accuracy on a downstream image classification benchmark, in contrast to supervised and self-supervised learning where reducing memorization typically hurts generalization.
Load-bearing premise
The metric assumes that comparing two models that differ by 5,000 training points isolates what happens to any one of those points, rather than being driven by interactions among all 5,000.
Editorial extensions
If this is right
- Mis-captioned pairs are the most memorized, so CLIPMem can act as a noisy-data filter for the uncurated web-scale datasets used to train CLIP.
- Because the text encoder carries more memorization than the image encoder, mitigation should target captions, such as training with multiple captions or noising text embeddings.
- In CLIP, unlike supervised and self-supervised learning, reducing memorization can simultaneously improve downstream accuracy.
- The neuron-level analysis places CLIP between supervised and self-supervised learning: early layers encode groups of data points, later layers memorize individual points.
- The same metric applied to a second contrastive vision-language model shows a similar memorization pattern, suggesting the findings generalize to other contrastive vision-language models.
Reading between the lines
- Editorial extension: High CLIPMem scores signal that a pair was present in training, so the metric could double as a membership-inference probe for contrastive vision-language models, with privacy-auditing consequences.
- Editorial extension: The dependence on a 5,000-point candidate split could be stress-tested with small-scale true leave-one-out training; if per-point scores do not track the split-based scores, the mitigation conclusions should be interpreted as batch-level rather than per-pair effects.
- Editorial extension: The finding that caption-side noise helps accuracy suggests that curating and verifying captions may matter more than image curation in CLIP training pipelines, a hypothesis that could be tested by comparing CLIPMem on deduplicated versus original alt-text datasets.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes CLIPMem, a memorization metric for CLIP models defined in Eq. (4) as the difference in image-text alignment between a model f trained on a given pair (I,T) and a reference model g trained without that pair, with the alignment score A_align defined in Eq. (3) as the pair's cosine similarity minus mean similarities to negative pairs from a held-out test set. Because training a model per data point is intractable, Section 4.1 replaces the leave-one-out comparison with two models trained on disjoint candidate sets (f on SS∪SC, g on SS∪SI, with 5000 points each), and CLIPMem is reported either per candidate point or as subset averages. Using OpenCLIP ViT-B/ViT-L models trained on COCO (with additional experiments on CC3M, YFCC100M, and BLIP), the paper claims that (i) candidate-only points receive significantly higher CLIPMem than shared or external points (Figure 2a), (ii) mis-captioned image-text pairs are the most memorized, as supported by a controlled shuffling experiment (Figure 2b), (iii) the text encoder contributes more to memorization than the image encoder (Sections 4.3–4.5), and (iv) text-side augmentations, text-embedding noising, and removal of the most memorized samples reduce memorization while improving ImageNet linear probing accuracy (Table 1, Figures 5–6), in contrast to the usual memorization-generalization trade-off in supervised and self-supervised learning.
Significance. If the central claims hold, the paper makes a genuinely useful contribution: it introduces the first joint multimodal memorization metric for CLIP, validates the metric with a well-designed shuffled-caption poisoning experiment and a clean SC-versus-SS separation, and produces a practically relevant and falsifiable finding—that the most memorized samples tend to be mis-captioned, and that removing them or augmenting the text side can improve downstream accuracy. The breadth of the evaluation is a clear strength: two architectures, two training data regimes (100-epoch COCO and single-epoch YFCC100M), a transfer check on BLIP, and direct measurement of utility through linear probing. These strengths should be credited. The significance is tempered, however, by two load-bearing issues: the per-point use of CLIPMem rests on an approximation (Section 4.1) whose per-point validity is never demonstrated, and the modality-dominance claim is made with SSLMem, a metric the paper itself shows is weak for CLIP, alongside an internal numerical inconsistency between Sections 4.4 and 4.5.
major comments (2)
- [§4.4–4.5] The claim that the text encoder contributes more to memorization than the image encoder is based on SSLMem (Figure 3a–b), a metric that the paper itself shows produces heavily overlapping SC and SS distributions for CLIP encoders; the reported mean difference (0.168 for the vision encoder vs 0.209 for the text encoder) is small relative to the spread of both distributions. Furthermore, the comparison with DINO in Section 4.5 is numerically inconsistent with Section 4.4: Section 4.4 and Figure 3 report the CLIP vision encoder's SSLMem as 0.168 and the text encoder's as 0.209, but Section 4.5 states that 'the CLIP vision encoder has a significantly lower SSLMem than the SSL encoder (0.209 vs. 0.279),' using the text encoder's value as if it were the vision encoder's. If 0.168 is the correct value, the sentence must be corrected; if 0.209 is intended, Section 4.4 and Figure 3 must be corrected. The qualitative conclusion survives either way (0.168 or 0.209 is below 0.279), but the factual inconsistency undermines confidence in the modality-level analysis. The modality conclusion would also be strengthened by direct evidence, such as ablating the text-side versus image-side negative terms in Eq. (3), or by a sensitivity analysis of CLIPMem to one encoder at a time.
- [§4.4, Table 1] Row 3 of Table 1 ('5 Images (generated), 1 Caption') is not comparable with the other rows: the text reports that 6000 mis-captioned samples were removed from this configuration to avoid performance collapse, and no such intervention is described for any other row. The number 6000 is a free parameter that is neither varied nor motivated. The claim that text augmentations reduce memorization more than image augmentations should therefore be argued from the comparable rows (row 2 vs row 4), where the differences are modest (0.428 vs 0.423 for CLIPMem; 63.97% vs 64.88% for accuracy), or the censoring of row 3 must be presented as part of a stated comparison protocol. As written, the table overstates the strength of the image-versus-text comparison.
minor comments (5)
- [§4.6] The dataset is spelled 'CCM3' once in the text ('CLIP models trained on COCO and on the CCM3 dataset') but 'CC3M' elsewhere, including the caption of Figure 6b; please unify the spelling.
- [Appendix A.2] The description of the SSLMem setup says 'an additional extra set (SI) from the test set,' but SI is already used for the independent training set and the extra set is called SE in Section 4.1; this notation collision should be fixed.
- [Appendix A.5, Table 6] The body text states that the infinite-data model has 'higher linear probing accuracy' and 'lower memorization scores,' while the table caption says both setups 'reach comparable downstream accuracy and memorization'; with 64.83% ± 1.04% versus 63.11% ± 0.91%, the difference is marginal and no significance test is reported, so the caption and the text should be aligned.
- [§4.6, Figure 5b and Table 8] The 'sweet spot' at noise standard deviation 0.15 is identified post hoc as the minimum of the observed memorization curve; the neighboring points (std 0.10: 64.95% ± 0.96%; std 0.15: 65.34% ± 0.84%) are within mutual error bars, and the memorization values across the plateau (0.417–0.421) differ by less than the apparent sampling variability. The claim of a precise sweet spot is stronger than the data support.
- [Appendix A.8] The GPT-3.5 caption-generation prompt in Appendix A.8 includes the CLIP embedding ('clip_features') of the corresponding image, and the resulting captions are then used for training and evaluation in Table 4 and Table 7; please clarify whether the embedding came from the same model instance used in the CLIPMem measurements, since conditioning on the model's own representation could partly explain the similarity in behavior between original and generated captions.
Circularity Check
No definitional circularity: CLIPMem is an explicit leave-one-out-style definition, validated against externally poisoned pairs, with mitigation effects measured by held-out linear probing.
full rationale
CLIPMem is introduced as an explicit definition (Eq. 4) rather than as a prediction derived from elsewhere: it compares the alignment of a pair in a model f trained with the pair against a model g trained without it, in the standard leave-one-out style. The main validations are external to the metric's own construction. The poisoning experiment imposes known ground-truth mis-captioning and observes higher CLIPMem for those pairs; the mitigation experiments are evaluated by ImageNet linear probing accuracy, not by CLIPMem itself; and the SC-versus-SS separation is a sanity check of the definition rather than a claimed discovery. The disjoint-candidate approximation in Section 4.1, where f is trained on SS∪SC and g on SS∪SI with 5,000 candidates each, is a genuine threat to per-point validity because the two models differ by 5,000 points rather than by a single point. However, this is an approximation and confounding concern, not a definitional identity: individual per-point scores are not forced by the construction, and the paper explicitly calls the procedure an approximation. The paper also relies on the authors' earlier SSLMem and UnitMem metrics; these are self-citations, but they are used as measurement tools from prior published work, not as a uniqueness theorem or as an ansatz that already contains this paper's conclusions. Those metrics are externally established and are applied here to new models, so they do not make the derivation circular. Overall, no load-bearing step reduces to its own input by construction.
Assumptions & free parameters
free parameters (3)
- Noise standard deviation for text embedding noising =
0.15
- Number of removed mis-captioned samples in Table 1, case 3 =
6000
- Subset sizes for the leave-one-out approximation =
|SS|=65000, |SC|=|SI|=|SE|=5000
assumptions (4)
- domain assumption The difference in alignment between a model trained with and without a sample measures that sample's memorization
- domain assumption Training two models with 5000 disjoint candidate points and taking per-point alignment differences approximates the leave-one-out memorization
- domain assumption The random test set dTtest provides stable negative baselines for both f and g
- domain assumption SSLMem and UnitMem from prior work (Wang et al., 2024a;b) are valid measures for the modality comparison and neuron-level analysis
Cite this review
Pith. "Pith review of Captured by Captions: On Memorization and its Mitigation in CLIP Models." pith.science (2026). https://pith.science/paper/57CDZ3AJ
@misc{pith2026250207830,
author = {Pith},
title = {Pith review of: Captured by Captions: On Memorization and its Mitigation in CLIP Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/57CDZ3AJ}},
note = {Machine review of arXiv:2502.07830}
}
read the original abstract
Multi-modal models, such as CLIP, have demonstrated strong performance in aligning visual and textual representations, excelling in tasks like image retrieval and zero-shot classification. Despite this success, the mechanisms by which these models utilize training data, particularly the role of memorization, remain unclear. In uni-modal models, both supervised and self-supervised, memorization has been shown to be essential for generalization. However, it is not well understood how these findings would apply to CLIP, which incorporates elements from both supervised learning via captions that provide a supervisory signal similar to labels, and from self-supervised learning via the contrastive objective. To bridge this gap in understanding, we propose a formal definition of memorization in CLIP (CLIPMem) and use it to quantify memorization in CLIP models. Our results indicate that CLIP's memorization behavior falls between the supervised and self-supervised paradigms, with "mis-captioned" samples exhibiting highest levels of memorization. Additionally, we find that the text encoder contributes more to memorization than the image encoder, suggesting that mitigation strategies should focus on the text domain. Building on these insights, we propose multiple strategies to reduce memorization while at the same time improving utility--something that had not been shown before for traditional learning paradigms where reducing memorization typically results in utility decrease.
Figures
Figures from the paper (16 more)
Reference graph
Works this paper leans on
-
[1]
A score of 0 indicates no memorization, +1 indicates the strongest memorization by f, and −1 indicates the strongest memorization by g. Normalization on CLIPMem. For improved interpretability, we normalize our CLIPMem scores to a range of [−1, 1]. A memorization score of 0 indicates no memorization, +1 indicates the strongest memorization on CLIP model f,...
work page 2025
-
[3]
Imagenet: A large-scale hierarchical image database
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255. Ieee,
2009
-
[4]
URL https://proceedings.neurips.cc/paper_files/paper/ 2023/file/6fa4d985e7c434002fb6289ab9b2d654-Paper-Conference.pdf. Vitaly Feldman. Does learning require memorization? a short tale about a long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pp. 954–959,
work page 2023
-
[5]
that causes increase in mis-captioned samples. Verifying the hypothesis on memorizing mis-captioned samples through supervised learning. We repeat the same experiment in the supervised learning setup to understand where the increase and then decrease in linear probing accuracy stems from. To test our hypothesis that it stems from "mis-captioned" samples, ...
work page 2020
-
[9]
On the memorization properties of contrastive learning
Ildus Sadrtdinov, Nadezhda Chirkova, and Ekaterina Lobacheva. On the memorization properties of contrastive learning. arXiv preprint arXiv:2107.10143,
-
[11]
Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li
Bart Thomee, David A. Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfcc100m: the new data in multimedia research. Commun. ACM, 59(2):64–73, January 2016a. ISSN 0001-0782. doi: 10.1145/2812802. URL https: //doi.org/10.1145/2812802. Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl N...
-
[12]
Localizing Memorization in SSL Vision Encoders
Wenhao Wang, Adam Dziedzic, Michael Backes, and Franziska Boenisch. Localizing memorization in ssl vision encoders. arXiv preprint arXiv:2409.19069, 2024a. Wenhao Wang, Muhammad Ahmad Kaleem, Adam Dziedzic, Michael Backes, Nicolas Papernot, and Franziska Boenisch. Memorization in self-supervised learning improves downstream generalization. In The Twelfth ...
-
[13]
The Déjà Vu memorization framework (Jayaraman et al.,
13 Published as a conference paper at ICLR 2025 A A PPENDIX A.1 E XTENDED BACKGROUND Déjà Vu Memorization in CLIP. The Déjà Vu memorization framework (Jayaraman et al.,
work page 2025
Show all 20 references
-
[14]
It uses the text embedding of a training image caption to retrieve relevant images from a public dataset of images
is the only existing other work that attempts to quantify memorization in vision-language models. It uses the text embedding of a training image caption to retrieve relevant images from a public dataset of images. It then measures the fraction of ground-truth objects from the ...
2020
-
[18]
Both models are trained using the same dataset and settings
A.4 T HE EFFECT OF MODEL SIZE In Table 5, we present how the model size affects the memorization level of CLIP models. Both models are trained using the same dataset and settings. We observe that with more parameters (larger model size), encoders have higher memorization capac...
2020
-
[20]
Here is an image with the caption: ’{image_caption}’
These results highly consistently indicate that in the early stages of training, neuronal memory occurs mainly in the lower layer of the clip model, while in the middle and later stages of training, neuronal memory is more concentrated in the later layer of the model. A.8 H UM...
2025
-
[21]
Airport Pkwy
In Figure 12b, we analyze the pairwise cosine similarity in the original COCO and the GPT3.5 generated captions. We find that the GPT3.5 generated captions are slightly more uniform than the original COCO captions, reflecting in a higher pairwise cosine similarity. A.9 E XAMPL...
2025
-
[2007]
Cumbres and Toltec Scenic Railroad, historical narrow gauge railroad between Chama, New Mexico, and Antonito, Colorado
I have not had time to develop individual captions for these, however happy to provide some comment if required. Cumbres and Toltec Scenic Railroad, historical narrow gauge railroad between Chama, New Mexico, and Antonito, Colorado. These photos are from the station and railya...
2009
-
[2009]
We use our evaluation setup withSC, SS, SI, and SE to approximate the memorization metric from Feldman (2020)
using supervised learning. We use our evaluation setup withSC, SS, SI, and SE to approximate the memorization metric from Feldman (2020). We use 5000 samples in SC, but before training, we flip the labels of 200 samples. We calculate memorization over all samples in SC and tes...
2020
-
[2014]
Hongbin Liu, Jinyuan Jia, Wenjie Qu, and Neil Zhenqiang Gong
URL http://arxiv.org/abs/ 1405.0312. Hongbin Liu, Jinyuan Jia, Wenjie Qu, and Neil Zhenqiang Gong. Encodermi: Membership inference against pre-trained encoders in contrastive learning. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, pp...
2021 arXiv
-
[2017]
Conditioned and composed image retrieval combining and partially fine-tuning clip-based features
10 Published as a conference paper at ICLR 2025 Alberto Baldrati, Marco Bertini, Tiberio Uricchio, and Alberto Del Bimbo. Conditioned and composed image retrieval combining and partially fine-tuning clip-based features. In Proceedings of the IEEE/CVF Conference on Computer Vis...
2025
-
[2018]
Machine learning models that remember too much
Congzheng Song, Thomas Ristenpart, and Vitaly Shmatikov. Machine learning models that remember too much. In Proceedings of the 2017 ACM SIGSAC Conference on computer and communications security, pp. 587–601,
2017
-
[2021]
URL https://doi.org/10.5281/ zenodo.5143773. 11 Published as a conference paper at ICLR 2025 Matthew Jagielski, Om Thakkar, Florian Tramer, Daphne Ippolito, Katherine Lee, Nicholas Carlini, Eric Wallace, Shuang Song, Abhradeep Guha Thakurta, Nicolas Papernot, et al. Measuring ...
2025
-
[2023]
Generalization vs
Antonis Antoniades, Xinyi Wang, Yanai Elazar, Alfonso Amayuelas, Alon Albalak, Kexun Zhang, and William Yang Wang. Generalization vs. memorization: Tracing language models’ capabilities back to pretraining data. In ICML 2024 Workshop on Foundation Models in the Wild,
2024
-
[2024]
Alex Krizhevsky, Geoffrey Hinton, et al
URL https://arxiv.org/abs/2402.02103. Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.