{"id":"e9258002-6337-42da-9d25-904e04b28e0f","arxiv_id":"2507.18915","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A scalable pipeline generates 1.7 million increasingly abstract MSCOCO captions, and fine-tuning CLIP on them improves zero-shot retrieval in poetry and metaphor tasks.","lead":"This paper builds a pipeline that mines word associations for objects in an image and then writes creative captions at five levels of abstraction, producing 1.7 million synthetic captions for MSCOCO. The authors fine-tune a vision-language model on those captions and report gains on poetry-to-image and metaphor-to-visualization retrieval tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No ablation isolates contextualized associations: retrieval gains at d=4/5 could come from synthetic captions or prefix-tuning alone.","rationale":"I read the paper as an engineering contribution: a scalable pipeline for producing abstracted, associative captions, plus evidence that training on them improves creative retrieval. The data-generation pipeline is clearly described, and the human abstraction ranking (Table 3) is persuasive that the degree labels are perceptible. The retrieval improvements at d=4/5 on Task 1 and on the average-rank metric in Task 2 are plausible. However, the central causal claim that contextualized visual associations are the active ingredient is not tested. The improvements might be a generic benefit of training on synthetic captions with abstract vocabulary, or of the per-degree prefix itself. The control is only against an untuned OpenCLIP; no ablation removes associations while holding everything else fixed. This is a correctness risk, not a novelty or consensus issue. The reader's conditional verdict is appropriate and my concern reinforces the same condition, so the verdict remains unchanged. I partially agree with the reader: the reader emphasizes visual grounding, whereas I emphasize the missing attribution control; both are load-bearing and would be settled by the proposed matched-control ablation.","tokens_in":888,"tokens_out":960,"duration_ms":42249,"concrete_test":"Run a matched-control fine-tuning study: fix the OpenCLIP prefix architecture, training budget, and prompt template, and generate three caption sets on the same MSCOCO images — (C1) the paper's mined contextualized associations, (C2) the original salient words only (degree-0 literal captions), and (C3) associations sampled uniformly from other images at the same degree. Evaluate Tasks 1-3 exactly as in Tables 4-6. If C2 or C3 closes the d=4/5 gap to within a standard error, the contextualized-association attribution fails; if C1 alone beats both, the central claim survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that contextualized associations, not synthetic captions or prefix-tuning, drive the retrieval gains. The evidence for this causal attribution is missing. In Section 5.4 (Tables 4-6), the only comparison is OpenCLIP versus OpenCLIP-FT(d); the fine-tuned system differs from the baseline by three confounded factors: a learned per-degree prefix, 1.7M synthetic captions, and LLM-mined association words. There is no control that trains the same prefix on the same images with (a) original salient words in place of associations, (b) associations randomly permuted across images, or (c) generic non-associative synthetic captions. Without these, the Task 1 and Task 2 gains at d=4/5 cannot be attributed to contextualization; they may come from distribution shift away from literal alt-text or from prefix adaptation itself. The human grounding evidence is too weak to close this gap: only 100 captions were grounding-annotated, Fleiss kappa is 0.303, and the grounding threshold was chosen post hoc after Gemini-based error analysis (Section 5.2). The Limitations section concedes that hallucination and bias in Molmo and GPT-4o-mini can be reflected and reinforced by both the method and the dataset. Thus the load-bearing assumption that the training captions are visually grounded and association-specific is not independently secured.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a scalable pipeline that, for any image with a short literal caption, mines contextualized visual associations for salient elements at five degrees of abstraction (using Molmo for detailed captions and GPT-4o-mini for association mining) and then uses those associations to generate creative captions that remain visually grounded. The authors construct a dataset of about 1.7M creative captions for MSCOCO, validate the caption quality through human annotation of grounding and abstraction ranking, and fine-tune an OpenCLIP ViT-B/32 encoder with degree-specific learnable prefixes on the synthetic captions. They evaluate the fine-tuned models on three zero-shot creative retrieval tasks: poetry-to-image retrieval on MultiM-Poem, visual-metaphor-to-linguistic-metaphor retrieval on HAIVMet, and linguistic-metaphor-to-visualization matching on HAIVMet. The reported results show improved retrieval at higher abstraction degrees (d=4 and d=5) on Task 1, improved average rank on Task 2, and mixed but mostly improved preference on Task 3.","tokens_in":11859,"tokens_out":3086,"duration_ms":33249,"significance":"If the central claim is fully established, the work would make a useful contribution: it provides a scalable method for generating creative, increasingly abstract captions and demonstrates that training on such captions can improve zero-shot performance in creative vision-language retrieval, a domain where standard CLIP models are known to struggle. The paper has notable strengths: the dataset, generation code, and models are released; the evaluation uses external benchmarks (MultiM-Poem and HAIVMet) that were not used to fit model parameters; fine-tuning is parameter-efficient (only prefix embeddings are updated); and the human evaluation directly assesses both grounding and perceived abstraction. The paper also explicitly acknowledges limitations about hallucination and bias in the underlying LLMs/VLMs. However, the causal attribution of the retrieval improvements to contextualized associations is not yet secured, because the fine-tuned model differs from the baseline along several confounded dimensions and the human grounding evidence is based on a small, post-hoc thresholded annotation study.","major_comments":[{"comment":"The central claim of the paper is that contextualized associations, rather than synthetic captions or prefix-tuning alone, drive the observed retrieval improvements. The current comparison between OpenCLIP and OpenCLIP-FT(d) changes at least three factors simultaneously: a learned per-degree prefix, the use of 1.7M synthetic captions as training data, and the replacement of literal salient words with LLM-mined association words. Without control experiments that train the same prefix on the same images with (a) the original salient words instead of associations, (b) associations randomly permuted across images, or (c) generic non-associative synthetic captions, the gains at d=4 and d=5 cannot be attributed to contextualization. These controls are necessary to support the causal language in the abstract and in §5.4.","section":"§5.4, Tables 4-6"},{"comment":"The grounding threshold (ratings 3 and 4 treated as 'visually grounded') was chosen after inspecting score-3 captions and using Gemini 2.0 Flash to argue that they contain no actual errors. This post-hoc thresholding can inflate the reported grounding percentages, and the supporting annotation is small (100 captions with a Fleiss kappa of 0.303). The claim that the captions 'remain visually grounded' is important because the training signal inherits caption quality. The authors should report the full distribution of raw ratings, the number of captions at each rating, and ideally a pre-registered or independently motivated threshold, or alternatively verify that the main retrieval results are robust to restricting training to captions with higher raw grounding scores.","section":"§5.2, Table 2"},{"comment":"In Task 3, the baseline OpenCLIP performance is 0.43, below random chance, and only the improvements at d=1 and d=3 are marked as statistically significant. The sentence 'All abstraction settings improve over the baseline' is misleading because d=2, d=4, and d=5 show non-significant improvements. The claim of 'meaningful improvements' across creative tasks should be tempered, or additional evidence (e.g., larger evaluation sets, confidence intervals, or multiple seeds) should be provided to establish that the Task 3 finding is reliable.","section":"§5.4, Table 6"}],"minor_comments":[{"comment":"The text first says annotators rank six captions for each of 100 images, but then states that three annotations were collected for 20% of these tasks (20 images). Please clarify the exact number of annotation tasks, images, and captions used for the abstraction ranking evaluation.","section":"§4.2"},{"comment":"The appendix states that the model was fine-tuned for a single epoch but also mentions early stopping with patience 3 on the validation loss. These statements are inconsistent; please clarify whether early stopping was used and what the final epoch count actually was.","section":"§A.1"},{"comment":"The appendix uses 'wilcoxin tests' and 'hyperparemters' in a few places; these should be corrected to 'Wilcoxon' and 'hyperparameters'.","section":"§A.3.1"},{"comment":"Minor typos include 'MSCOCOvalidation' (missing space) and 'gpt4o-mini' (missing hyphen). These do not affect the substance of the paper.","section":"§4.1 and §7"},{"comment":"The uniqueness statistic in Table 1 would be more informative if compared to a baseline such as randomly sampled associations or associations mined from a generic (non-contextual) prompt, since even non-contextual associations may be mostly unique per image by chance.","section":"§5.1, Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable fit for the journal and the released resources are valuable. The main issue is not circularity (the evaluation benchmarks are external) but causal attribution: the fine-tuning comparison conflates prefix adaptation, synthetic caption exposure, and association content. The requested control experiments appear feasible within the manuscript's scope and would substantially strengthen the central claim. The human evaluation, while small, supports the literal claim that abstraction increases in perceived ranking; the grounding threshold concern is important but fixable with additional reporting. I do not see grounds for rejection, but the current evidence does not yet support the strongest form of the abstract's claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The first is that the abstraction ladder itself is the real contribution. The authors take a short COCO caption, generate a detailed caption with Molmo, pull out salient words, ask GPT-4o-mini for associations at five levels of abstraction, and then use Molmo to write captions using those associations. The human ranking in Table 3 is the strongest part of the paper: average rank rises monotonically from original (1.47) through d=5 (4.98), so the five-level ladder is doing real work. The corpus is large (1.7M captions) and they release it, which gives the community something reusable.\n\nThe second thing is that the downstream retrieval claims are less well-supported than the abstract suggests. Fine-tuning OpenCLIP with a per-degree prefix on these captions does improve zero-shot poetry-to-image and metaphor retrieval at d=4/5 in Task 1 and Task 2, and Task 3 shows gains. But the fine-tuned model differs from the baseline in three confounded ways: the prefix, the synthetic captions, and the association words. There is no control that trains on the same images with literal salient words instead of associations, or with associations shuffled across images, or with generic non-associative synthetic captions. So you cannot attribute the gains to contextualized associations; they could come from a distribution shift away from literal alt-text or from the prefix itself. This is the paper's load-bearing gap.\n\nThe grounding threshold is also slightly post hoc. They settled on the >=3 cutoff after analyzing the score-3 captions with Gemini, and the Fleiss kappa is only 0.303 on 100 captions. That said, the rationale is not crazy — the qualitative examples in Figure A.6 make a decent case that annotators were penalizing interpretive language rather than errors. It is a minor-to-moderate concern, not a fatal one.\n\nThe paper is honest in its Limitations section that LLM/VLM hallucination and bias can be baked in. Good to see.\n\nWho is this for? Researchers working on creative vision-language tasks, or on synthetic captions for representation learning. The dataset and the abstraction-taxonomy design are worth serious referee time. The empirical retrieval result is plausible but the attribution is unproven. I would send it to reviewers, but I would push hard for at least one control condition in the revision.","headline":"A genuinely useful dataset and a clean abstraction-ladder result, but the retrieval gains are confounded: no ablation isolates contextualized associations from synthetic captions or prefix-tuning.","tokens_in":12374,"tokens_out":2059,"would_cite":true,"duration_ms":21596,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning a visual encoder on mined abstract captions improves zero-shot retrieval in poetry and metaphor tasks.","keywords":["contextualized associations","creative captions","abstraction ladder","zero-shot retrieval","CLIP fine-tuning","poetry-to-image retrieval","visual metaphor","synthetic captions"],"falsifier":"Retrain the fine-tuned model on the same 1.671 million captions with each association word replaced by a randomly drawn association for the same word at the same abstraction degree; if the poetry-to-image and metaphor retrieval gains persist unchanged, then image-specific contextualization is not what drives the improvement, and the paper's central explanation would collapse.","tokens_in":11402,"feed_emoji":"🎨","tokens_out":7596,"duration_ms":70881,"temperature":0.7,"pith_summary":"The paper introduces a scalable method for mining contextualized associations—concepts that a salient object evokes in its specific scene context—and uses them to generate 1.671 million creative captions for images in MSCOCO at five degrees of abstraction. The central claim is that training a visual encoder on these captions sharpens its ability to match images with poems and metaphors, two tasks where standard alt-text-trained CLIP models underperform. The authors find statistically significant gains in zero-shot poetry-to-image retrieval, metaphor-to-visualization matching, and visual-metaphor-to-text retrieval after fine-tuning with per-abstraction prefix embeddings. Human evaluation supports that the captions stay visually grounded while exhibiting recognizably increasing abstraction. If correct, the method gives creative vision-language tasks a simple, scalable source of training signal without hand-curated datasets.","feed_headline":"Creative captions teach CLIP to match poems and metaphors","feed_subtitle":"Fine-tuning on 1.7 million mined abstract captions lifts zero-shot poetry and metaphor retrieval.","key_machinery":"The load-bearing object is the abstraction-indexed visual association, generated along a five-rung ladder: near synonyms, slightly broader categories, broader situational or emotional context, conceptual themes, and full metaphors. For each salient word, the pipeline filters MSCOCO short captions by part of speech and a concreteness lexicon, conditions a text-only LLM on a detailed VLM-written caption so the association reflects scene context rather than the word alone, and then asks a VLM to produce a sub-10-word creative caption that must include the association word. Fine-tuning adds five trainable prefix embeddings to a frozen CLIP visual encoder, one per degree, optimized with the contrastive image-text loss; at inference the prefix selects how abstractly the model should match an image to text.","core_discovery":"The paper claims that a three-step pipeline can turn any image with a short literal caption into a set of creative captions at controlled abstraction levels, and that these synthetic captions carry useful learning signal for creative vision-language understanding. First, a vision-language model writes a detailed caption; then a text-only LLM, conditioned on that detailed caption, mines associations for each salient word on a five-rung abstraction ladder from near-synonym to full metaphor; finally, a vision-language model rewrites the short caption into a grounded creative caption that includes the association. Fine-tuning a frozen CLIP visual encoder on this corpus, with a learnable prefix for each abstraction degree, improves zero-shot retrieval on two creative benchmarks compared with the unfine-tuned baseline, with the largest poetry retrieval gains at the highest abstraction degrees. The paper's own results also show that unfine-tuned CLIP prefers literal captions over abstract ones roughly as often as it prefers literal captions over hallucinated ones, which motivates why the mined associations are needed.","pith_inferences":["One implication the paper leaves implicit: the same pipeline could mine culturally specific associations by prompting the LLM in different languages, since the association step is text-only and the authors note the current corpus is English-centric and Western-centric.","A natural ablation would replace the detailed VLM caption with the short caption when mining associations; if scene context is doing the real work, association quality and downstream retrieval should drop.","The grounding evaluation relies on the same VLM for detailed captions and creative captions, so shared error patterns could in principle inflate the grounding scores; using different VLMs for the two steps would be a sturdier test.","The varying optimal abstraction degree across tasks suggests a future model that predicts the right abstraction level per query, rather than using a single setting."],"forward_implications":["Poetry-to-image retrieval improves at abstraction degrees 4 and 5, with average rank of the correct image falling from 70.46 to 67.60 and recall@1 rising from 0.1505 to 0.1624.","The fine-tuned encoder prefers the correct AI-generated visualization over its literal counterpart at every abstraction degree, with near-synonym captions improving the preference rate from 0.43 to 0.59.","Since the generation pipeline only needs an image and a short literal caption, it can be applied to any unlabeled corpus, extending the approach to art images, social media photos, or other domains.","The per-degree prefix design gives practitioners a dial: the same model can be used in a literal mode or an abstract mode depending on the creative task.","The synthetic captions expose a weakness of literalism in standard CLIP, which at the highest abstraction degree cannot distinguish between hallucinated captions and abstract ones better than random."],"supporting_citations":[{"why":"Supplies the MSCOCO images and short captions the corpus is built on.","marker":"Lin et al., 2014"},{"why":"Provides the vision-language model used to write detailed captions and generate creative captions.","marker":"Deitke et al., 2024"},{"why":"Concreteness lexicon used to filter salient visual elements from short captions.","marker":"Brysbaert et al., 2014"},{"why":"Defines the ladder of abstraction that structures the five association degrees.","marker":"Hayakawa, 1967"},{"why":"Prefix-tuning technique used to fine-tune CLIP with per-degree prefixes.","marker":"Li and Liang, 2021"},{"why":"MultiM-Poem benchmark, the poetry-to-image retrieval evaluation task.","marker":"Liu et al., 2018"},{"why":"HAIVMet benchmark providing the linguistic metaphor and visualization matching tasks.","marker":"Chakrabarty et al., 2023b"},{"why":"FOIL captions used to compare hallucination versus abstraction in the similarity analysis.","marker":"Shekhar et al., 2017"},{"why":"Documents task bias of contrastive vision-language models, motivating the creative-domain fine-tuning.","marker":"Menon et al., 2024"}],"fun_headline_variants":["Mined visual associations boost CLIP on poetry and metaphors","1.7M creative captions sharpen CLIP's creative vision","From literal to metaphor: mining associations for creative AI","Mining associations turns CLIP into a poetry reader","Fine-tuning CLIP on 1.7M abstract captions boosts creative retrieval"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a text-only LLM, given a machine-written detailed caption, mines associations that are genuinely visually grounded for the specific image; if the detailed caption contains an error or the association is generic, every downstream creative caption and the fine-tuning signal inherits that flaw.","fun_headline_variants_meta":{"raw":{"variants":["Mined visual associations boost CLIP on poetry and metaphors","1.7M creative captions sharpen CLIP's creative vision","From literal to metaphor: mining associations for creative AI","Mining associations turns CLIP into a poetry reader","Fine-tuning CLIP on 1.7M abstract captions boosts creative retrieval"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00086,"raw_usage":{"total_tokens":3711,"prompt_tokens":902,"completion_tokens":2809,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":518,"completion_tokens_details":{"reasoning_tokens":2723}},"tokens_in":518,"tokens_out":2809,"duration_ms":20420,"temperature":1.0,"reasoning_tokens":2723,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:05:43.225644+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the fine-tuned model on the same 1.671 million captions with each association word replaced by a randomly drawn association for the same word at the same abstraction degree; if the poetry-to-image and metaphor retrieval gains persist unchanged, then image-specific contextualization is not what drives the improvement, and the paper's central explanation would collapse.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the ladder of abstraction that structures the five association degrees."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MultiM-Poem benchmark, the poetry-to-image retrieval evaluation task."},{"cited_title":"FOIL it! Find One mismatch between Image and Language caption","cited_arxiv_id":"1705.01359","evidence_quote":"FOIL captions used to compare hallucination versus abstraction in the similarity analysis."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents task bias of contrastive vision-language models, motivating the creative-domain fine-tuning."}],"review_version":1}