{"id":"fc1b584d-1d7b-4b57-8c50-014c36764d8e","arxiv_id":"2411.16828","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"CLIPS improves CLIP-style pretraining by contrastive learning on one sampled synthetic sentence plus an asymmetric decoder that predicts the full synthetic caption, achieving state-of-the-art zero-shot retrieval on MSCOCO and Flickr30K.","lead":"CLIPS changes how CLIP is trained on machine-written image captions: it feeds only one random sentence from each long synthetic caption into the contrastive text encoder, and adds a captioner that reconstructs the full synthetic caption from the image and the original web caption. On zero-shot image-text retrieval it reports new state-of-the-art scores on MSCOCO and Flickr30K, and its vision encoder also improves a LLaVA multimodal chatbot.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Table 4 ablation adds the short-caption design before the decoder, so the decoder with full-length captions is never compared; the claimed benefit of single-sentence sampling may vanish when the generative loss is already present.","rationale":"The reader's weakest assumption concerns whether the inverse effect transfers from S/B/L at 2,000 epochs to L/H at 10,000 epochs. The concern raised here is more direct: the paper never establishes that the inverse effect matters at all once the generative loss is added, because the ablation order in Table 4 measures SC only in a contrastive-only model and then builds MP and GL on top of SC. The full model with MP+GL and full-length contrastive captions is absent. This is a load-bearing gap because the paper's narrative attributes part of the gain to the single-sentence design, and the SOTA numbers are obtained with that design active. The concrete test isolates the contribution of SC in the full framework, which is a stronger and more targeted check than rerunning Figure 3 at larger scale. I do not see an internal contradiction or obvious error in the reported results; the method is clearly described and the controlled comparisons in Table 1 support the overall effectiveness of the combined framework. However, the specific design principle of short synthetic captions is under-validated, so the reader's CONDITIONAL verdict remains appropriate, and the condition should be the missing ablation rather than only scale transfer.","tokens_in":101,"tokens_out":11988,"duration_ms":181734,"concrete_test":"Using the B/16 setup of Table 4, train (A) Baseline + MP + GL with full-length synthetic captions in the contrastive branch, and (B) the identical configuration but with the SC single-sentence sampling, keeping the 80/20 mixture and all other hyperparameters fixed. Run at least three seeds and compare MSCOCO I->T and T->I R@1 plus IN1K accuracy. If A and B are within roughly 0.5 points on all metrics, the single-sentence design is not load-bearing for the final method; if B is consistently better, the concern is refuted. A secondary run at L/16 would test whether model scale changes the answer.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central design choice is to feed only a randomly sampled sentence of each synthetic caption to the text encoder (Sec. 3.3), motivated by the inverse effect in Fig. 3. The evidence for this choice, however, is not tested in the configuration where it would matter most. Table 4 is a forward ablation: Baseline -> +SC -> +SC&MP -> +SC&MP&GL. The marginal contribution of SC (+1.4 R@1) is measured only against Plain CLIP, without the multi-positive loss or the decoder. The contributions of MP and GL are then measured on top of SC, so the full model is never compared to the same model with full-length synthetic captions in the contrastive branch. Because the asymmetric decoder (GL) already supervises the full synthetic caption, the text encoder may no longer be hurt by long contrastive captions; the inverse effect may be an artifact of contrastive-only training. If so, the +5.7 point gain of the full model over Baseline (Table 4) could be attributable to MP+GL alone, and the single-sentence sampling, which the paper presents as the first of two 'simple yet effective designs,' would not be load-bearing for the SOTA results. Footnote 2 reinforces the fragility: increasing the padding length from 32 to 80 changes performance by about 1 point, so the exact caption-sampling recipe is a sensitive engineering choice rather than a robust principle.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CLIPS, a CLIP training framework for learning from long synthetic captions. Two design changes are introduced: (1) instead of feeding the full synthetic caption to the text encoder, the authors randomly sample a single sentence from each synthetic caption and use it in a multi-positive contrastive loss alongside the original web-crawled caption; (2) an asymmetric autoregressive decoder predicts the full-length synthetic caption from the image and the web-crawled caption, imitating the recaptioning process. The authors report consistent gains over reproduced CLIPA and CoCa baselines on MSCOCO and Flickr30K zero-shot retrieval, new state-of-the-art numbers at ViT-L/14 and ViT-H/14, and improved LLaVA-1.5 performance when the visual encoder is swapped in. The paper includes ablations of the sub-caption strategy, multi-positive loss, generative loss, fusion type, and token lengths.","tokens_in":41,"tokens_out":3574,"duration_ms":75107,"significance":"If the reported results hold, the paper makes a useful empirical contribution: it shows that long synthetic captions can be exploited more effectively by shortening the contrastive text input while recovering the omitted information through a generative decoder. The controlled comparison in Table 1, where CLIPA and CoCa are reproduced on the same data mixture, is a genuine strength, as are the component ablations in Table 4. The observed 'inverse effect' for synthetic captions is clearly presented and could inform future CLIP-style pretraining designs. The authors also openly acknowledge weaker ImageNet zero-shot accuracy and provide a plausible explanation. However, the load-bearing claim that single-sentence sampling is beneficial in the full model is not directly tested, and the headline numbers are single runs without released code or checkpoints, so the significance of the specific SOTA gains is currently moderate.","major_comments":[{"comment":"The ablation sequence is Baseline -> +SC -> +SC&MP -> +SC&MP&GL, so the marginal benefit of the sub-caption strategy (+1.4% I->T R@1) is measured only against Plain CLIP, without the multi-positive loss or the generative decoder. Since the generative loss already supervises the full-length synthetic caption, it is possible that the text encoder no longer suffers from long contrastive captions once the decoder is present; in that case, the single-sentence sampling would not be load-bearing for the final model. Please add an ablation that applies the multi-positive loss and generative loss with full-length synthetic captions in the contrastive branch, i.e., compare Baseline+MP+GL versus Baseline+SC+MP+GL. This is necessary to support the claim in Section 3.3 that feeding only a single sampled sentence is one of the two key designs.","section":"Section 4.3, Table 4"},{"comment":"The inverse effect shown in Figure 3 is measured on ViT-S/16, ViT-B/16, and ViT-L/16 trained for 2,000 ImageNet-equivalent epochs at 112x112 resolution, with token lengths of 128, 64, 32, and 16. The headline SOTA experiments in Table 2 use ViT-L/14 and ViT-H/14 trained for 10,000 epochs at 84x84 resolution, with the text input padded to length 80. The paper does not verify that a single randomly sampled sentence remains optimal at this larger scale, higher resolution schedule, or different model size. If the optimal fragment length depends on any of these factors, the main results could be specific to this recipe rather than reflecting a general principle. Please either provide a small-scale or mid-scale sweep of the number of sampled sentences under the final training setup, or explicitly state this transfer as a limitation and temper the claim that the single-sentence design is a robust principle.","section":"Section 3.2 and Section 4.1"},{"comment":"All headline results are reported as single runs without error bars, multiple seeds, or significance tests, and code/checkpoints are not released on the project page. Given that several SOTA differences are small (e.g., CLIPS-H/14 versus CLOC-H/14 on MSCOCO I->T R@1 differs by 0.7 points, and several LLaVA benchmark differences are a few points), the claim of new state-of-the-art results would be materially strengthened by either reporting variance over at least two or three runs or releasing the trained checkpoints so that the results can be independently verified. This is a standard expectation for empirical SOTA claims in the current literature.","section":"Section 4.2, Tables 2 and 3"}],"minor_comments":[{"comment":"The text says 'all ViTs generally achieve the strongest performance at an input token length of 32', but Footnote 2 and Figure 4a show that padding the single sampled sentence to 80 gives an additional ~1% improvement. Please clarify whether 32 refers to the effective number of real tokens before padding or the target token length, and reconcile the statement with the 80-token choice used in the main experiments.","section":"Section 3.3 and Footnote 2"},{"comment":"The formal definition of Subcaption(S, L) is ambiguous: it shows only one concatenation step, but the prose describes an iterative process of selecting sub-captions and checking the length again. Please rewrite the equation or pseudocode to make the iterative selection explicit, including the handling of ties when |S_i| = L.","section":"Section 3.2, Eq. (4)"},{"comment":"The header of Table 7 is difficult to read: the checkmarks under 'Causal mask', 'L-tokens', and 'Content' are not self-explanatory, and the rows labeled 'first' and 'random' need a short explanation in the caption. Please add a caption and, if possible, row labels that describe each configuration in words.","section":"Section 4.3, Table 7"},{"comment":"Table 4 uses a 'mixed ratio of 0.6' baseline, while Table 1 and Section 4.2 use an 80% web-captions / 20% synthetic-captions mixture. Please state explicitly how the mixed ratio in the ablation relates to the main setup, since the marginal contributions in Table 4 may not carry over exactly to the 80/20 setting.","section":"Section 4.1"},{"comment":"The sentence 'Due to the short length of the original captions, we rely entirely on synthetic captions to train CoCa' appears to contradict the previous paragraph, which says web-crawled captions were maintained for contrastive learning. Please clarify which inputs are used for the contrastive branch and which for the generative branch in these experiments.","section":"Section 4.3, 'Generated sequence length'"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of the journal and the central idea is interesting, but the missing ablation isolating the sub-caption strategy in the presence of the generative loss is a load-bearing gap. The transfer of the inverse effect from small-scale, short-horizon training to the large-scale SOTA recipe is also an unverified assumption. I would support a major revision rather than rejection, provided the authors add the requested ablation and either release code/checkpoints or add variance estimates. The heavy reliance on the authors' own prior work [29,30,31] is not problematic in itself, since the new inverse effect on synthetic captions is presented as a fresh experiment."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a well-run empirical paper with honest reporting. The headline retrieval numbers are new and the main comparisons are controlled on the same data. The single-sentence sampling design is the least well-supported piece, and the stress-test note lands: the forward ablation never tests MP+GL without SC, so we don't know if short captions still help once the generative loss is in. That is a gap, not a refutation.\n\nWhat's actually new: the asymmetric decoder that predicts the full synthetic caption from the web caption plus image, as an auxiliary task alongside contrastive learning. That is a clean idea, distinct from CoCa's symmetric decoder and DreamLip's multi-caption contrastive setup. The paper also documents a genuine inverse effect for synthetic captions (short works better than long) across S/16, B/16, L/16 with several reduction strategies, and reports SOTA retrieval numbers on MSCOCO and Flickr30K. The LLaVA transfer results are a nice extra.\n\nWhat the paper does well: Table 1 compares against CLIPA and CoCa reproduced with the same data mixture, so the gains are not just dataset luck. The ablations in Table 4 and the generation-target, fusion-type, and causal-mask studies in Tables 5-7 are thorough and clearly described. The authors also flag the ImageNet zero-shot drop and give plausible reasons. No obvious cherry-picking in the tables I checked.\n\nSoft spots, in order:\n- The missing ablation is the main one. Table 4 walks Baseline -> +SC -> +SC+MP -> +SC+MP+GL. The +SC row only measures sub-caption sampling before multi-positive or generative loss are added. Once GL is present, the text encoder may no longer be hurt by long contrastive captions, so the paper's first design might not be load-bearing in the full model. The authors should run MP+GL without SC and also SC+GL without MP. This is easily fixable and doesn't contradict anything they report, but it would settle whether single-sentence sampling actually matters in the full setup.\n- Headline numbers are single runs, no error bars, and no code or checkpoints are released. This is recipe-style work; releasing the training code would make it much more useful and would let others verify the exact numbers.\n- The inverse effect in Fig. 3 is measured at 2,000 epochs with token length 32; the SOTA runs use 10,000 epochs with ViT-L/H and input length 80. The footnote about padding length (32 vs 80, about 1 point) shows the recipe is sensitive. Transfer to the largest scale is plausible but not demonstrated directly.\n\nWho this is for: anyone working on CLIP-style pretraining with synthetic or long captions. It is a useful recipe even with the caveats.\n\nRecommendation: send to peer review. The central claim is defensible, the controlled comparisons are valuable, and the missing ablation is a reasonable request rather than a fatal flaw. I would ask the authors to add the missing ablation and release code before final acceptance.","headline":"Solid, well-executed empirical paper on synthetic-caption CLIP training; the headline retrieval numbers are new and the comparisons are controlled, but the single-sentence sampling design is under-tested because the ablation never isolates it against the generative loss.","tokens_in":17713,"tokens_out":4728,"would_cite":true,"duration_ms":45039,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Learning CLIP with one sampled sentence per synthetic caption, plus an asymmetric captioner, sets new retrieval records on MSCOCO and Flickr30K.","keywords":["CLIP","synthetic captions","contrastive pretraining","inverse scaling effect","autoregressive captioner","zero-shot retrieval","vision-language pretraining","LLaVA transfer"],"falsifier":"Train the full CLIPS pipeline at ViT-L/14 scale for 10,000 epochs while sweeping input token length (16, 32, 64, 128); if a length other than roughly one sentence yields higher MSCOCO or Flickr30K recall@1, the central inverse-effect claim for large-scale training fails.","tokens_in":16651,"feed_emoji":"🖼️","tokens_out":5620,"duration_ms":46697,"temperature":0.7,"pith_summary":"The paper argues that long, richly detailed synthetic captions should not be fed to a CLIP text encoder in full. Instead, the authors show an inverse effect: with synthetic captions, shortening the text to roughly one randomly sampled sentence consistently improves zero-shot cross-modal retrieval across model sizes, opposite to what happens with web-crawled captions. To keep the full synthetic caption useful, they add an asymmetric autoregressive decoder that takes the image and the short web-crawled caption as input and predicts the full synthetic caption, mimicking the recaptioning process. On this recipe, CLIPS sets state-of-the-art recall@1 scores on MSCOCO (76.4% image-to-text) and Flickr30K (96.6%) with a ViT-H/14 encoder, and swapping its vision encoder into LLaVA improves most multimodal benchmarks. The result matters because it suggests synthetic captions are best exploited by splitting their role: short text for contrastive alignment, full text for generation.","feed_headline":"Sampling one caption sentence lifts CLIP retrieval to new highs","feed_subtitle":"An asymmetric decoder that regenerates the full synthetic caption pushes MSCOCO and Flickr30K scores past prior methods.","key_machinery":"The load-bearing object is the inverse effect of synthetic captions: with recaptioned text, shorter inputs are consistently better for contrastive learning, so the text encoder is fed one randomly sampled sentence from each synthetic caption alongside the original web caption, with a multi-positive contrastive loss. The second mechanism is an asymmetric captioner: an autoregressive decoder with learnable tokens, image tokens, and web-caption tokens concatenated under a combination mask, trained to predict the full-length synthetic caption. Together they separate the two roles of synthetic captions, short fragments for alignment and full text for generative modeling, so the model exploits the complete caption without forcing the text encoder to process long sequences.","core_discovery":"The central discovery is a strong inverse effect in learning with synthetic captions: reducing token length from 128 to 16 consistently improves CLIP retrieval performance across ViT-S, ViT-B, and ViT-L, with sub-caption and block masks working best, and a single randomly sampled sentence, about 32 tokens, being the strongest encoder input. Building on this, the paper proposes CLIPS, which combines a multi-positive contrastive loss over the original web-crawled caption and the short synthetic sub-caption with an asymmetric autoregressive decoder. Unlike the symmetric CoCa design, the decoder conditions on image features plus web-crawled caption tokens and predicts the full-length synthetic caption, using learnable tokens and a combination mask. This setup yields gains over CLIPA, CoCa, SigLIP, and CLOC at matched model sizes, and the resulting vision encoder transfers to LLaVA, improving eight of nine evaluated benchmarks.","pith_inferences":["Editorial inference: the same inverse effect may apply to other recaptioning pipelines, so caption length could be treated as a tunable hyperparameter rather than a fixed design choice.","Editorial inference: the asymmetric decoder's success suggests web captions act as cheap summaries of the richer synthetic text; this could be tested by replacing web captions with machine-generated summaries and measuring whether retrieval degrades.","Editorial inference: the framework should combine with caption-diversity methods, since the one-sentence sampler already injects stochasticity; whether diversity and shortness are complementary is a direct extension."],"forward_implications":["CLIP pretraining with synthetic captions should use one sampled sentence per caption in the contrastive loss rather than the full recaptioned text.","The same framework lifts smaller models to the retrieval level of larger baselines: CLIPS-B/16 matches CLIPA and CoCa L/16 on MSCOCO and Flickr30K.","Long synthetic captions are more valuable as generation targets than as contrastive inputs, so the information discarded by short-sentence sampling is recovered through the captioner.","A CLIPS-trained vision encoder can replace a standard CLIP encoder in an MLLM like LLaVA and improve most evaluated benchmarks, not only retrieval."],"supporting_citations":[{"why":"Supplies the inverse-scaling observation that shortening captions can help CLIP training, which the paper extends to synthetic captions.","marker":"[31]"},{"why":"Provides the CoCa autoregressive captioning architecture that the asymmetric decoder adapts.","marker":"[52]"},{"why":"Provides the Recap-DataComp-1B recaptioned dataset and the CLIPA training baseline that CLIPS is evaluated against.","marker":"[30]"},{"why":"Supplies the sub-caption extraction idea and multi-positive contrastive setup that the encoder design simplifies.","marker":"[55]"},{"why":"Provides the SigLIP retrieval results used as the prior state of the art that CLIPS surpasses.","marker":"[54]"},{"why":"Provides evidence that visual-enriched synthetic captions improve CLIP training, motivating the synthetic-caption setting.","marker":"[23]"}],"fun_headline_variants":["One sentence beats full captions for CLIP training","Short synthetic captions push CLIP retrieval to SOTA","CLIPS: a single caption sentence outperforms long ones","Asymmetric decoder plus short captions boost CLIP","Truncated captions give CLIP a big retrieval boost"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The design assumes that the one-sentence/32-token optimum observed for ViT-S, ViT-B, and ViT-L at 2,000 epochs transfers unchanged to ViT-L/14 and ViT-H/14 trained for 10,000 epochs on the same recaptioned data.","fun_headline_variants_meta":{"raw":{"variants":["One sentence beats full captions for CLIP training","Short synthetic captions push CLIP retrieval to SOTA","CLIPS: a single caption sentence outperforms long ones","Asymmetric decoder plus short captions boost CLIP","Truncated captions give CLIP a big retrieval boost"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000181,"raw_usage":{"total_tokens":1301,"prompt_tokens":930,"completion_tokens":371,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":291}},"tokens_in":546,"tokens_out":371,"duration_ms":3873,"temperature":1.0,"reasoning_tokens":291,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:50:50.652259+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the full CLIPS pipeline at ViT-L/14 scale for 10,000 epochs while sweeping input token length (16, 32, 64, 128); if a length other than roughly one sentence yields higher MSCOCO or Flickr30K recall@1, the central inverse-effect claim for large-scale training fails.","supporting_citations":[{"cited_title":"An inverse scal- ing law for clip training","cited_arxiv_id":null,"evidence_quote":"Supplies the inverse-scaling observation that shortening captions can help CLIP training, which the paper extends to synthetic captions."},{"cited_title":"Dreamlip: Language- image pre-training with long captions","cited_arxiv_id":null,"evidence_quote":"Supplies the sub-caption extraction idea and multi-positive contrastive setup that the encoder design simplifies."},{"cited_title":"Sigmoid loss for language image pre-training","cited_arxiv_id":null,"evidence_quote":"Provides the SigLIP retrieval results used as the prior state of the art that CLIPS surpasses."},{"cited_title":"Veclip: Improving clip training via visual-enriched captions","cited_arxiv_id":null,"evidence_quote":"Provides evidence that visual-enriched synthetic captions improve CLIP training, motivating the synthetic-caption setting."}],"review_version":1}