{"id":"acbb3030-b442-4181-a182-dbf1e5b36dd3","arxiv_id":"2412.00440","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Holistic CLIP trains a multi-branch image encoder with multi-to-multi contrastive learning on multiple VLM-generated captions per image and reports consistent gains over one-to-one and one-to-multi CLIP variants.","lead":"The paper augments CLIP pretraining by generating several captions per image and splitting the visual encoder into multiple branches, then matching each caption type to its own branch. The authors report consistent gains over vanilla CLIP and one-to-multi baselines on retrieval, classification, and dense vision benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The M2M-vs-O2M comparison never holds the encoder/fusion constant: gains attributed to Eq. (5) may come from multi-branch capacity plus AVG fusion.","rationale":"The paper's strongest and most repeated claim is that M2M contrastive learning beats O2M contrastive learning on the same data. The reader's weakest-assumption analysis focuses on caption diversity and arbitrary branch-to-caption assignment. Those are legitimate concerns, but the more fundamental issue is that the comparison is not held at equal architecture and inference protocol: M2M always gets H image embeddings and AVG fusion, while O2M gets one embedding and no fusion. Table 6 shows that fusion choice alone changes COCO I2T R@1 by about 2 points (36.3 to 38.2), which is the same order as several reported M2M-vs-O2M gaps. Section 3.3.3 introduces the multi-branch encoder as part of the method, and Section 4.3 claims the loss is the principal driver because ΨCLS and ΨMLP behave similarly; however, both variants include H embeddings, so this argument does not isolate the loss. A fair O2M baseline with identical multi-branch capacity and identical inference fusion is the minimal control needed. Without it, the quantitative gains attributed to Eq. (5) are not established. This does not reject the method; the idea is plausible and the ablation structure is otherwise internally consistent. It does mean the central attribution should remain conditional until the controlled experiment is run or code and trained checkpoints are released to allow others to run it.","tokens_in":21866,"tokens_out":3119,"duration_ms":33637,"concrete_test":"Run a controlled ablation on CC3M with the multi-prompt data: (1) M2M-ΨCLS as reported; (2) O2M-ΨCLS, using the same H CLS tokens and same AVG fusion but trained with Eq. (4)'s one-to-multi loss, treating all M captions as positives for each branch; (3) O2M-ΨCLS with randomly permuted caption-to-branch assignment per image. Report MSCOCO and Flickr30 R@1 plus ImageNet Top-1. If arm (2) closes most of the M2M-O2M gap, the gain is from the multi-branch encoder/fusion, not the M2M loss; if arm (3) matches M2M, the specific matching in Eq. (5) is not the operative ingredient.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that multi-to-multi contrastive loss (Eq. 5) outperforms one-to-multi loss (Eq. 4) on identical (image, multi-text) data. In every reported comparison, M2M uses H image embeddings from H CLS tokens or parallel MLP heads, fused by averaging (Table 6), while O2M uses a single image embedding with no such fusion. Thus the consistent 1.5-7% gains attributed to the loss could instead be explained by (a) an ensemble of H learned features, (b) the extra attention/MLP capacity of the multi-branch encoder, or (c) the AVG fusion strategy. Section 4.3 notes that ΨCLS and ΨMLP behave similarly, but both add H embeddings, so this does not remove the confound. No O2M arm with the same multi-branch encoder and same inference fusion is reported. The reader's caption-diversity concern is real, but the missing controlled baseline is more load-bearing because it attacks attribution of the central mechanism itself.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a holistic extension of CLIP in which each image is paired with multiple VL M-generated captions (from a single VLM with diverse prompts or from multiple VLMs) and trained with a new multi-to-multi (M2M) contrastive loss. The image encoder is modified to produce H visual embeddings, either through extra CLS tokens (ΨCLS) or parallel MLP branches (ΨMLP), and the loss in Eq. (5) aligns each visual branch with one caption. The authors compare M2M against vanilla CLIP and one-to-multi (O2M) contrastive baselines on short-text retrieval, long-text retrieval, zero-shot classification, captioning, and LLaVA-based dense evaluation, reporting consistent gains on CC3M and CC12M.","tokens_in":22089,"tokens_out":4014,"duration_ms":42040,"significance":"If the attribution were clean, the paper would offer a practical recipe for CLIP-style pre-training: generate diverse captions, dedicate one image branch to each caption type, and train with per-branch contrastive matching. The empirical sweep is broad, the two multi-branch implementations are parameter-efficient, and the flexible inference customization in Table 7 is a nice interpretability bonus. The paper is also honest about limited compute and about not exploring very large caption counts. However, the central claim that M2M learning, rather than the accompanying multi-branch architecture and fusion strategy, causes the gains is not currently established, because the O2M and M2M comparisons never hold the encoder and inference fusion fixed.","major_comments":[{"comment":"The O2M-vs-M2M comparison confounds the loss with the encoder architecture and the inference fusion. In Eq. (4), O2M uses a single image embedding v, while M2M uses H embeddings from ΨCLS or ΨMLP and averages them at inference (Section 3.3.3, Table 6). The reported 1.5–7% gains attributed to Eq. (5) could therefore be explained by ensembling H learned features, by the added attention/MLP capacity of the multi-branch encoder, or by the AVG fusion strategy. Section 4.3 only compares ΨCLS with ΨMLP, both of which produce H embeddings, so it does not remove this confound. Please add an O2M arm that uses the same multi-branch encoder and the same inference fusion, or otherwise isolate the effect of the M2M loss from the effect of the multi-branch representation.","section":"§3.3.2–3.3.3, Tables 1–5"},{"comment":"No seed variance or error bars are reported anywhere, and several central M2M-over-O2M gains are small (e.g., 0.3–3% in Table 1 and 1–4% in Table 2). Single-run comparisons at this scale may be within training noise. Please report mean and standard deviation over at least three seeds for the central comparisons, or justify why single seeds are adequate for these effect sizes.","section":"Tables 1–8"},{"comment":"The paper's diversity measure is only the pairwise text similarity, not the semantic accuracy or distinctness of each prompted caption. If, for example, the 'Style' and 'Details' prompts produce overlapping captions, the one-to-one branch-to-caption assignment in Eq. (5) becomes arbitrary, and the reported gains could be due to more training text rather than to true part-to-part matching. Please add a human or automatic evaluation that verifies each prompt reliably produces captions with the intended semantic dimension.","section":"Section 3.2, Remark; Section B.3"},{"comment":"Reproducibility is incomplete for a data-centric method: the generated (image, multi-texts) captions are not released, and the GitHub link has no commit hash. The captioning pipeline (prompt set, filtering thresholds, and VLM versions) is described only in high-level terms, yet the captions are the core new data. Please release the captions for CC3M/CC12M or, at minimum, a documented sample with exact prompts and filtering code.","section":"Section 4.1; Appendix B.3; code URL"}],"minor_comments":[{"comment":"The section header numbering is duplicated: 'Sec 3.2 reports image-text holistic alignment data' and 'Sec 3.2 details multi-to-multi contrast' should refer to Sections 3.2 and 3.3, respectively.","section":"Section 3"},{"comment":"The notation in Eq. (5) should explicitly define v_{i,j} and t_{i,j}; currently the reader must infer that i indexes the batch and j indexes the M text branches, and the capitalization of K and M in the sums is inconsistent with Eq. (4).","section":"Eq. (5)"},{"comment":"Table 4's header is confusing: the 'Training Text' column appears to mix the training dataset name and the number of captions without a clear sub-header. Please restructure the table so each row's training data and caption count are explicit.","section":"Table 4"},{"comment":"The dataset name is written inconsistently as 'Sharegpt4-5K' in Table 2 and 'Sharegpt4v-5K' in the text; use one name throughout.","section":"Table 2 and Section 4.2"},{"comment":"There are multiple typographical and spacing issues, including 'constrastive' in several places and 'LLaV A' instead of 'LLaVA'; a careful proofreading pass is needed.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The central idea is interesting and the experimental sweep is broad, but the missing controlled O2M-with-multi-branch baseline is load-bearing for the paper's main attribution claim. I would like to see that experiment, plus seed variance and released captions, before judging the method's advantage. I do not see any ethical concerns, and the paper's scope fits cs.CV."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is worth a serious look, but the headline number is not backed by the right comparison. The multi-branch image encoder plus per-branch contrastive matching to multiple captions is a plausible recipe, and the experiments are extensive and internally consistent. Yet the O2M baseline uses one image embedding; M2M uses H embeddings and averages them at inference. So the reported 1.5-7% gains could just be an ensemble/model-capacity effect, not the loss in Eq. (5). The authors note that ΨCLS and ΨMLP behave similarly, but both add H embeddings, so that doesn't remove the confound. A simple control—O2M with the same multi-branch encoder and AVG fusion—is missing. Until that exists, the paper's central attribution is unsupported.\n\nWhat is genuinely new: the combination of a single VLM with five hand-designed prompts to generate diverse captions, a multi-branch encoder (extra CLS tokens or parallel MLP heads), and the M2M objective. The attention visualizations show semantically distinct branches, which is encouraging. The comparisons across short/long retrieval, classification, captioning, and LLaVA-based tasks are broad, and the data pipeline is cheap and reproducible in spirit.\n\nSoft spots beyond the confound: no error bars or seed variance; code link lacks a commit hash and generated captions aren't released; per-caption accuracy/diversity is only indirectly measured (text-similarity stats plus attention maps). The long-context test sets are filtered to 5K samples, and possible overlap with captioner pretraining is not discussed, though this would affect both arms equally.\n\nBottom line: this is a serious empirical paper that deserves referee time. I'd recommend sending it out, and making the O2M-with-multi-branch control a condition of acceptance.","headline":"A useful multi-caption CLIP recipe, but the headline M2M-vs-O2M comparison never holds the encoder/fusion constant; add that control before believing the loss attribution.","tokens_in":22665,"tokens_out":4314,"would_cite":false,"duration_ms":41205,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Contrastive pretraining can shed its 'myopic' one-short-text bias by giving each image multiple captions and one visual branch per caption, trained with a multi-to-multi loss — consistently beating one-to-multi on the same data.","keywords":["contrastive language-image pre-training","multi-to-multi contrastive learning","multi-branch image encoder","diverse caption generation","image-text retrieval","zero-shot classification","dense visual understanding"],"falsifier":"Train the identical multi-to-multi pipeline but randomly permute which prompt type is assigned to which image branch across images. If retrieval and classification performance stays essentially the same, then the per-branch matching is not actually carrying the gains. A second check: measure average pairwise cosine similarity among the five generated captions on a held-out sample; if it approaches the similarity of captioning the same image twice with the same prompt, the 'diversity' premise fails.","tokens_in":21686,"feed_emoji":"🖼️","tokens_out":9978,"duration_ms":84563,"temperature":0.7,"pith_summary":"The paper argues that standard CLIP-style contrastive pretraining is 'myopic': because each web image is paired with a single short text and squeezed into one embedding vector, the model misses the many valid descriptions of an image and underperforms on long-text, fine-grained, and dense visual tasks. To fix this, the authors generate several captions per image with one captioning model using five different prompt 'spirits' (details, nouns, main object, background, style), then give the image encoder one branch per caption and train with a multi-to-multi contrastive loss that aligns each branch to its own caption rather than all captions to one vector. The paper reports that this holistic pipeline beats both the original CLIP and one-to-multi contrastive variants on the same data across image-text retrieval, zero-shot classification, captioning, and dense visual question-answering benchmarks. If the claim holds, the recipe is simple: diversify captions, dedicate a branch per caption type, and let contrastive pre-training decompose the image accordingly.","feed_headline":"Matching every caption to its own image branch beats one-to-multi CLIP","feed_subtitle":"Five captions per image, each aligned to its own visual branch, beat one-to-multi CLIP on the same data.","key_machinery":"The load-bearing machinery is the multi-to-multi contrastive loss together with a multi-branch image encoder. The encoder is modified in one of two cheap ways: initialize H extra class tokens that each attend to the image and output their own embedding, or extend the last three MLP layers into H parallel heads (about 3 percent parameter increase per head). The loss extends InfoNCE from one-to-one to one-to-multi: for each of M captions, it pulls the matching visual branch toward that caption and pushes other images' same-branch embeddings away; text-to-image and image-to-text directions are averaged symmetrically. The prompt set is equally central: five prompts (Details, Nouns, Main Object, Background, Style) steer one captioning model to produce texts that are genuinely different from each other, which is what gives the branches distinct semantics to learn.","core_discovery":"The central discovery claimed is that an image embedding does not have to be a single summary vector. By starting with a (image, multi-text) dataset where each image has M captions from different perspectives, and modifying the CLIP image encoder to output H parallel embeddings (either H initialized class tokens or H parallel MLP heads), the model can learn part-to-part alignment. The multi-to-multi contrastive loss treats each (visual branch, text) pair as a positive pair and pulls them together, so each branch learns a specific aspect of the image. The paper's experiments across ten benchmarks show consistent gains over the one-to-multi baseline on identical data, with the largest margins when the captions are made diverse through carefully designed prompts rather than through using multiple different captioning models.","pith_inferences":["An editor would want to test the mechanism directly by randomizing which prompt type is assigned to which image branch across images; if performance barely moves, the gains come from extra training text rather than from part-to-part matching.","The paper shows caption-pair similarity dropping as caption count grows, but not per-caption quality; measuring whether each of the five captions is individually accurate and distinct from the others on a held-out sample would pin down the data premise.","The disentangled branches suggest task-specific inference without retraining, such as using only the 'Main Object' branch for object classification and the 'Long Text' branch for retrieval-heavy queries, and the paper's Table 7 supports the feasibility.","Naturally multi-text data, such as product listings with attribute descriptions, merchant text, and user reviews, would be a direct transfer domain for the same branch-per-text recipe."],"forward_implications":["On the same multi-text training data, the multi-to-multi objective outperforms one-to-multi on short-text and long-text retrieval, zero-shot classification, image captioning, and dense visual question-answering benchmarks.","Five prompts aimed at different aspects (details, nouns, main object, background, style) produce more useful training signal than five captions from five different captioning models, and generally more captions help.","Because the visual branches are disentangled, a branch that hurts a particular task (for example, background details on object-centric classification) can be dropped at inference without retraining.","The extra image branches add only a small parameter overhead (about 3 percent per head for the MLP variant), and average fusion restores a single embedding at no extra retrieval cost."],"supporting_citations":[{"why":"The one-to-one contrastive baseline that defines CLIP; supplies the architecture and losses the paper modifies.","marker":"[57]"},{"why":"Recent one-to-multi contrastive methods the paper reproduces on identical multi-text data and compares against.","marker":"[18, 47, 83]"},{"why":"The image-captioning model used with the five prompts to generate multi-text pairs.","marker":"[9]"},{"why":"The web-scale training dataset whose images are re-captioned into multi-text pairs.","marker":"[63]"},{"why":"The larger web-scale training dataset used for most experiments and ablations.","marker":"[4]"},{"why":"The captioning head trained on the holistic image encoder to measure embedding quality.","marker":"[55]"},{"why":"The evaluation suite used for the dense visual reasoning benchmarks.","marker":"[17]"}],"fun_headline_variants":["Multi-branch CLIP: one image, many views, better alignment","Holistic CLIP: beyond one-to-one contrastive learning","Diverse captions, multiple embeddings: CLIP gets holistic","CLIP's myopia cured: part-to-part matching wins","Multi-to-multi contrastive learning boosts CLIP"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The five hand-written prompts must make one captioning model produce captions that are both diverse and individually accurate enough that each caption can be cleanly assigned to one image branch; if the captions are redundant, noisy, or inconsistently ordered across images, the one-to-one branch-to-caption matching in the loss becomes arbitrary and most of the reported gains would come from having extra training text rather than from the multi-to-multi mechanism.","fun_headline_variants_meta":{"raw":{"variants":["Multi-branch CLIP: one image, many views, better alignment","Holistic CLIP: beyond one-to-one contrastive learning","Diverse captions, multiple embeddings: CLIP gets holistic","CLIP's myopia cured: part-to-part matching wins","Multi-to-multi contrastive learning boosts CLIP"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000239,"raw_usage":{"total_tokens":1506,"prompt_tokens":931,"completion_tokens":575,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":547,"completion_tokens_details":{"reasoning_tokens":489}},"tokens_in":547,"tokens_out":575,"duration_ms":5108,"temperature":1.0,"reasoning_tokens":489,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:23:09.396921+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the identical multi-to-multi pipeline but randomly permute which prompt type is assigned to which image branch across images. If retrieval and classification performance stays essentially the same, then the per-branch matching is not actually carrying the gains. A second check: measure average pairwise cosine similarity among the five generated captions on a held-out sample; if it approaches the similarity of captioning the same image twice with the same prompt, the 'diversity' premise fails.","supporting_citations":[{"cited_title":"Learn- ing transferable visual models from natural language super- vision","cited_arxiv_id":null,"evidence_quote":"The one-to-one contrastive baseline that defines CLIP; supplies the architecture and losses the paper modifies."},{"cited_title":"Conceptual captions: A cleaned, hypernymed, im- age alt-text dataset for automatic image captioning","cited_arxiv_id":null,"evidence_quote":"The web-scale training dataset whose images are re-captioned into multi-text pairs."}],"review_version":1}