Pith. sign in

REVIEW 5 major objections 5 minor 3 cited by

CF-VLM:CounterFactual Vision-Language Fine-tuning

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Counterfactual fine-tuning lifts VLM compositional accuracy by 5 points.

desk verdict A plausible counterfactual fine-tuning recipe with genuinely useful components, but the empirical claim is not yet well-defined due to internal inconsistencies and an implausible generalization number. read the letter →

arxiv 2506.17267 v1 pith:IA43GNDK submitted 2025-06-10 cs.LG cs.AI

Jusheng Zhang , Kaitong Cai , Yijia Fan , Jian Wang , Keze Wang This is my paper
classification cs.LGcs.AI
keywords counterfactuallearningvision-languagemodelscompositionalreasoningfine-graineddiscriminationcausalcontrastivevisualhallucinationmitigation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a general vision-language model can be taught to reason about fine-grained semantic differences by fine-tuning it on counterfactual samples: pairs in which one image or one caption element is changed so that the scene's logical meaning flips. It introduces three objectives: keep ordinary image-text alignment, separate an anchor scene from complete 'parallel' counterfactual scenes, and force a margin between an image-text pair and the same text paired with a minimally edited image. On compositional benchmarks, fine-tuning Qwen-VL 7B this way raises ConMe accuracy from 82.6% to 87.57% and lifts ARO and VL-Checklist scores several points, with smaller but consistent gains on LLaVA-1.5 and CLIP-ViT-B/32. The paper also reports slightly lower object-existence and attribute hallucinations on POPE and MME. If these results hold, counterfactual fine-tuning is a transferable recipe for making VLMs more causally sensitive and factually consistent.

What carries the argument

The load-bearing object is the counterfactual sample and the fine-grained causal discrimination loss $\mathcal{L}_{\mathrm{fcd}}$ that consumes it. A counterfactual is defined as a minimal, semantically decisive edit: a change to color, category, pose, spatial relation, quantity, or a flipped action-outcome relation. $\mathcal{L}_{\mathrm{fcd}}$ pairs the anchor text with each minimally edited image and applies a hinged margin against the anchor image-text similarity, using only the hardest negative; $\mathcal{L}_{\mathrm{csd}}$ separates the anchor from jointly edited complete scenarios; and $\mathcal{L}_{\mathrm{align}}$ preserves base alignment. Together the losses convert the generative editing process into structured supervision over the causal variables the benchmarks probe.

What would settle it

Sample a few hundred generated counterfactual pairs and have independent annotators label how many edits, if any, actually changed; if a large fraction of pairs differ in more than the one intended attribute, the claimed causal-sensitivity mechanism is not being trained and the observed benchmark gains would be explainable as generic augmentation.

Watch

Extended reading notes

Core claim

The central claim is that VLMs struggle with compositional reasoning not because they cannot represent attributes and relations, but because standard contrastive training never forces them to notice which differences matter. CF-VLM therefore constructs, for each factual pair $(I_a, T_a)$, a set of complete counterfactual scenarios $(I^{\mathrm{cf}}_k, T^{\mathrm{cf}}_k)$ and a set of minimally edited counterfactual images $I^{\mathrm{cf\_edit}}_j$, generated by an SDXL image pipeline and by LLM-driven caption rewriting. The model is trained with three losses: alignment ($\mathcal{L}_{\mathrm{align}}$), counterfactual scenario discrimination ($\mathcal{L}_{\mathrm{csd}}$), and fine-grained causal discrimination ($\mathcal{L}_{\mathrm{fcd}}$), where the last selects the hardest negative among the edited images and penalizes it whenever $S(I^{\mathrm{cf\_edit}}_j, T_a)$ comes within margin $m_2$ of $S(I_a, T_a)$. The paper's claim is that this explicit counterfactual margin makes the embedding space sensitive to the exact causal decision points that compositional benchmarks test, and that the same sensitivity transfers to hallucination-sensitive tasks.

Load-bearing premise

The load-bearing premise is that the generator changes exactly one intended semantic element and nothing else; if SDXL alters identity, pose, background, or lighting along with the target attribute, the fine-grained loss trains the model to rely on spurious differences rather than the intended causal feature.

Editorial extensions

If this is right

  • Compositional reasoning on benchmarks like ConMe, ARO, and VL-Checklist should improve for any large VLM fine-tuned with counterfactual pairs, not just Qwen-VL and LLaVA-1.5.
  • Object-existence and attribute hallucination scores should improve even though hallucination is not directly optimized.
  • The method can be applied as a plug-in fine-tuning step on top of an existing pretrained VLM, requiring only the counterfactual generation pipeline and the three losses.
  • Both image and text counterfactuals are needed; the ablations show that text-only or image-only supervision captures only part of the gain.
  • More counterfactuals per anchor help up to roughly a 4:1 ratio and then plateau, so the recipe has a cheap operating point for limited training budgets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A strong test of the causal claim would be to train with counterfactuals whose edits are human-verified to be single-attribute; if the gains grow with edit fidelity, the mechanism is truly causal sensitivity rather than generic data augmentation.
  • The hardest-negative selection could double as a diagnostic: the identities of the most confusing edits reveal which attributes or relations the model still conflates.
  • The paper's own comparison shows real human-edited counterfactuals score higher than synthetic ones, so investing in higher-fidelity edit pipelines could push the reported results further.
  • The same recipe may transfer to tasks the paper lists as future work, such as VQA and image editing, where a model must answer 'what would change if...' questions.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes CF-VLM, a fine-tuning framework that augments a pretrained vision-language model with counterfactual image-text pairs generated by SDXL and an LLM, and optimizes three objectives: a standard cross-modal alignment loss (Lalign, Eq. 1), a counterfactual scenario discrimination loss (Lcsd, Eq. 2), and a fine-grained causal discrimination loss (Lfcd, Eq. 3). The authors apply the method to CLIP ViT-B/32, Qwen-VL 7B, and LLaVA-1.5, and report consistent improvements over baselines on ConMe, ARO, VL-Checklist, POPE, and MME, as well as zero-shot ImageNet and retrieval numbers. The central claim is that exposing the model to minimal image-text edits during fine-tuning teaches it to attend to the causal decision points that determine image-text matching, thereby improving compositional reasoning and factual consistency.

Significance. If the reported gains are reproducible, CF-VLM would be a simple and broadly applicable fine-tuning recipe for improving compositional understanding and reducing hallucinations in both CLIP-style and LLM-based VLMs. The paper is commendable for testing the recipe on multiple architectures, for including ablations of each loss component and counterfactual type, and for evaluating hallucination on POPE and MME. However, the empirical section is not currently sufficient to support the headline claim. The manuscript contains direct contradictions about the data-generation protocol (1:1 vs 1:4, Qwen2-72B vs LLaMA-3-70B, Qwen-VL vs Qwen2.5-VL), the main hyperparameters were selected by optimizing on ARO and ConMe-Avg, and one reported generalization number (ImageNet top-1 8.83%) is implausibly low. These are load-bearing issues because they change the quantity being measured, not merely the presentation.

major comments (5)
  1. [§4, App. A, D.3, D.5, F/G] The factual-to-counterfactual ratio and the counterfactual text generator are stated inconsistently across the paper. Section 4 says 'each batch containing a 1:1 ratio of factual and counterfactual samples' and Appendix D.3/Table 5 repeats 1:1, while Appendix A says 'each batch maintains a fixed 1:4 ratio' and Appendix D.5 says 'four counterfactual samples ... maintaining a 1:4 ratio'; Appendix F/G then recommend K=4 as default. Section 4 names Qwen2-72B-Instruct as the text generator, whereas Appendix D.5 names LLaMA-3-70B-Instruct. Because batch composition and generation model directly affect the contrastive losses in Eqs. (1)-(3), the method is not uniquely specified and the gains in Tables 1-2 cannot be attributed to a single configuration.
  2. [§4/Table 2/App. D.3/Fig. 7] The backbone identity is inconsistent: Section 4 and Table 2 report CF-VLM on 'Qwen-VL (7B)', while Appendix D.3 says the model was fine-tuned on 'Qwen2.5-VL (7B)' and Figure 7's caption also reads 'Qwen2.5-VL-7B'. Table 2 lists 'LLaVA-1.5 (13B, Reported)' as a comparison baseline but the CF-VLM row says 'LLaVA-1.5 7B'. These are different model versions and sizes, so the reported transferability and the baseline comparisons are not well-defined; please state exactly which checkpoint was used for every row and correct or rerun the affected experiments.
  3. [App. D.3] Appendix D.3 states that the loss weights α, β, γ and margins m1, m2 were selected by grid search 'optimizing for average performance Improvement on key compositional reasoning benchmarks (ARO, Conme-Avg)'. ARO and ConMe are the same benchmarks on which the paper claims state-of-the-art results in Tables 1-2, so the reported improvements are partly a selection artifact. Please specify a validation split disjoint from the evaluation test sets, report the default-configuration results, and state whether the chosen hyperparameters were ever evaluated on truly held-out data.
  4. [§4.1, Fig. 3] Section 4.1 reports zero-shot ImageNet-1k top-1 accuracy of 8.83% (top-5 19.54%) for CF-VLM on CLIP ViT-B/32 and calls this 'strong generalization'. This value is far below the commonly reported ~60% top-1 for zero-shot CLIP ViT-B/32 and is not credible for any standard evaluation of this model. Either the evaluation protocol is broken (e.g., incorrect class prompts, wrong normalization, mismatched label order) or the number is a typo. Given the text claims CF-VLM 'outperforming or matching CLIP-based SOTA', this number and the missing baselines in Figure 3 must be corrected before the generalization claim can be assessed.
  5. [Eq. (3), App. I, App. H.3] Eq. (3) trains the model to rank the original image above minimally edited counterfactual images paired with the original text, which is exactly the operation measured by ConMe, ARO, and VL-Checklist. Combined with the benchmark-aware hyperparameter selection in Appendix D.3, the reported gains on these benchmarks may reflect specialization to the evaluation paradigm rather than a general improvement in causal reasoning. The manuscript should add an out-of-distribution evaluation (e.g., counterfactuals generated by a different pipeline or semantically unseen edit types) and per-sample verification that the SDXL/LLM edits are truly minimal; the aggregate CLIP similarity, entropy, and KL divergence in Appendix I do not verify edit fidelity, and Appendix H.3 itself acknowledges the lack of comprehensive validation of synthetic samples.
minor comments (5)
  1. [§4] Section 4 says 'All results are averaged over three random seeds, with standard deviations reported', but Tables 1, 2, 3, 4 and Figure 3 show no standard deviations; please add them or remove the claim.
  2. [§3.1] Section 3.1 refers to counterfactual types 'as defined in Section 2.3', but the definitions appear in Section 2.2; please fix the cross-reference.
  3. [Table 4] Table 4's header row is garbled in the manuscript text, with the loss-component checkmarks not aligned to the columns, making the ablation matrix hard to read.
  4. [Figures 1 and 4] Figure 1 contains the typo 'Triplet Cip' and the labels in Figure 4's right-hand plot are missing, reducing interpretability.
  5. [References] References [44] and [57] both point to the same 'Causal graphical models for vision-language compositional understanding' paper, and references [58] and [60] both point to Rao et al.; please merge the duplicates.

Circularity Check

1 steps flagged · score 4.0 of 10

Main ARO/ConMe gains are benchmark-tuned: loss weights and margins are grid-searched on ARO and ConMe-Avg, so the headline compositional-reasoning comparison is partly fitted rather than independently predicted.

  1. fitted input called prediction [Appendix D.3, 'Hyperparameter Settings for CF-VLM Fine-tuning']
    "The selection of loss weights ( α, β, γ) and margins (m1, m2) was performed via grid search on a small subset of the CC3M validation data, optimizing for average performance Improvement on key compositional reasoning benchmarks (ARO, Conme-Avg)."

    This is the load-bearing validation step for the claim that CF-VLM 'consistently outperforms strong baselines and state-of-the-art methods on compositional reasoning and generalization benchmarks.' The loss weights and margins are fitted by maximizing average performance on ARO and ConMe-Avg, and Tables 1-2 then report ARO and ConMe as the main evidence of superiority. The reported gains on those benchmarks are outputs of the same selection criterion used to choose the configuration, not independent predictions; the paper does not state that the final ARO/ConMe numbers come from a held-out split withheld from the grid search.

full rationale

The derivation of CF-VLM's losses is not circular by construction: Lfcd (Eq. 3) supervises S(Ia,Ta) > S(Icf_edit,Ta) with a minimally edited image, whereas ConMe/ARO/VL-Checklist rank an original text against a minimally edited text for a fixed image; the two share an edit taxonomy but are not the same function, and the evaluation images/texts are not the training samples. No load-bearing self-citation chain appears: the cited prior work is background, and no 'uniqueness' claim is imported from the authors' own papers. The one genuine circular-validation step is the hyperparameter selection in Appendix D.3, which optimizes the loss weights and margins for ARO and ConMe-Avg and then presents ARO/ConMe as the main evidence; this makes the headline compositional-reasoning gain partially a fitted result. The remaining independent evidence (POPE/MME hallucination checks and transfer to LLaVA) keeps the paper from being fully circular, hence score 4. The internal inconsistencies (1:1 vs 1:4 batch ratio, Qwen2-72B vs LLaMA-3-70B text generator, Qwen-VL vs Qwen2.5-VL backbone, and the implausible 8.83% ImageNet top-1) are correctness/reproducibility risks rather than circularity; they do not raise the circularity score but do weaken confidence in the reported numbers. The paper's own H.3 limitation about synthetic-data bias is acknowledged, but it is a generality caveat, not a circular step.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on standard contrastive math, on two strong domain assumptions about the fidelity of generative counterfactuals, and on an ad hoc equivalence between minimal edits and causal interventions. The hyperparameters of the loss are fitted to the evaluation benchmark family, which increases the circularity burden.

free parameters (4)
  • Loss weights alpha, beta, gamma = 1.0, 0.45, 0.55
    Selected by grid search on a small subset of CC3M validation to optimize ARO and ConMe-Avg, which are also the main evaluation benchmarks (Appendix D.3).
  • Hinge margins m1, m2 = 0.25, 0.30
    Selected in the same grid search on CC3M validation targeting ARO and ConMe (Appendix D.3).
  • Counterfactual ratio K = 4
    Chosen after ablation on ConMe showing peak at K=4 (Appendix F, Figure 9); main text inconsistently describes the batch ratio as 1:1 in Section 4 and 1:4 in Appendix A.
  • Temperature tau = 0.07 (initial, learnable for CLIP)
    Standard contrastive temperature, not fitted to the evaluation benchmarks; included for completeness.
assumptions (4)
  • standard math Cosine similarity and InfoNCE/hinge losses are valid measures of semantic alignment.
    Used in Lalign, Lcsd, and Lfcd (Eqs. 1-3); standard practice in contrastive learning.
  • domain assumption SDXL-generated counterfactual images preserve all non-target attributes of the anchor image.
    Section 3.1 and Appendix E rely on this for Lfcd; if edits are not minimal, the loss trains on unintended changes. Not validated per-sample.
  • domain assumption LLM-generated counterfactual texts alter exactly one semantic element and remain logically coherent.
    Used in the counterfactual scenario pairs and Lcsd; the generation prompt (Appendix A) instructs this, but outputs are not checked for violations.
  • ad hoc to paper Single-attribute edits and action-reversal edits constitute 'causal decision points' that determine image-text matching.
    The paper equates sensitivity to these edits with causal reasoning (Section 1, 3.2). This is a modeling assumption, not derived from a causal theory.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CF-VLM:CounterFactual Vision-Language Fine-tuning." pith.science (2026). https://pith.science/paper/IA43GNDK

@misc{pith2026250617267,
  author       = {Pith},
  title        = {Pith review of: CF-VLM:CounterFactual Vision-Language Fine-tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IA43GNDK}},
  note         = {Machine review of arXiv:2506.17267}
}
read the original abstract

Recent advances in vision-language models (VLMs) have greatly improved cross-modal semantic understanding, yet significant limitations remain in fine-grained discrimination and deep causal reasoning tasks. Existing VLMs often rely on superficial statistical correlations, lacking the ability to capture the underlying causal logic between visual and textual content. To address this, we propose CounterFactual Vision-Language Fine-tuning (CF-VLM), a novel framework that enhances the causal reasoning capabilities of VLMs through the targeted use of counterfactual samples. CF-VLM introduces three complementary training objectives: maintaining foundational cross-modal alignment, reinforcing the uniqueness and stability of factual scene representations against coherent counterfactuals, and sharpening the model's sensitivity to minimal but critical causal edits. Extensive experiments demonstrate that CF-VLM consistently outperforms strong baselines and state-of-the-art methods on compositional reasoning and generalization benchmarks. Furthermore, it shows promise in mitigating visual hallucinations, indicating improved factual consistency. Our CF-VLM provides a robust foundation for deploying VLMs in high-stakes, real-world scenarios requiring reliable reasoning and interpretability.

Figures

Figures reproduced from arXiv: 2506.17267 by the authors.

Figure 1
Figure 1. Illustration of CF-VLM’s training frame [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. CF-VLM training pipeline. Given a factual image-text anchor, the framework generates [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Generalization results. Left (3 columns): CF-VLM boosts baseline performance on [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Removing any counterfactual type hurts task accuracy (left); more counterfactuals improve [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Hallucination evaluation on the POPE benchmark. CF-VLM improves accuracy, precision, [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Hallucination evaluation on the MME benchmark. Relative performance gains on [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Qualitative comparisons showing CF-VLM’s improvements in fine-grained attribute [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Examples of counterfactual data generated via controlled edits to attributes, objects, [PITH_FULL_IMAGE:figures/full_fig_p021_8.png]
Figure 9
Figure 9. Figure 9: Examples of counterfactual data generated via controlled edits to attributes, objects, [PITH_FULL_IMAGE:figures/full_fig_p022_9.png]
Figure 10
Figure 10. Figure 10: Evaluation results across four dimensions. Higher is better for entropy, CLIP similarity, [PITH_FULL_IMAGE:figures/full_fig_p028_10.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SIVA-RL: Sensitivity-Invariance Visual Alignment for Multimodal Reinforcement Learning

    cs.CV 2026-07 conditional novelty 6.0 of 10

    SIVA-RL uses the observed reward drop between clean and locally edited images to route training toward sensitivity or invariance, improving GRPO/DAPO-based multimodal RL across nine benchmarks.

  2. When Vision Overrides Language: Evaluating and Mitigating Counterfactual Failures in VLAs

    cs.CV 2026-02 conditional novelty 5.0 of 10

    VLAs fail most counterfactual instructions because vision shortcuts dominate language; the new LIBERO-CF benchmark quantifies this, and CAG, an inference-time action mixer, improves grounding and success.

  3. OSC: Cognitive Orchestration through Dynamic Knowledge Alignment in Multi-Agent LLM Collaboration

    cs.AI 2025-09 reject novelty 5.0 of 10

    OSC uses learned Collaborator Knowledge Models and RL-trained communication policies to make LLM agents communicate adaptively, claiming gains on AlpacaEval 2.0 and MT-Bench.

Reference graph

Works this paper leans on

81 extracted references · 52 canonical work pages · cited by 3 Pith papers

  1. [1]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,

    J. Li, D. Li, C. Xiong, and S. Hoi, “Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,” 2022. [Online]. Available: https://arxiv.org/abs/2201.12086

    arXiv 2022

  2. [2]

    A survey of vision-language pre-trained models,

    Y . Du, Z. Liu, J. Li, and W. X. Zhao, “A survey of vision-language pre-trained models,” 2022. [Online]. Available: https://arxiv.org/abs/2202.10936

    arXiv 2022

  3. [3]

    Flamingo: a visual language model for few-shot learning,

    J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y . Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, R. Ring, E. Rutherford, S. Cabi, T. Han, Z. Gong, S. Samangooei, M. Monteiro, J. Menick, S. Borgeaud, A. Brock, A. Nematzadeh, S. Sharifzadeh, M. Binkowski, R. Barreira, O. Vinyals, A. Zisser- man, and K. Simonyan, “Flamingo: a visual language m...

    2022

  4. [4]

    Causal inference with large language model: A survey,

    J. Ma, “Causal inference with large language model: A survey,” 2025. [Online]. Available: https://arxiv.org/abs/2409.09822

    arXiv 2025

  5. [5]

    Cello: Causal evaluation of large vision-language models,

    M. Chen, B. Peng, Y . Zhang, and C. Lu, “Cello: Causal evaluation of large vision-language models,” 2024. [Online]. Available: https://arxiv.org/abs/2406.19131

    arXiv 2024

  6. [6]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervision,” 2021. [Online]. Available: https://arxiv.org/abs/2103.00020

    arXiv 2021

  7. [7]

    Measuring progress in fine-grained vision-and-language understanding,

    E. Bugliarello, L. Sartran, A. Agrawal, L. A. Hendricks, and A. Nematzadeh, “Measuring progress in fine-grained vision-and-language understanding,” inProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), A. Rogers, J. Boyd-Graber, and N. Okazaki, Eds. Toronto, Canada: Association for Computational...

    work page 2023

  8. [8]

    Synthesize, diagnose, and optimize: Towards fine-grained vision-language understanding,

    W. Peng, S. Xie, Z. You, S. Lan, and Z. Wu, “Synthesize, diagnose, and optimize: Towards fine-grained vision-language understanding,” in2024 IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion (CVPR), 2024, pp. 13 279–13 288

    work page 2024

Show all 81 references
  1. [9]

    Finer: Investigating and enhancing fine-grained visual concept recognition in large vision language models,

    J. Kim and H. Ji, “Finer: Investigating and enhancing fine-grained visual concept recognition in large vision language models,” inProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y . Al-Onaizan, M. Bansal, and Y .-N. Chen, Eds. Miami, Flo...

    2024

  2. [10]

    Benchmarking zero-shot recognition with vision-language models: Challenges on granularity and specificity,

    Z. Xu, Y . Zhu, S. Deng, A. Mittal, Y . Chen, M. Wang, P. Favaro, J. Tighe, and D. Modolo, “Benchmarking zero-shot recognition with vision-language models: Challenges on granularity and specificity,” in2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Worksho...

    2024

  3. [11]

    Vilta: Enhancing vision-language pre-training through textual augmentation,

    W. Wang, Z. Yang, B. Xu, J. Li, and Y . Sun, “Vilta: Enhancing vision-language pre-training through textual augmentation,” in2023 IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 3135–3146

    2023

  4. [12]

    Towards vision-language mechanistic interpretability: A causal tracing tool for blip,

    V . Palit, R. Pandey, A. Arora, and P. P. Liang, “Towards vision-language mechanistic interpretability: A causal tracing tool for blip,” in2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), 2023, pp. 2848–2853

    2023

  5. [13]

    What matters when building vision-language models?

    H. Laurençon, L. Tronchon, M. Cord, and V . Sanh, “What matters when building vision-language models?”

  6. [14]

    Fine-grained alignment for cross-modal recipe retrieval,

    M. Wahed, X. Zhou, T. Yu, and I. Lourentzou, “Fine-grained alignment for cross-modal recipe retrieval,” in2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2024, pp. 5572–5581

    2024

  7. [15]

    Localized triplet loss for fine-grained fashion image retrieval,

    A. D’Innocente, N. Garg, Y . Zhang, L. Bazzani, and M. Donoser, “Localized triplet loss for fine-grained fashion image retrieval,” in2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2021, pp. 3905–3910

    2021

  8. [16]

    Tripletclip: Improving compositional reasoning of clip via synthetic vision-language negatives,

    M. Patel, A. Kusumba, S. Cheng, C. Kim, T. Gokhale, C. Baral, and Y . Yang, “Tripletclip: Improving compositional reasoning of clip via synthetic vision-language negatives,” 2024. [Online]. Available: https://arxiv.org/abs/2411.02545

    2024 arXiv

  9. [17]

    Facenet: A unified embedding for face recognition and clustering,

    F. Schroff, D. Kalenichenko, and J. Philbin, “Facenet: A unified embedding for face recognition and clustering,” in2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Jun. 2015, p. 815–823. [Online]. Available: http://dx.doi.org/10.1109/CVPR.2015.7298682 10

    2015

  10. [19]

    CPL: Counterfactual prompt learning for vision and language models,

    X. He, D. Yang, W. Feng, T.-J. Fu, A. Akula, V . Jampani, P. Narayana, S. Basu, W. Y . Wang, and X. Wang, “CPL: Counterfactual prompt learning for vision and language models,” inProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Y . Goldber...

    2022

  11. [21]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” 2023. [Online]. Available: https://arxiv.org/abs/1706.03762

    2023 arXiv

  12. [22]

    A survey on evaluation of large language models,

    Y . Chang, X. Wang, J. Wang, Y . Wu, L. Yang, K. Zhu, H. Chen, X. Yi, C. Wang, Y . Wang, W. Ye, Y . Zhang, Y . Chang, P. S. Yu, Q. Yang, and X. Xie, “A survey on evaluation of large language models,”ACM Trans. Intell. Syst. Technol., vol. 15, no. 3, Mar. 2024. [Online]. Availa...

    2024 doi

  13. [23]

    A survey of visual transformers,

    Y . Liu, Y . Zhang, Y . Wang, F. Hou, J. Yuan, J. Tian, Y . Zhang, Z. Shi, J. Fan, and Z. He, “A survey of visual transformers,”IEEE Transactions on Neural Networks and Learning Systems, vol. 35, no. 6, pp. 7478–7498, 2024

    2024

  14. [24]

    A simple framework for contrastive learning of visual representations,

    T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” inProceedings of the 37th International Conference on Machine Learning, ser. ICML’20. JMLR.org, 2020

    2020

  15. [25]

    Understanding contrastive representation learning through alignment and uniformity on the hypersphere,

    T. Wang and P. Isola, “Understanding contrastive representation learning through alignment and uniformity on the hypersphere,” inProceedings of the 37th International Conference on Machine Learning, ser. ICML’20. JMLR.org, 2020

    2020

  16. [26]

    Scaling up visual and vision-language representation learning with noisy text supervision,

    C. Jia, Y . Yang, Y . Xia, Y .-T. Chen, Z. Parekh, H. Pham, Q. V . Le, Y . Sung, Z. Li, and T. Duerig, “Scaling up visual and vision-language representation learning with noisy text supervision,” 2021. [Online]. Available: https://arxiv.org/abs/2102.05918

    2021 arXiv

  17. [27]

    Cogs: A compositional generalization challenge based on semantic interpretation,

    N. Kim and T. Linzen, “Cogs: A compositional generalization challenge based on semantic interpretation,”

  18. [28]

    Learning what makes a difference from counterfactual examples and gradient supervision,

    D. Teney, E. Abbasnedjad, and A. van den Hengel, “Learning what makes a difference from counterfactual examples and gradient supervision,” inComputer Vision – ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part X. Berlin, Heidelberg: Springe...

    2020 doi

  19. [29]

    Teaching clip to count to ten,

    R. Paiss, A. Ephrat, O. Tov, S. Zada, I. Mosseri, M. Irani, and T. Dekel, “Teaching clip to count to ten,” in 2023 IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 3147–3157

    2023

  20. [30]

    Disco: Distilling counterfactuals with large language models,

    Z. Chen, Q. Gao, A. Bosselut, A. Sabharwal, and K. Richardson, “Disco: Distilling counterfactuals with large language models,” 2023. [Online]. Available: https://arxiv.org/abs/2212.10534

    2023 arXiv

  21. [31]

    Counterfactually measuring and eliminating social bias in vision-language pre-training models,

    Y . Zhang, J. Wang, and J. Sang, “Counterfactually measuring and eliminating social bias in vision-language pre-training models,” inProceedings of the 30th ACM International Conference on Multimedia, ser. MM ’22. New York, NY , USA: Association for Computing Machinery, 2022, p...

    2022

  22. [32]

    Counterfactual attention learning for fine-grained visual categorization and re-identification,

    Y . Rao, G. Chen, J. Lu, and J. Zhou, “Counterfactual attention learning for fine-grained visual categorization and re-identification,” in2021 IEEE/CVF International Conference on Computer Vision (ICCV), 2021, pp. 1005–1014

    2021

  23. [33]

    Counterfactual samples synthesizing and training for robust visual question answering,

    L. Chen, Y . Zheng, Y . Niu, H. Zhang, and J. Xiao, “Counterfactual samples synthesizing and training for robust visual question answering,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 11, pp. 13 218–13 234, 2023

    2023

  24. [34]

    Qwen2.5 technical report,

    Qwen, :, A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Tang, T. Xia, ...

    2025 arXiv

  25. [35]

    Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts,

    S. Changpinyo, P. Sharma, N. Ding, and R. Soricut, “Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts,” inCVPR, 2021

    2021

  26. [36]

    Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning,

    P. Sharma, N. Ding, S. Goodman, and R. Soricut, “Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning,” inProceedings of ACL, 2018

    2018

  27. [37]

    Microsoft coco captions: Data collection and evaluation server,

    X. Chen, H. Fang, T.-Y . Lin, R. Vedantam, S. Gupta, P. Dollar, and C. L. Zitnick, “Microsoft coco captions: Data collection and evaluation server,” 2015. [Online]. Available: https://arxiv.org/abs/1504.00325

    2015 arXiv

  28. [38]

    Conme: Rethinking evaluation of compositional reasoning for modern vlms,

    I. Huang, W. Lin, M. J. Mirza, J. A. Hansen, S. Doveh, V . I. Butoi, R. Herzig, A. Arbelle, H. Kuehne, T. Darrell, C. Gan, A. Oliva, R. Feris, and L. Karlinsky, “Conme: Rethinking evaluation of compositional reasoning for modern vlms,” 2024. [Online]. Available: https://arxiv....

    2024 arXiv

  29. [39]

    When and why vision-language models behave like bags-of-words, and what to do about it?

    M. Yuksekgonul, F. Bianchi, P. Kalluri, D. Jurafsky, and J. Zou, “When and why vision-language models behave like bags-of-words, and what to do about it?” inInternational Conference on Learning Representations, 2023. [Online]. Available: https://openreview.net/forum?id=KRLUvxh8uaX

    2023

  30. [40]

    Vl-checklist: Evaluating pre-trained vision-language models with objects, attributes and relations,

    T. Zhao, T. Zhang, M. Zhu, H. Shen, K. Lee, X. Lu, and J. Yin, “Vl-checklist: Evaluating pre-trained vision-language models with objects, attributes and relations,” 2023. [Online]. Available: https://arxiv.org/abs/2207.00221

    2023 arXiv

  31. [41]

    ImageNet Large Scale Visual Recognition Challenge,

    O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bern- stein, A. C. Berg, and L. Fei-Fei, “ImageNet Large Scale Visual Recognition Challenge,”International Journal of Computer Vision (IJCV), vol. 115, no. 3, pp. 211–252, 2015

    2015

  32. [42]

    From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions,

    P. Young, A. Lai, M. Hodosh, and J. Hockenmaier, “From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions,”Transactions of the Association for Computational Linguistics, vol. 2, pp. 67–78, 2014

    2014

  33. [43]

    Contrasting intra-modal and ranking cross-modal hard negatives to enhance visio-linguistic compositional understanding,

    L. Zhang, R. Awal, and A. Agrawal, “Contrasting intra-modal and ranking cross-modal hard negatives to enhance visio-linguistic compositional understanding,” 2024. [Online]. Available: https://arxiv.org/abs/2306.08832

    2024 arXiv

  34. [45]

    Structure-clip: Towards scene graph knowledge to enhance multi-modal structured representations,

    Y . Huang, J. Tang, Z. Chen, R. Zhang, X. Zhang, W. Chen, Z. Zhao, Z. Zhao, T. Lv, Z. Hu, and W. Zhang, “Structure-clip: Towards scene graph knowledge to enhance multi-modal structured representations,” 2023. [Online]. Available: https://arxiv.org/abs/2305.06152

    2023 arXiv

  35. [46]

    Improved baselines with visual instruction tuning,

    H. Liu, C. Li, Y . Li, and Y . J. Lee, “Improved baselines with visual instruction tuning,” 2024. [Online]. Available: https://arxiv.org/abs/2310.03744

    2024 arXiv

  36. [47]

    Instructblip: Towards general-purpose vision-language models with instruction tuning,

    W. Dai, J. Li, D. Li, A. M. H. Tiong, J. Zhao, W. Wang, B. Li, P. Fung, and S. Hoi, “Instructblip: Towards general-purpose vision-language models with instruction tuning,” 2023. [Online]. Available: https://arxiv.org/abs/2305.06500

    2023 arXiv

  37. [48]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models,

    D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny, “Minigpt-4: Enhancing vision-language understanding with advanced large language models,” 2023. [Online]. Available: https://arxiv.org/abs/2304.10592

    2023 arXiv

  38. [49]

    Sdxl: Improving latent diffusion models for high-resolution image synthesis,

    D. Podell, Z. English, K. Lacey, A. Blattmann, T. Dockhorn, J. Müller, J. Penna, and R. Rombach, “Sdxl: Improving latent diffusion models for high-resolution image synthesis,” 2023. [Online]. Available: https://arxiv.org/abs/2307.01952

    2023 arXiv

  39. [50]

    Evaluating object hallucination in large vision-language models,

    Y . Li, Y . Du, K. Zhou, J. Wang, W. X. Zhao, and J.-R. Wen, “Evaluating object hallucination in large vision-language models,” 2023. [Online]. Available: https://arxiv.org/abs/2305.10355

    2023 arXiv

  40. [51]

    Mme: A comprehensive evaluation benchmark for multimodal large language models,

    C. Fu, P. Chen, Y . Shen, Y . Qin, M. Zhang, X. Lin, J. Yang, X. Zheng, K. Li, X. Sun, Y . Wu, and R. Ji, “Mme: A comprehensive evaluation benchmark for multimodal large language models,” 2024. [Online]. Available: https://arxiv.org/abs/2306.13394

    2024 arXiv

  41. [52]

    Vision-and-language pretrained models: A survey,

    S. Long, F. Cao, S. C. Han, and H. Yang, “Vision-and-language pretrained models: A survey,” 2022. [Online]. Available: https://arxiv.org/abs/2204.07356

    2022 arXiv

  42. [53]

    Exploring the frontier of vision-language models: A survey of current methodologies and future directions,

    A. Ghosh, A. Acharya, S. Saha, V . Jain, and A. Chadha, “Exploring the frontier of vision-language models: A survey of current methodologies and future directions,” 2024. [Online]. Available: https://arxiv.org/abs/2404.07214 12

    2024

  43. [54]

    Vision-language models for vision tasks: A survey,

    J. Zhang, J. Huang, S. Jin, and S. Lu, “Vision-language models for vision tasks: A survey,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 8, pp. 5625–5644, 2024

    2024

  44. [55]

    Counterfactual vision and language learning,

    E. Abbasnejad, D. Teney, A. Parvaneh, J. Shi, and A. van den Hengel, “Counterfactual vision and language learning,” in2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 10 041–10 051

    2020

  45. [56]

    Counterfactual reasoning for multi-label image classification via patching-based training,

    M.-K. Xie, J.-H. Xiao, P. Peng, G. Niu, M. Sugiyama, and S.-J. Huang, “Counterfactual reasoning for multi-label image classification via patching-based training,” inProceedings of the 41st International Conference on Machine Learning, ser. ICML’24. JMLR.org, 2024

    2024

  46. [57]

    Causal graphical models for vision-language compositional understanding,

    F. Parascandolo, N. Moratelli, E. Sangineto, L. Baraldi, and R. Cucchiara, “Causal graphical models for vision-language compositional understanding,” 2025. [Online]. Available: https://arxiv.org/abs/2412.09353

    2025 arXiv

  47. [58]

    Cpl: Counterfactual prompt learning for vision and language models,

    X. He, D. Yang, W. Feng, T.-J. Fu, A. Akula, V . Jampani, P. Narayana, S. Basu, W. Y . Wang, and X. E. Wang, “Cpl: Counterfactual prompt learning for vision and language models,” 2022. [Online]. Available: https://arxiv.org/abs/2210.10362

    2022 arXiv

  48. [59]

    Counterfactual visual explanations,

    Y . Goyal, Z. Wu, J. Ernst, D. Batra, D. Parikh, and S. Lee, “Counterfactual visual explanations,” 2019. [Online]. Available: https://arxiv.org/abs/1904.07451

    2019 arXiv

  49. [60]

    Counterfactual attention learning for fine-grained visual categorization and re-identification,

    Y . Rao, G. Chen, J. Lu, and J. Zhou, “Counterfactual attention learning for fine-grained visual categorization and re-identification,” 2021. [Online]. Available: https://arxiv.org/abs/2108.08728 13 A Dataset Details Our training and evaluation are conducted on a series of sta...

    2021 arXiv

  50. [63]

    Identify a cause-effect link in the sentence

  51. [64]

    Invert it: If cause A didn’t happen, effect B would change

  52. [65]

    Rules: • Modifyonly one thing(either one attribute or one causal link)

    Rewrite the sentence naturally under the new causal logic. Rules: • Modifyonly one thing(either one attribute or one causal link). • Keep all other elements in the sentenceexactly the same. • Donotrepeat any previously generated sentence. • The output must bea single fluent, g...

  53. [66]

    Example Input: A young woman holding a racket hit the ball, and the ball flew outward

    [sentence #3] ... Example Input: A young woman holding a racket hit the ball, and the ball flew outward. Output (attribute):

  54. [67]

    An old woman holding a racket hit the ball, and the ball flew outward

  55. [68]

    Output (causal):

    A transparent woman holding a racket hit the ball, and the ball flew outward. Output (causal):

  56. [69]

    dirt” with “paved roads

    A young woman holding a racket missed the ball, and the ball dropped to the ground. Now returnKdistinct counterfactual rewrites in the required format. 16 B Qualitative Analysis of Post-training Performance B.1 Diversity and Quality Control of Counterfactual Samples CF-VLM emp...

  57. [70]

    Cross-modal Alignment Loss Lalign maintains baseline image–text matching and prevents forgetting

  58. [71]

    Counterfactual Scene Distinction Loss Lcsd contrasts factual versus complete counterfac- tual scenarios to reinforce representational uniqueness

  59. [72]

    causal decision points

    Fine-grained Causal Distinction Loss Lfcd sharpens sensitivity to minimal but critical causal edits. Other works typically use a single or simpler combination of objectives: TripletCLIP employs a standard triplet loss; CPL uses contrastive learning on textual prompts; Goyal et...

  60. [73]

    Diverse, Controlled Counterfactuals: Combines high-quality minimal-edits and complete scenarios to offer multi-level contrast

  61. [74]

    Multi-Objective Synergy: Lalign, Lcsd, and Lfcd work together to maintain alignment, distinguish scenes, and detect causal edits

  62. [75]

    Deep Focus on Causal Decision Points: Guides the model to learn why small edits affect semantics, advancing causal understanding beyond feature matching

  63. [76]

    Two people on motorcycles riding them

    End-to-End, Controllable Fine-Tuning Framework: Integrates SDXL and LLM-based generation into a deployable pipeline to retrofit existing VLMs. D Detailed Experimental Setup and Hyperparameters This section provides a comprehensive overview of the experimental setup, including ...

  64. [77]

    the type and generation method of counterfactual samples

  65. [78]

    the design of core learning objectives and loss functions

  66. [79]

    causality

    the emphasis on “causality” or “fine-grained discrimination”; and

  67. [80]

    parallel realities

    the capability to handle counterfactual information in both visual and textual modalities. Type and Generation of Counterfactual SamplesCF-VLM adopts a comprehensive and refined strategy for constructing counterfactual samples. It integrates bothjointly edited image-text pairs...

  68. [81]

    These are employed in Lcsd to help the model learn semantic scene boundaries

    Leveraging diverse, high-quality, and semantically controllable counterfactual samples: CF-VLM utilizes not onlyminimally edited image counterfactuals(paired with original texts and optimized via Lfcd to enhance sensitivity to critical visual changes), but also innovatively in...

  69. [82]

    Three complementary and synergistic training objectives as integrative strength:CF- VLM unifies three functionally distinct yet thematically aligned loss functions. Lalign preserves foundational image-text alignment; Lcsd enhances the uniqueness and stability of factual repres...

  70. [83]

    Explicit and deep emphasis on causal decision points:In contrast to works that mainly improve discrimination via generic hard negatives or robustness through data augmenta- tion, CF-VLM explicitly guides the model—particularly through Lfcd—to identify, localize, and comprehend...

  71. [84]

    kicking a ball

    An end-to-end, controllable counterfactual fine-tuning framework:Beyond conceptual contribution, CF-VLM delivers a complete and practically applicable fine-tuning pipeline. It combines the generation of high-quality, semantically controllable counterfactuals—using a fine-tuned...

  72. [2020]

    Available: https://arxiv.org/abs/2010.05465

    [Online]. Available: https://arxiv.org/abs/2010.05465

    2010 arXiv

  73. [2024]

    Available: https://arxiv.org/abs/2405.02246

    [Online]. Available: https://arxiv.org/abs/2405.02246

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.