Pith. sign in

REVIEW 4 major objections 6 minor 77 references

Causality-aligned Prompt Learning via Diffusion-based Counterfactual Generation

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that diffusion-generated counterfactual images, used as hard negatives in contrastive prompt learning, align prompts with causal image features and improve accuracy on unseen classes.

desk verdict A useful empirical recipe for contrastive prompt learning with diffusion counterfactuals, but the paper's advertised identifiability guarantee is built on a theorem whose assumptions contradict its own data-generating model. read the letter →

arxiv 2507.19882 v1 pith:WOZJSV7G submitted 2025-07-26 cs.AI

Xinshu Li , Ruoyu Wang , Erdun Gao , Mingming Gong , Lina Yao This is my paper · ORCID
classification cs.AI
keywords promptlearningcounterfactualgenerationdiffusionmodelscausalinvariancecontrastivevision-languageidentifiabilityerrorbounds
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Prompt learning for vision-language models usually tunes text prompts while keeping encoders frozen, but such prompts can latch onto spurious correlations and fail on unseen classes. This paper proposes DiCap, a pipeline that generates counterfactual images with a classifier-guided diffusion model steered by an anti-causal predictor, then trains prompts against those images as hard negatives. The central claim is that these counterfactuals satisfy a minimal-sufficiency criterion, so only label-relevant features change, and that the counterfactual estimation error is bounded by the reconstruction error of the diffusion process. Under stated invertibility and independence conditions the counterfactual outcome is identifiable, and the paper reports consistent gains over CLIP and prior causal prompt-learning baselines, including an average 3.87 percentage-point improvement on unseen image-classification classes.

What carries the argument

The load-bearing mechanism is the pairing of an invertible encoding $g(\boldsymbol{x})$ with an anti-causal predictor gradient that drives diffusion sampling. The paper models images as $\boldsymbol{x}=f(y,\boldsymbol{n},\boldsymbol{u}_x)$, with label $y$ causing causal features and $\boldsymbol{n}$ causing non-causal ones; Theorem 3.2 states conditions, including $g(\boldsymbol{x})\perp y,\boldsymbol{n}$, under which the encoding is recoverable, so that replacing $y$ by the counterfactual label $y_{cf}$ in the decoder produces the true counterfactual. In practice this is implemented by a DDIM-style forward pass that produces structured noise as abduction, gradient guidance from the anti-causal predictor as action, and a dual contrastive loss with the counterfactual as hardest negative as estimation. This mechanism is what turns a generic diffusion model into a counterfactual generator with a stated error bound.

What would settle it

On a synthetic dataset with known structural equations, compute the true counterfactual $f(y_{cf},\boldsymbol{n},\boldsymbol{u}_x)$ for each sample, run DiCap, and compare the observed counterfactual error to the reconstruction error; if the observed error exceeds the reconstruction error for any sample, Corollary 3.4's bound is false.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a classifier-guided diffusion model, when guided by the gradient of an anti-causal predictor $\nabla_{\boldsymbol{x}}p_{\phi}(y_{cf}\mid \boldsymbol{x})$, generates counterfactual images that are minimally sufficient: the smallest perturbation that changes the label while preserving non-causal factors. Theorem 3.2 gives invertibility and independence conditions under which the recovered latent is identifiable, and Corollary 3.4 states that if the reconstruction error is at most $\delta$, the counterfactual estimation error is at most $\delta$. Training the tunable prompt with a dual contrastive loss, using the counterfactual as the hardest negative, then aligns the prompt embedding with causal features and yields the reported generalization gains on unseen classes.

Load-bearing premise

The argument's load-bearing assumption is that the latent code a pretrained denoiser recovers from an image is statistically independent of the image's label and its non-causal background factors, and the paper does not demonstrate that any real pretrained model satisfies this condition.

Editorial extensions

If this is right

  • Reconstruction error on factual images can serve as a certificate for counterfactual quality, removing the need for ground-truth counterfactual labels during training.
  • Seen-class-only training in this framework yields improved unseen-class accuracy, reducing the annotation burden for new categories.
  • The same counterfactual-hardest contrastive objective transfers across image classification, image-text retrieval, and visual question answering from a single prompt learner.
  • Automatically choosing the closest alternative class as the counterfactual label produces harder negatives and better prompts than random class selection.
  • Shorter prompts combined with counterfactual negatives behave as a regularizer, avoiding spurious task-specific cues.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A sharper test of the causal mechanism would hold the prompt learner fixed and vary only the anti-causal predictor's training data; if a predictor trained purely on non-causal features still improves prompts, the benefit is not exclusively causal.
  • The error-bound argument suggests a cheap diagnostic: on any dataset with a known generative model, compute reconstruction error and counterfactual error, and if the ratio exceeds one, one of Theorem 3.2's independence or invertibility conditions is violated.
  • The same pipeline could be extended to multimodal prompts by feeding both image and text into the diffusion guidance, which the authors note as future work but do not develop.
  • If the minimal-sufficiency claim transfers to medical imaging, counterfactual explanations of this kind could reveal which visual features drive a diagnosis, but the paper's own caution about responsible deployment applies.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes DiCap, a prompt-learning framework that generates counterfactual images using classifier-guided diffusion with an anti-causal predictor and then uses these images as hard negatives in a contrastive loss to align prompts with causal features. The authors claim theoretical guarantees for identifiability and error bounds of the counterfactual generation, and report accuracy improvements over CoOp, CoCoOp, and CPL on image classification, image-text retrieval, and visual question answering, particularly on unseen classes.

Significance. If correct, the method would be a significant contribution because it couples diffusion-based counterfactual generation with prompt learning and demonstrates consistent gains across three tasks. The empirical study is broad and the reported results are uniformly positive. However, the central theoretical claim is not merely unproven; it rests on an internally inconsistent set of assumptions, so the advertised guarantee does not hold. The paper also omits the derivations promised by its 'rigorous theoretical' framing, and the connection between the theory and the actual algorithm is not established. These issues affect the core contribution of the manuscript rather than its presentation.

major comments (4)
  1. [Section 3.4, Theorem 3.2] Condition (3) of Theorem 3.2 is inconsistent with the structural causal model in Eq. (4) and Figure 3. For any invertible function g, mutual information is invariant, so I(g(x); y, n) = I(x; y, n). If g(x) is independent of (y, n), then I(x; y, n) = 0, which implies x is independent of (y, n). This contradicts Eq. (4) whenever f depends on y or n, which is exactly the setting the paper aims to model. The premise set is satisfiable only in the degenerate case where f is constant in (y, n), making the counterfactual outcome equal to the factual outcome and the identifiability conclusion vacuous. Consequently, Corollaries 3.3 and 3.4 and the abstract's claim that the approach 'guarantees the identifiability of counterfactual outcomes' are not supported.
  2. [Section 3.4] No proofs or derivations are provided for Theorem 3.2 or for Corollaries 3.3 and 3.4. The manuscript states the results and asserts that they follow from rigorous mathematical reasoning, but the actual arguments are absent. This is not a presentation issue: the central theoretical contribution of the paper consists of these statements, and without proofs or precise references the claims cannot be verified by a reader.
  3. [Section 3.4 vs. Section 3.3] The error bound in Corollary 3.4 concerns the reconstruction pair (g, h) with h(g(x), y, n) = x, but the algorithm actually generates counterfactuals through the classifier-guided sampling procedure in Eq. (2), and the relation between that procedure and the function h is never established. In addition, the distance d(·,·) used in the reconstruction-error assumption is not defined, and no continuity or Lipschitz property of h is stated to transfer a bound on d(h(g(x), y, n), x) to d(h(g(x), y_cf, n), x_cf). The claimed error transfer is therefore not a formal consequence of the stated assumptions.
  4. [Section 4.3.2, Definition 3.1] The paper claims that the generated images satisfy 'minimal sufficiency' according to Definition 3.1, but the only quantitative evaluation uses the CLD metric borrowed from Sanchez and Tsaftaris, which measures a predictor's confidence change and not the distance constraint in Eq. (3). No experiment checks whether the generated counterfactual is the smallest perturbation that changes the label. Thus the central property that the method is designed to guarantee is not actually validated.
minor comments (6)
  1. [Tables 1 and 2] The tables report accuracies and percentage improvements inline in a way that is difficult to parse, for example 'CoOp 97.93 8.04 92.82 1.87 …'; using separate columns or a delta format would improve readability.
  2. [Equation (7)] The counterfactual loss L_cf simplifies to a sum of log(1 + exp((v_cf_i - v_i) · g(w_{y_i})/τ)) after cancellation of the −log exp(v_i · g(w)/τ) terms, which is a softplus of the score difference rather than the stated two-term contrastive objective; please clarify the intended formula and its derivation.
  3. [Algorithm 1] The pseudocode does not initialize the accumulators L_basic and L_cf, and the loop body updates these variables inside the for-loop while the total loss is computed after the loop, making the intended optimization procedure ambiguous.
  4. [Throughout] The method name is spelled inconsistently as both 'DiCap' and 'Dicap'; please standardize the spelling.
  5. [Corollary 3.3] The notation 'do(y := y_cf)' is non-standard; in Pearl's do-calculus the operation should be written do(Y = y_cf), and the label variable Y should be defined consistently with Section 3.2.
  6. [Reference [65]] Reference [65] is cited as the source of the data generation process in Figure 3, but the cited paper is about contrastive learning inverting the data generating process; please verify the citation and provide the appropriate source for the graph structure.

Circularity Check

2 steps flagged · score 6.0 of 10

Theorem 3.2's identifiability conclusion restates its own condition (3); hardest-negative counterfactuals are selected and generated by the same anti-causal predictor.

  1. self definitional [Section 3.4, Theorem 3.2 (conditions (2)-(3), Eq. (4)-(5), and conclusion)]
    "Assume the following conditions hold: ... (3) The recovered latent variable is independent of the parent variables, g(x)⊥ y, n. ... Then, g(f(y, n, u_x)) = \tilde q(u_x) for some invertible function \tilde q."

    Condition (3) already states that the encoder output g(x) is independent of the label y and the non-causal factors n. Since Eq. (4) defines x = f(y,n,u_x), and since g is assumed invertible in condition (2), this independence forces g(f(y,n,u_x)) to be a function of u_x alone—exactly the theorem's conclusion. The identifiability result is therefore assumed, not derived. Moreover, with both g and f invertible, g(x) ⊥ (y,n) implies x ⊥ (y,n), contradicting the paper's own SCM in Sec. 3.2 where y and n cause x. The premise set is satisfiable only in the degenerate case where f does not depend on y or n, making Corollaries 3.3 and 3.4 vacuous for the problem the paper addresses.

  2. other [Section 3.5, "Closest as Counterfactual"]
    "Therefore, we select the class with the second closest predicted probability to the factual label y as the counterfactual label y_cf. ... Thus, the gradients related to 'tiger' is used to synthesize the counterfactual image, which subsequently functions as the most challenging negative sample in the contrastive learning task."

    The anti-causal predictor p_phi is used twice: first to select y_cf as the class p_phi already considers closest to the factual label, and then to provide the guidance gradient ∇_x log p_phi(y_cf|x) in Eq. (2) that generates the counterfactual image. The "hardest negative" is therefore hard for p_phi by construction; its difficulty does not independently certify that the image differs only in causal features. Since p_phi is a fitted classifier on the seen classes, the counterfactual set encodes p_phi's confusions, and the contrastive loss in Eq. (7) repels prompts from those images. This makes the claimed causal alignment partly defined by the same fitted predictor that generated the negatives, though the final prompt-learning evaluation is against external baselines.

full rationale

The paper's advertised theoretical guarantee is not independently derived. In Theorem 3.2, condition (3)—that g(x) is independent of y and n—is essentially the theorem's own conclusion, because Eq. (4) sets x=f(y,n,u_x) and g is assumed invertible; Corollaries 3.3 and 3.4 then inherit this assumed invariance and present it as a bound on counterfactual error. Further, condition (3) is inconsistent with condition (2) and the SCM of Sec. 3.2: invertibility plus g(x) ⊥ (y,n) forces x ⊥ (y,n), which contradicts the generative model x=f(y,n,u_x) unless f ignores y,n. The theoretical 'guarantee' is thus vacuous for the intended setting. Separately, the 'closest as counterfactual' strategy selects y_cf from the same anti-causal predictor whose gradient generates the counterfactual, so the hardest negative is hardest for that predictor by construction, tying the training signal to one fitted classifier. The empirical comparisons against CoOp, CoCoOp, and CPL are external and not fitted constants, and there is no load-bearing self-citation chain, so the circularity is concentrated in the theoretical framing and the negative-sampling loop rather than the whole experimental evaluation.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical entities, forces, or particles. Its new content is a training objective and a sampling strategy, not a new ontological entity. The analysis relies on standard SCM assumptions and strong unverified conditions on pretrained models.

free parameters (3)
  • scale s = not reported; optimal range 5 < s < 20 on ImageNet
    Controls the strength of the anti-causal gradient in Eq. 2. The paper analyzes its impact in Section 4.3.2 but does not fix a single value for all experiments.
  • loss weight lambda = not reported; best values shown only as symbols in Figure 5b
    Balances the basic contrastive loss and counterfactual loss in Eq. 8; the ablation shows stability but no precise default is given.
  • prompt length = 4 (optimal in Table 3, but main experiments do not state the default)
    Prompt length is ablated and shorter prompts are better; the main tables do not specify which length was used.
assumptions (5)
  • domain assumption The image generation process follows an SCM with x = f(y, n, u_x), with y causing causal features and n causing non-causal features.
    Section 3.2 and Figure 3 assume this structure for all datasets; it is not tested.
  • domain assumption The learned encoding g(X) is independent of y and n (Theorem 3.2, condition 3).
    Required for identifiability, but never verified for the pretrained diffusion model or CLIP encoder.
  • domain assumption The pretrained diffusion model's forward process produces independent exogenous variables that align with the SCM abduction step.
    Section 3.3 relies on this equivalence, following Sanchez and Tsaftaris [51], without verification on the actual pretrained model.
  • domain assumption The anti-causal classifier gradient provides a valid score for intervention on y.
    Used in Eq. 2 to guide generation; the gradient of a classifier trained on factual data is assumed to reflect a do-operation on the label.
  • standard math Classical Pearl SCM framework and nonlinear-ICA identifiability conditions are valid for the image setting.
    The theory imports results from Pearl [45] and Zimmermann et al. [65].

how reviews work

0 comments
Cite this review

Pith. "Pith review of Causality-aligned Prompt Learning via Diffusion-based Counterfactual Generation." pith.science (2026). https://pith.science/paper/WOZJSV7G

@misc{pith2026250719882,
  author       = {Pith},
  title        = {Pith review of: Causality-aligned Prompt Learning via Diffusion-based Counterfactual Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WOZJSV7G}},
  note         = {Machine review of arXiv:2507.19882}
}
abstract

Prompt learning has garnered attention for its efficiency over traditional model training and fine-tuning. However, existing methods, constrained by inadequate theoretical foundations, encounter difficulties in achieving causally invariant prompts, ultimately falling short of capturing robust features that generalize effectively across categories. To address these challenges, we introduce the $\textit{\textbf{DiCap}}$ model, a theoretically grounded $\textbf{Di}$ffusion-based $\textbf{C}$ounterf$\textbf{a}$ctual $\textbf{p}$rompt learning framework, which leverages a diffusion process to iteratively sample gradients from the marginal and conditional distributions of the causal model, guiding the generation of counterfactuals that satisfy the minimal sufficiency criterion. Grounded in rigorous theoretical derivations, this approach guarantees the identifiability of counterfactual outcomes while imposing strict bounds on estimation errors. We further employ a contrastive learning framework that leverages the generated counterfactuals, thereby enabling the refined extraction of prompts that are precisely aligned with the causal features of the data. Extensive experimental results demonstrate that our method performs excellently across tasks such as image classification, image-text retrieval, and visual question answering, with particularly strong advantages in unseen categories.

Figures

Figures reproduced from arXiv: 2507.19882 by the authors.

Figure 1
Figure 1. A motivating example: In camel classification tasks, [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. DiCap Method Overview: The process begins by inputting the factual image [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The Data Generation Process of the images: The [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Diffusion-Based Counterfactual Generation: During diffusion, noise is progressively added to the factual image [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 3
Figure 3. Figure 3: This parallels the forward mechanism in diffusion models, [PITH_FULL_IMAGE:figures/full_fig_p004_3.png]
Figure 5
Figure 5. Figure 5: (a) Counterfactuals enhance prompt learning. As the scale parameter [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

77 extracted references · 31 canonical work pages

  1. [51]

    Nick Pawlowski, Daniel Coelho de Castro, and Ben Glocker. 2020. Deep struc- tural causal models for tractable counterfactual inference. Advances in neural information processing systems 33 (2020), 857–869

    work page 2020

  2. [1]

    Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition . 6077–6086

    2018

  3. [2]

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision . 2425–2433

    2015

  4. [3]

    Maximilian Augustin, Valentyn Boreiko, Francesco Croce, and Matthias Hein

    work page

  5. [4]

    Maximilian Augustin, Yannic Neuhaus, and Matthias Hein. 2024. DiG-IN: Dif- fusion Guidance for Investigating Networks-Uncovering Classifier Differences Neuron Visualisations and Visual Counterfactual Explanations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 11093– 11103

    work page 2024

  6. [5]

    Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. 2014. Food-101–mining discriminative components with random forests. In Computer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part VI 13. Springer, 446–461

    2014

  7. [6]

    Patrick Chao, Patrick Blöbaum, and Shiva Prasad Kasiviswanathan. 2023. Inter- ventional and counterfactual inference with diffusion models. arXiv preprint arXiv:2302.00860 (2023)

    arXiv 2023

  8. [7]

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In Interna- tional conference on machine learning . PMLR, 1597–1607

    2020

Show all 77 references
  1. [8]

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition . Ieee, 248–255

    2009

  2. [9]

    Prafulla Dhariwal and Alexander Nichol. 2021. Diffusion models beat gans on image synthesis. Advances in neural information processing systems 34 (2021), 8780–8794

    2021

  3. [10]

    Tsu-Jui Fu, Xin Eric Wang, Matthew F Peterson, Scott T Grafton, Miguel P Eckstein, and William Yang Wang. 2020. Counterfactual vision-and-language navigation via adversarial path sampler. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, ...

    2020

  4. [11]

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh

  5. [12]

    Yash Goyal, Ziyan Wu, Jan Ernst, Dhruv Batra, Devi Parikh, and Stefan Lee

  6. [13]

    Gregory Griffin, Alex Holub, Pietro Perona, et al. 2007. Caltech-256 object cat- egory dataset. Technical Report. Technical Report 7694, California Institute of Technology Pasadena

    2007

  7. [14]

    Jason Hartford, Greg Lewis, Kevin Leyton-Brown, and Matt Taddy. 2017. Deep IV: A flexible approach for counterfactual prediction. In International Conference on Machine Learning. PMLR, 1414–1423

    2017

  8. [15]

    Xuehai He, Diji Yang, Weixi Feng, Tsu-Jui Fu, Arjun Akula, Varun Jampani, Pradyumna Narayana, Sugato Basu, William Yang Wang, and Xin Eric Wang

  9. [16]

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusion probabilistic models. Advances in neural information processing systems 33 (2020), 6840–6851

    2020

  10. [17]

    Jonathan Ho and Tim Salimans. 2022. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 (2022)

    2022 arXiv

  11. [18]

    Paul Jacob, Éloi Zablocki, Hedi Ben-Younes, Mickaël Chen, Patrick Pérez, and Matthieu Cord. 2022. STEEX: steering counterfactual explanations with semantics. In European Conference on Computer Vision . Springer, 387–403

    2022

  12. [19]

    arXiv preprint arXiv:2210.10362 (2022)

    Cpl: Counterfactual prompt learning for vision and language models. arXiv preprint arXiv:2210.10362 (2022)

    2022 arXiv

  13. [20]

    Guillaume Jeanneret, Loïc Simon, and Frédéric Jurie. 2023. Adversarial counterfac- tual visual explanations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 16425–16435

    2023

  14. [21]

    Fredrik Johansson, Uri Shalit, and David Sontag. 2016. Learning representations for counterfactual inference. In International conference on machine learning . PMLR, 3020–3029

    2016

  15. [22]

    Andrej Karpathy and Li Fei-Fei. 2015. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition . 3128–3137

    2015

  16. [23]

    Guillaume Jeanneret, Loïc Simon, and Frédéric Jurie. 2022. Diffusion models for counterfactual explanations. In Proceedings of the Asian Conference on Computer Vision. 858–876

    2022

  17. [24]

    Saeed Khorram and Li Fuxin. 2022. Cycle-consistent counterfactuals by latent transformations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10203–10212

    2022

  18. [25]

    Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020. Supervised contrastive learning. Advances in neural information processing systems 33 (2020), 18661– 18673

    2020

  19. [26]

    Siwon Kim, Jinoh Oh, Sungjin Lee, Seunghak Yu, Jaeyoung Do, and Tara Taghavi

  20. [27]

    Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. 2023. Maple: Multi-modal prompt learning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 19113–19122

    2023

  21. [28]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning . PMLR, 19730–19742

    2023

  22. [29]

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning . PMLR, 12888–12900

    2022

  23. [30]

    Jiangmeng Li, Wenyi Mo, Wenwen Qiang, Bing Su, and Changwen Zheng. 2022. Supporting vision-language model inference with causality-pruning knowledge prompt. arXiv preprint arXiv:2205.11100 (2022)

    2022 arXiv

  24. [31]

    Mingjie Li, Haokun Lin, Liang Qiu, Xiaodan Liang, Ling Chen, Abdulmotaleb Elsaddik, and Xiaojun Chang. 2025. Contrastive learning with counterfactual explanations for radiology report generation. InEuropean Conference on Computer Vision. Springer, 162–180

    2025

  25. [32]

    Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 2013. 3d object repre- sentations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops . 554–561

    2013

  26. [33]

    Xinshu Li and Lina Yao. 2022. Contrastive individual treatment effects estimation. In 2022 IEEE International Conference on Data Mining (ICDM) . IEEE, 1053–1058

    2022

  27. [34]

    Xinshu Li and Lina Yao. 2024. Distribution-conditioned adversarial variational autoencoder for valid instrumental variable generation. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 38. 13664–13672

    2024

  28. [35]

    Yang Li, Canran Xu, Guodong Long, Tao Shen, Chongyang Tao, and Jing Jiang

  29. [36]

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proce...

    2014

  30. [37]

    Xinshu Li, Mingming Gong, and Lina Yao. 2024. Self-distilled disentangled learning for counterfactual prediction. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining . 1667–1678

    2024

  31. [38]

    Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. Comput. Surveys 55, 9 (2023), 1–35

    2023

  32. [39]

    Chaochao Lu, Biwei Huang, Ke Wang, José Miguel Hernández-Lobato, Kun Zhang, and Bernhard Schölkopf. 2020. Sample-efficient reinforcement learning via counterfactual-based data augmentation. arXiv preprint arXiv:2012.09092 (2020)

    2020 arXiv

  33. [40]

    Zhiheng Lyu, Zhijing Jin, Justus Mattern, Rada Mihalcea, Mrinmaya Sachan, and Bernhard Schölkopf. 2023. Psychologically-inspired causal prompts. arXiv preprint arXiv:2305.01764 (2023)

    2023 arXiv

  34. [41]

    Arash Nasr-Esfahany and Emre Kiciman. 2023. Counterfactual (non-) identi- fiability of learned structural causal models. arXiv preprint arXiv:2301.09031 (2023)

    2023 arXiv

  35. [42]

    Maria-Elena Nilsback and Andrew Zisserman. 2008. Automated flower classifica- tion over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing . IEEE, 722–729

    2008

  36. [43]

    Jintao Liu, Zequn Zhang, Zhi Guo, Li Jin, Xiaoyu Li, Kaiwen Wei, and Xian Sun

  37. [44]

    Knowledge-Based Systems 259 (2023), 110064

    Kept: Knowledge enhanced prompt tuning for event causality identification. Knowledge-Based Systems 259 (2023), 110064

    2023

  38. [45]

    Judea Pearl. 2009. Causality. Cambridge university press

    2009

  39. [46]

    Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hock- enmaier, and Svetlana Lazebnik. 2015. Flickr30k entities: Collecting region-to- phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on comput...

    2015

  40. [47]

    Viraj Prabhu, Sriram Yenamandra, Prithvijit Chattopadhyay, and Judy Hoffman

  41. [48]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learni...

    2021

  42. [49]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2023. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. arXiv:1910.10683 [cs.LG] https://arxiv.org/abs/1910.10683

    2023 arXiv

  43. [50]

    Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. 2012. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition . IEEE, 3498–3505

    2012

  44. [52]

    Bernhard Schölkopf, Dominik Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris Mooij. 2012. On Causal and Anticausal Learning. In 29th Interna- tional Conference on Machine Learning (ICML 2012) . International Conference on Machine Learning

    2012

  45. [53]

    Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh

  46. [54]

    Jiaming Song, Chenlin Meng, and Stefano Ermon. [n. d.]. Denoising Diffusion Implicit Models. In International Conference on Learning Representations

  47. [55]

    Lance: Stress-testing visual models by generating language-guided coun- terfactual images. Advances in Neural Information Processing Systems 36 (2023), Conference acronym ’XX, June 03–05, 2018, Woodstock, NY Xinshu Li, Ruoyu Wang, Erdun Gao, Mingming Gao, & Lina Yao 25165–25184

    2023

  48. [56]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35 (2022), 24824–24837

    2022

  49. [57]

    Nina Weng, Paraskevas Pegios, Eike Petersen, Aasa Feragen, and Siavash Bigdeli

  50. [58]

    Joshua Robinson, Ching-Yao Chuang, Suvrit Sra, and Stefanie Jegelka. 2020. Contrastive learning with hard negative samples. arXiv preprint arXiv:2010.04592 (2020)

    2020 arXiv

  51. [59]

    Pedro Sanchez and Sotirios A Tsaftaris. 2022. Diffusion Causal Models for Coun- terfactual Estimation. In Causal Learning and Reasoning 2022

    2022

  52. [60]

    Mehdi Zemni, Mickaël Chen, Éloi Zablocki, Hédi Ben-Younes, Patrick Pérez, and Matthieu Cord. 2023. Octet: Object-aware counterfactual explanations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 15062–15071

    2023

  53. [61]

    Congzhi Zhang, Linhai Zhang, Jialong Wu, Deyu Zhou, and Yulan He. 2024. Causal prompting: Debiasing large language model prompting based on front- door adjustment. arXiv preprint arXiv:2403.02738 (2024)

    2024 arXiv

  54. [62]

    Kun Zhang and Aapo Hyvarinen. 2012. On the identifiability of the post-nonlinear causal model. arXiv preprint arXiv:1205.2599 (2012)

    2012 arXiv

  55. [63]

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. 2022. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 16816–16825

    2022

  56. [64]

    Adith Swaminathan and Thorsten Joachims. 2015. The self-normalized estimator for counterfactual learning. advances in neural information processing systems 28 (2015)

    2015

  57. [65]

    Roland S Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. 2021. Contrastive learning inverts the data generating process. In International Conference on Machine Learning . PMLR, 12979–12990

    2021

  58. [68]

    Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba

  59. [70]

    Mengyue Yang, Furui Liu, Zhitang Chen, Xinwei Shen, Jianye Hao, and Jun Wang

  60. [76]

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. 2022. Learning to prompt for vision-language models. International Journal of Computer Vision 130, 9 (2022), 2337–2348

    2022

  61. [2010]

    In 2010 IEEE computer society conference on computer vision and pattern recognition

    Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition . IEEE, 3485–3492

    2010

  62. [2017]

    In Proceedings of the IEEE conference on computer vision and pattern recognition

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition . 6904–6913

  63. [2019]

    In International Conference on Machine Learning

    Counterfactual visual explanations. In International Conference on Machine Learning. PMLR, 2376–2384

  64. [2020]

    arXiv preprint arXiv:2010.15980 (2020)

    Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.15980 (2020)

    2020 arXiv

  65. [2021]

    In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Causalvae: Disentangled representation learning via neural structural causal models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 9593–9602

  66. [2022]

    Advances in Neural Informa- tion Processing Systems 35 (2022), 364–377

    Diffusion visual counterfactual explanations. Advances in Neural Informa- tion Processing Systems 35 (2022), 364–377

    2022

  67. [2023]

    In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Grounding counterfactual explanation of image classifiers to textual concept space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10942–10950

  68. [2024]

    In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)

    CCPrefix: Counterfactual Contrastive Prefix-Tuning for Many-Class Clas- sification. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) . 2977–2988

  69. [2025]

    In European Conference on Computer Vision

    Fast diffusion-based counterfactuals for shortcut removal and generation. In European Conference on Computer Vision . Springer, 338–357

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.