Pith. sign in

REVIEW 2 major objections 4 minor 47 references

Avoiding Leakage Poisoning: Concept Interventions Under Distribution Shifts

T0 review · 2 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Bypass concept models lose the benefit of human concept corrections when test inputs shift out of distribution; the paper proposes MixCEM, which gates leaked residuals by concept-prediction entropy and keeps interventions effective.

desk verdict A well-executed empirical paper identifying a real failure mode in concept-based models; the headline entropy-gating mechanism is not isolated by ablation, but the empirical contribution stands. read the letter →

arxiv 2504.17921 v3 pith:PV5PWMLA submitted 2025-04-24 cs.LG cs.AIcs.CRcs.HC

Mateo Espinosa Zarlenga , Gabriele Dominici , Pietro Barbiero , Zohreh Shams , Mateja Jamnik This is my paper · ORCID
classification cs.LGcs.AIcs.CRcs.HC
keywords conceptbottleneckmodelsleakagepoisoninginterventionsdistributionshiftout-of-distributiondetectionembeddingsinterpretablemachinelearningentropygating
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that concept-based models which let task information bypass the concept bottleneck become unreliable exactly when they are most needed: when a human expert corrects mispredicted concepts on out-of-distribution inputs. It names this failure "leakage poisoning": the bypassed information persists in the bottleneck even after an intervention, and because that leaked information is itself out-of-distribution, corrections no longer pull the prediction back toward the right answer. To fix it, the paper proposes MixCEM, which represents each concept as a global embedding plus a sample-specific residual and scales the residual down by the entropy of the concept prediction, so OOD samples fall back to clean global embeddings. The authors show, on complete and concept-incomplete tasks, that MixCEM keeps task and concept accuracy competitive while making interventions improve accuracy for both in-distribution and OOD samples. A sympathetic reader would care because concept interventions are the main practical mechanism for humans to steer an interpretable model at test time, and existing bypass models silently break that mechanism under shift.

What carries the argument

The load-bearing object is the entropy-gated contextual concept embedding, defined as $c_i^{(\pm)} = \bar c_i^{(\pm)} + (1-H(\hat p_i)) r_i^{(\pm)}(x)$, where $\bar c_i^{(\pm)}$ are learnable global embeddings for concept $i$ active or inactive, $r_i^{(\pm)}(x)$ are sample-specific residuals produced from the backbone, and $H(\hat p_i)$ is the entropy of the predicted concept probability. Gating turns uncertainty into a switch: confident in-distribution concepts keep the residual and therefore preserve completeness-agnosticism, while uncertain OOD concepts drop it and therefore avoid leakage poisoning. The training objective reinforces this by adding the prior-error term $L_{task}(y, f(\bar c))$, so predictions from the global embeddings alone are accurate.

What would settle it

Run a trained MixCEM on an OOD shift such as 10% salt-and-pepper noise and measure the entropy of its concept predictions: if most mispredicted concepts have low entropy, the gate stays open and MixCEM's all-intervened OOD accuracy should resemble CEM's rather than stay near the Bayes classifier. Alternatively, adversarially perturb inputs to force confidently wrong concept predictions and check whether leakage poisoning returns.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is a previously unnamed failure mode and a mechanism that avoids it. Bypass-based concept models achieve completeness-agnosticism by leaking input information around the concept bottleneck; under distribution shift, that leaked information is no longer in-distribution, so even after all concepts are corrected the bottleneck remains OOD and intervention accuracy collapses. MixCEM avoids this by decomposing each concept embedding into a global, sample-agnostic part and a residual, sample-specific part, and forming the bottleneck as $\hat c_i = \hat p_i c_i^{(+)} + (1-\hat p_i)c_i^{(-)}$ with $c_i^{(\pm)} = \bar c_i^{(\pm)} + (1 - H(\hat p_i)) r_i^{(\pm)}(x)$. Because $H(\hat p_i)$ is higher for OOD concept predictions after Platt calibration, the residual contribution vanishes on OOD inputs and the bottleneck reduces to global embeddings, which remain within the training distribution and make interventions effective. The paper also adds a prior-error training term that forces the global embeddings alone to predict the task well, which acts as an implicit intervention-aware regularizer.

Load-bearing premise

The gating mechanism assumes that, after Platt calibration, concept-prediction entropy is a reliable indicator that the input is out-of-distribution, so scaling residuals by $1-H(\hat p_i)$ removes poisonous leakage.

Editorial extensions

If this is right

  • If leakage poisoning is real, every current bypass-based concept model deployed under distribution shift can fail to benefit from, or even be harmed by, human concept corrections; MixCEM-style gating is needed for trustworthy interventions.
  • MixCEM is the only completeness-agnostic baseline whose OOD intervention curves stay at or above the Bayes-classifier lower bound across most tasks, so concept-incomplete models can remain safely intervenable under shift.
  • Entropy-gated residuals let the same model be complete-agnostic on in-distribution data and safely intervenable on OOD data, dissolving the previously assumed trade-off between the two objectives.
  • Combining MixCEM with intervention-aware training yields learned intervention policies whose OOD curves beat random-policy baselines, suggesting the design composes with existing intervention-learning machinery.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The entropy gate is effectively a built-in OOD detector; replacing $1-H(\hat p_i)$ with a dedicated OOD score such as an energy score or feature distance should block leakage similarly, at the cost of training an extra detector.
  • Because MixCEM's global embeddings are constant across samples, the model is biased toward concept-only prediction, which likely explains its spurious-correlation robustness and suggests the decomposition could be used as a regularizer in non-conceptual architectures.
  • A testable extension is to apply the same global-plus-gated-residual split to energy-based concept models and other embedding-based bottlenecks, generalizing the mechanism beyond the architectures tested here.
  • The reported gains depend on entropy staying calibrated under shift; on shifts that produce confident wrong concept predictions, the gate would stay open and leakage poisoning would return unless a stronger OOD indicator is added.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper studies concept-based models (CMs) under distribution shifts and introduces the notion of 'leakage poisoning': when a CM uses residual or bypass mechanisms to overcome incomplete concept annotations, the leaked information becomes out-of-distribution for OOD inputs, so concept interventions no longer improve task accuracy. To address this, the authors propose MixCEM, an architecture that combines learnable global concept embeddings with input-dependent residual embeddings and scales the residuals by (1-H(p_hat_i)), where H is the entropy of the concept prediction. Training adds a prior-error loss on a bottleneck built from global embeddings alone, plus residual dropout, Monte Carlo averaging, and Platt scaling. The paper reports intervention experiments on CUB, AwA2, CIFAR-10, CelebA, TravelingBirds, and an MNIST-to-SVHN shift, comparing MixCEM with Vanilla CBM, Hybrid CBM, CEM, IntCEM, ProbCBM, P-CBM, and Residual P-CBM.

Significance. If the claims hold, the paper identifies a practically important failure mode for concept-based interpretable models and proposes a sensible architectural response. The empirical evaluation is extensive: six datasets, concept-complete and concept-incomplete settings, multiple distribution-shift types, three-seed means with standard deviations, and comparison against strong external baselines. The release of code and configurations is a further strength. However, the paper's central mechanistic claim--that the entropy-gated residual scaling is what avoids leakage poisoning--is not supported by the experiments as reported, because no ablation isolates this component from the simultaneously introduced prior-error loss, residual dropout, and Platt calibration. The contribution is therefore currently a promising architecture plus an unsupported attribution of the mechanism.

major comments (2)
  1. [Section 4, "Entropy-based Gating"; Section 5.2, Figure 4; Appendix J] The central claim is that MixCEM 'learns to dynamically exploit leaked information missing from its concepts only when this information is in-distribution' (abstract, contribution 3), and the mechanism is the factor (1-H(p_hat_i)) in the contextual embedding definition c_i^(+/-) := c_bar_i^(+/-) + (1-H(p_hat_i)) r_i^(+/-)(x). In every experiment, this factor is confounded with three other changes introduced simultaneously: the prior-error loss L_task(y, f(c_bar)), residual dropout with Monte Carlo averaging, and Platt scaling. Appendix J ablates lambda_p, p_drop, lambda_c, E_cal, and p_int individually, but no experiment replaces the gating factor by a constant (e.g., setting (1-H) to 1) while keeping the rest of MixCEM fixed. The t-SNE and entropy histograms in Figure 2 show a correlation between OOD inputs and higher concept entropy, but correlation is not causal evidence for the gate. Please add a gate-off ablation (residual scaling fixed to 1) and report ID/OOD intervention curves and all-intervened accuracies, at least on CUB-Incomplete and AwA2-Incomplete, so that the attribution of the gains to the entropy gate is actually testable. Without this ablation, the empirical curves may stand but the stated novelty and title are unsupported.
  2. [Section 4 and Appendix A; Section 6, Limitations] The gate relies on concept-prediction entropy being a reliable and calibrated OOD indicator. Platt scaling is fit on an in-distribution validation set only, and the paper does not test a shift in which concept predictions are confidently wrong. The first limitation acknowledged in Section 6 is exactly such a scenario: when a distribution shift makes a concept incomprehensible to an expert, MixCEM 'may fail to completely block leakage poisoning,' because an intervention cannot be performed. The paper should either report an experiment on a shift where model concept predictions are confidently incorrect (e.g., a semantic concept shift rather than pixel-level noise), or temper the statement in Section 4 that MixCEM drops residuals for OOD samples. Reporting calibration error or the entropy distribution of correct vs. incorrect OOD concept predictions would make the gate's operating assumption visible.
minor comments (4)
  1. [Appendix B, Eq. (3)] The factorization P(Y|C*) = P(Y|C,r(X),C_bar) P(Y|c_bar) is presented as though it is a probability factorization, but the product of two conditional distributions for the same target Y is not a normalized probability and does not follow from the graphical model in Figure 7. The claim that MixCEM's objective 'naturally arises as the MLE' is therefore not established as written; please either provide a proper derivation or present the objective as a regularized multi-task loss.
  2. [Abstract and Introduction, Contribution 3] The wording 'outperform strong baselines by significantly improving their accuracy ... in the presence and absence of concept interventions' is stronger than Table 1 supports: for example, on AwA2 the ID task accuracy of MixCEM is 89.94 vs. CEM's 91.07, and on AwA2 OOD unintervened accuracy MixCEM is 17.75 vs. CEM's 20.22. A more quantitative summary, noting where MixCEM is not the best, would be more accurate.
  3. [Table 6 and Figure 4] The Bayes Classifier rows in Table 6 are identical for ID and OOD by construction, because the approximated Bayes classifier takes only ground-truth concept labels as input and ignores the corrupted image. This is not explained in the caption or text and may confuse readers; please add a short note.
  4. [Figure 2] The right panel shows unnormalized count histograms of concept-prediction entropies for ID and OOD samples. Normalizing the histograms (or plotting densities) would make the claimed separation between ID and OOD entropy distributions easier to assess.

Circularity Check

1 steps flagged · score 2.0 of 10

One minor self-definitional step in the appendix's MLE derivation; the empirical MixCEM claims are otherwise self-contained and evaluated against external benchmarks.

  1. self definitional [Appendix B, 'Maximum likelihood of MixCEM', Eq. (3) and the final objective; referenced in Section 4 ('In Appendix B, we prove ...')]
    "Here, we make a practical modelling assumption. ... we factorise the likelihood of the downstream task Y with respect to C* as: P(Y |C*) = P (Y |C,r (X), ¯C) P (Y | ¯c) ... Plugging in MixCEM’s likelihoods for the task and concept predictions derived in Section 4 and minimising the negative log-likelihood, we get a final training objective: ... L_t(y,f(g(x))) + λ_p L_t(y,f(¯c)) + λ_c BCE(c, p_hat)."

    The graphical model (Figure 7) does not imply the factorization P(Y|C*) = P(Y|C,r(X),bar C) P(Y|bar c); the second factor is introduced as a 'practical modelling assumption' precisely to create the prior-error term L_task(y,f(bar c)). Taking logs turns that assumed factor into log P(y|bar c), so the 'derived' MLE objective is the same as the assumed factorization up to weighting hyperparameters. The appendix thus assumes the characteristic prior-error term it claims to derive, making the theoretical justification self-definitional. This is not load-bearing for the empirical intervention curves, which are measured against external benchmarks and baselines, but it is a genuine reduction-by-construction in the paper's own derivation chain.

full rationale

The paper's central claims (MixCEM improves ID and OOD intervention accuracy on CUB, AwA2, CelebA, CIFAR-10, TravelingBirds, and SVHN) are evaluated against external benchmarks and multiple outside baselines. The model is not constructed to teach the test set: the Bayes-classifier bound is an independently trained masked MLP on concept labels, and OOD test sets are defined by external corruptions or domain shifts, not by the entropy values used in the gate. Platt scaling is fit on an ID validation set to concept labels, not to intervention outcomes, and no fitted parameter is renamed as a prediction. The entropy-gated residual scaling is an architectural mechanism whose effectiveness is an empirical question, and the paper candidly states a failure mode (incomprehensible concepts may not be blockable), which is inconsistent with the success being true by definition. The only self-definitional step is the appendix's MLE derivation, where the prior-error term is inserted by an explicit factorization assumption and then recovered in the objective; this is a theoretical framing that does not underwrite the empirical comparisons. Therefore, no significant circularity infects the central derivation, and the score reflects only the minor appendix-level reduction.

Assumptions & free parameters 4 free parameters · 5 assumptions · 1 invented entities

The central claim depends on several modelling choices. The free parameters are hyperparameters selected by validation (lambda_p, p_drop, E_cal) plus the per-concept Platt scaling parameters fit on the validation set. The key axioms are the standard complete-concept-set assumption of concept-based models, the explicit factorization in the appendix derivation that creates the prior-error term, the empirical claim that concept-prediction entropy signals OOD inputs, and the assumption that interventions set concepts to ground truth. The paper introduces one latent construct, the residual variables R, as a modelling device with no independent falsifiable handle.

free parameters (4)
  • lambda_p (prior loss weight) = 1 for most tasks; 0.1 for CIFAR10
    Controls the prior-error term L_task(y, f(c_bar)) that trains the label predictor from global embeddings with ground-truth concepts; selected per task by validation on the intervention-curve area (Appendix D.4).
  • p_drop (residual dropout probability) = 0.1, 0.5, or 0.9 per task
    Probability of zeroing residual embeddings during training; affects both ID accuracy and OOD intervenability; selected by validation (Appendix D.4, ablations in Appendix J.3).
  • E_cal (Platt scaling epochs) = 30 except 0 for CelebA
    Number of epochs for calibrating concept probabilities on validation data; affects entropy gating effectiveness (Appendix A, J.4).
  • Platt scaling parameters a_i, b_i per concept = not reported, per concept
    Learned on validation set via maximum likelihood to calibrate logits before entropy gating (Appendix A). These are fitted values used at inference.
assumptions (5)
  • domain assumption Existence of a complete concept set C* that generates inputs X and labels Y
    Standard assumption in concept-based interpretability (Figure 6a); invoked in the graphical model derivation in Appendix B.
  • ad hoc to paper Factorization P(Y|C*) = P(Y|C,r(X),C_bar) P(Y|c_bar)
    Equation 3 in Appendix B; a 'practical modelling assumption' chosen to yield the prior-error training term lambda_p L_task(y, f(c_bar)). Not derived from the graphical model.
  • domain assumption Concept prediction entropy H(p_hat_i) is a reliable indicator of OOD inputs
    Used for gating in Section 4 (c_i := c_bar_i + (1 - H(p_hat_i)) r_i(x)); empirically observed in Figure 2 (right) but not guaranteed; can fail when concepts themselves are OOD or already uncertain for ID data.
  • domain assumption Interventions set concept values to their ground-truth labels
    Standard setup in concept bottleneck literature, used throughout evaluation.
  • domain assumption The masked MLP trained on concept labels approximates the Bayes classifier P(y|c_S)
    Appendix D.5; used to define bounded intervenability; the approximation is only as good as the MLP fit.
invented entities (1)
  • Residual variables R
    purpose: Latent factor in MixCEM's graphical model (Figure 7) representing task-relevant information missing from training concepts; implemented as deterministic residual embeddings r(x) in Section 4.
    Used to derive the training objective in Appendix B; a modeling construct with no falsifiable handle outside this paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Avoiding Leakage Poisoning: Concept Interventions Under Distribution Shifts." pith.science (2026). https://pith.science/paper/PV5PWMLA

@misc{pith2026250417921,
  author       = {Pith},
  title        = {Pith review of: Avoiding Leakage Poisoning: Concept Interventions Under Distribution Shifts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PV5PWMLA}},
  note         = {Machine review of arXiv:2504.17921}
}
read the original abstract

In this paper, we investigate how concept-based models (CMs) respond to out-of-distribution (OOD) inputs. CMs are interpretable neural architectures that first predict a set of high-level concepts (e.g., stripes, black) and then predict a task label from those concepts. In particular, we study the impact of concept interventions (i.e., operations where a human expert corrects a CM's mispredicted concepts at test time) on CMs' task predictions when inputs are OOD. Our analysis reveals a weakness in current state-of-the-art CMs, which we term leakage poisoning, that prevents them from properly improving their accuracy when intervened on for OOD inputs. To address this, we introduce MixCEM, a new CM that learns to dynamically exploit leaked information missing from its concepts only when this information is in-distribution. Our results across tasks with and without complete sets of concept annotations demonstrate that MixCEMs outperform strong baselines by significantly improving their accuracy for both in-distribution and OOD samples in the presence and absence of concept interventions.

Figures

Figures reproduced from arXiv: 2504.17921 by the authors.

Figure 1
Figure 1. (Left) A concept intervention on a CBM triggering a prediction update. (Right) Task accuracy as concepts are intervened on in a concept-incomplete task. When intervening on ID samples (solid), bypass-enabling models (e.g., CEMs) overcome the “incompleteness gap.” However, for OOD samples (dashed), the same models underperform due to “leakage poisoning.” MixCEM overcomes the incompleteness gap and leakage poisoning, … view at source ↗
Figure 2
Figure 2. CEM concept bottlenecks and predicted concept en￾tropies for ID and noisy (OOD) test CUB samples. (Left and centre) t-SNE projections of CEM’s bottlenecks before and after all concepts are intervened on. (Right) Distribution of predicted concept entropies for all concepts. concept-incomplete tasks. However, we also observe that these approaches struggle to properly incorporate interven￾tions for OOD inputs, even whe… view at source ↗
Figure 3
Figure 3. Given x, a MixCEM predicts concepts pˆ and task labels yˆ. It achieves this by (1) learning global concept embeddings c¯ (+) , c¯ (−) ∈ R k×m and residual embeddings r (+)(x), r(−) (x) ∈ R k×m for each training concept ci, (2) using these embeddings to estimate pˆi = P(ci = 1 | x, C¯) and to construct contextual concept embeddings (c (+) i , c (−) i ), (3) mixing contextual embeddings to produce a single embedding c… view at source ↗
Figures from the paper (18 more)
Figure 4
Figure 4. Figure 4: Task accuracy as we intervene on concepts, selected at random, for ID (top) and OOD (bottom) test samples. OOD samples have a form of salt & pepper noise injected into at most 10% of their channels (similar results on other forms of distribution shifts can be seen in A…
Figure 5
Figure 5. Figure 5: (Left) CUB-Incomplete intervention curves. Test samples are perturbed by adding “Salt & Pepper” noise with increasing levels (% pixels corrupted). (Centre) TravelingBirds intervention curves. We show results on a spuriously correlated validation set (dashed) and a test…
Figure 6
Figure 6. Figure 6: Assumptions underlying CMs. (a) Generative process assumed by CMs, and (b) graphical model underlying CBMs, where darkened nodes represent observed variables (i.e., inputs). The implied generative process for CMs assumes the existence of a set of complete concepts C ∗ …
Figure 7
Figure 7. Figure 7: MixCEM’s graphical model. MixCEM assumes that a complete set of concepts C ∗ can be constructed from (1) a set of concepts aligned with the training concept annotations C, that follow some prior C¯, and (2) a set of residual variables R that describe all information mi…
Figure 8
Figure 8. Figure 8: Randomly selected training and test samples of TravelingBirds for the class “Yellow-headed Blackbird”. Notice that training samples all have “aircraft-carrier” backgrounds while the testing samples have “sauna” backgrounds. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: Task accuracy of several Vanilla CBM variants as we intervene on concepts, selected at random, for ID (top) and OOD (bottom) test samples. We use the same setup as the experiments described in [PITH_FULL_IMAGE:figures/full_fig_p022_9.png]
Figure 10
Figure 10. Figure 10: Examples of random images in CUB with our form of “Salt & Pepper” noise as we vary the noise’s strength level. H. Interventions on Different Forms of Distribution Shifts In this section, we explore different forms of distribution shifts beyond those studied in Section…
Figure 11
Figure 11. Figure 11: Task accuracy as we intervene on concepts, selected at random, for OOD test samples on CUB-Incomplete and our AwA2 variants. Out-of-distribution samples are generated by applying different forms of visual transformations to the test set (shift type shown on the y-axis…
Figure 12
Figure 12. Figure 12: Task accuracy as we intervene on concepts, selected at random, for models trained on the digit addition task. Models are trained using MNIST digits. However, when we evaluate OOD interventions, we use test samples whose digits are drawn from the SVHN dataset. I. Exten…
Figure 13
Figure 13. Figure 13: OOD task accuracy vs intervention curves for all baselines in the CUB (top) and CUB-Incomplete tasks. Test samples are perturbed by adding “Salt & Pepper” noise with increasing levels (% pixels corrupted). 0 20 40 60 80 100 0 20 40 60 80 100 ID Task Accuracy (%) Trave…
Figure 14
Figure 14. Figure 14: TravelingBirds (left) and TravelingBirds-Incomplete (right) intervention curves. We show results on a spuri￾ously correlated ID validation set (top) and on an OOD test set without the spurious correlation (bottom). J. Hyperparameter Recommendations and Ablation Studie…
Figure 15
Figure 15. Figure 15: Ablation study for λc in a smaller version of CUB-Incomplete. On top of each subplot, we indicate whether we show results for an ID test set or an OOD test set (generated using 10% salt & pepper noise). The right-most two plots show the task accuracy when all concepts…
Figure 16
Figure 16. Figure 16: Ablation study for λp in a smaller version of CUB-Incomplete. On top of each subplot, we indicate whether we show results for an ID test set or an OOD test set (generated using 10% salt & pepper noise). The right-most two plots show the task accuracy when all concepts…
Figure 17
Figure 17. Figure 17: Ablation study for pdrop in a smaller version of CUB-Incomplete. On top of each subplot, we indicate whether we show results for an ID test set or an OOD test set (generated using 10% salt & pepper noise). The right-most two plots show the task accuracy when all conce…
Figure 18
Figure 18. Figure 18: Ablation study for MixCEM’s residual dropout probability pdrop hyperparameter in CelebA. Notice that in this task, there is a significant improvement in OOD intervenability when pdrop is greater than 0. We mark in red and bold the model corresponding to the hyperparam…
Figure 19
Figure 19. Figure 19: Task accuracy of MixCEMs with and without Platt scaling as we intervene on concepts, selected at random, for ID (top) and OOD (bottom) test samples. We use the same setup as the experiments described in [PITH_FULL_IMAGE:figures/full_fig_p029_19.png]
Figure 20
Figure 20. Figure 20: Task accuracy of MixCEMs with and without randomly intervening at training time with probability pint (i.e., using RandInt). We use the same setup as the experiments described in [PITH_FULL_IMAGE:figures/full_fig_p029_20.png]
Figure 21
Figure 21. Figure 21: Effect of integrating IntCEM’s training procedure into MixCEM. Intervention curves for ID (top) and OOD (bottom) test sets across our CUB-based tasks for learnt policies across intervention-aware models. As a baseline, we include intervention curves for CEM and MixCEM…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 28 canonical work pages

  1. [4]

    J.5. Effect of Training Intervention Probability (pint) Finally, we study the effect of training-time interventions on MixCEM’s performance across ID and OOD tasks (so-called RandInt (Espinosa Zarlenga et al., 2022)). Our results, shown in Figure 20 suggest that randomly intervening on concepts during training withpint = 0.25 is generally beneficial for I...

    work page 2022

  2. [5]

    and Dietterich, T

    Hendrycks, D. and Dietterich, T. Benchmarking neural network robustness to common corruptions and perturba- tions. arXiv preprint arXiv:1903.12261,

    arXiv 1903

  3. [8]

    T., D ¨ahne, S., Erhan, D., and Kim, B

    Kindermans, P.-J., Hooker, S., Adebayo, J., Alber, M., Sch¨utt, K. T., D ¨ahne, S., Erhan, D., and Kim, B. The (un) reliability of saliency methods. arXiv preprint arXiv:1711.00867,

    arXiv

  4. [9]

    Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980,

    arXiv

  5. [12]

    Salt & Pepper

    Task accuracy as we intervene on concepts, selected at random, for models trained on the digit addition task. Models are trained using MNIST digits. However, when we evaluate OOD interventions, we use test samples whose digits are drawn from theSVHN dataset. I. Extended Robustness Experiments We complement our results in Section 5.3 by showing extended ve...

    work page 2020

  6. [14]

    We show results on a spuri- ously correlated ID validation set (top) and on an OOD test set without the spurious correlation (bottom)

    TravelingBirds (left) and TravelingBirds-Incomplete (right) intervention curves. We show results on a spuri- ously correlated ID validation set (top) and on an OOD test set without the spurious correlation (bottom). J. Hyperparameter Recommendations and Ablation Studies In this section, we thoroughly examine the impact of MixCEM’s hyperparameters on its p...

    work page 2020

  7. [15]

    Pytorch: An imperative style, high-performance deep learning library

    Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, high-performance deep learning library. arXiv preprint arXiv:1912.01703,

    arXiv 1912

  8. [16]

    W., Hashimoto, T

    Sagawa, S., Koh, P. W., Hashimoto, T. B., and Liang, P. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case gener- alization. arXiv preprint arXiv:1911.08731,

    arXiv 1911

Show all 47 references
  1. [17]

    R., Sheth, I., Farahnak, F., Kahou, S

    Sevyeri, L. R., Sheth, I., Farahnak, F., Kahou, S. E., and Enger, S. A. Transparent anomaly detection via concept- based explanations. arXiv preprint arXiv:2310.10702,

    arXiv

  2. [18]

    Shen, M. W. Trust in AI: Interpretability is not necessary or sufficient, while black-box interaction is necessary and sufficient. arXiv preprint arXiv:2202.05302,

    arXiv

  3. [20]

    A closer look at the intervention procedure of concept bottleneck models

    Shin, S., Jo, Y ., Ahn, S., and Lee, N. A closer look at the intervention procedure of concept bottleneck models. arXiv preprint arXiv:2302.14260,

    arXiv

  4. [21]

    Dropout: a simple way to prevent neural networks from overfitting

    Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929–1958,

    1929

  5. [24]

    The caltech-ucsd birds-200-2011 dataset

    Wah, C., Branson, S., Welinder, P., Perona, P., and Belongie, S. The caltech-ucsd birds-200-2011 dataset. Technical Report CNS-TR-2011-001, California Institute of Tech- nology,

    2011

  6. [26]

    Energy-based concept bottleneck models: unifying prediction, con- cept intervention, and conditional interpretations

    Xu, X., Qin, Y ., Mi, L., Wang, H., and Li, X. Energy-based concept bottleneck models: unifying prediction, con- cept intervention, and conditional interpretations. arXiv preprint arXiv:2401.14142,

    arXiv

  7. [27]

    Post-hoc con- cept bottleneck models

    Yuksekgonul, M., Wang, M., and Zou, J. Post-hoc con- cept bottleneck models. In ICLR 2022 Workshop on PAIR2Struct: Privacy, Accountability, Interpretability, Robustness, Reasoning on Structured Data,

    2022

  8. [30]

    black nape

    RGB image of a bird (normalised in [0, 1]), annotated with one of 200 bird species. Here, each image has 312 binary attribute annotations (e.g., “black nape ”, “yellow wing colour”, etc.). We construct a set of 112 binary concepts following the selection of attributes used by ...

    2020

  9. [32]

    aeroplanes

    normalised RGB image that can be one out of 10 object types (e.g., “aeroplanes”, “cars”, “birds”, “cats”, etc.). As done by Marcinkeviˇcs et al. (2024) and Vandenhirtz et al. (2024), we annotate all samples in this dataset with 143 textual concepts whose semantics were obtaine...

    2024

  10. [33]

    In contrast to how Marcinkeviˇcs et al

    similarity score between each image and each concept’s textual description. In contrast to how Marcinkeviˇcs et al. (2024) and Vandenhirtz et al. (2024) binarise these concept scores, however, we do not use a zero-shot CLIP classifier selecting between a concept description an...

    2024

  11. [34]

    seashores

    and for consistency with previous works, our training set here is formed by randomly subsampling the original CelebA’s training set to a 12th of its size. TravelingBirds and TravelingBirds-Incomplete The TravelingBirds task and its incomplete version are variations of their re...

    2020

  12. [35]

    Yellow-headed Blackbird

    Randomly selected training and test samples of TravelingBirds for the class “Yellow-headed Blackbird”. Notice that training samples all have “aircraft-carrier” backgrounds while the testing samples have “sauna” backgrounds. 18 Avoiding Leakage Poisoning: Concept Interventions ...

    2020

  13. [37]

    Similarly, we use weight decay 1× 10−6, a learning rate 10 times smaller for the non-pretrained weights, and clip gradient norms to 2 as the authors do in their official code base2

    optimiser, (2) use a starting learning rate of 0.001 (except for CIFAR10 where we increase it to 0.01 as otherwise the model severely underperformed), (3) fix the number of training and inference samples to 50, (4) intervene on concepts during training with probabilitypint = 0...

    2023

  14. [38]

    When fine-tuning P-CBMs’s sparse linear classifier, we fine-tune the complexity regulariserλcomplex and fix the elastic net’s ℓ1 ratio to be 0.1

    for conceptci using the vector perpendicular to the decision boundary of a linear SVM, with ℓ2 penaltyC = 1, trained to predict concept ci from the activations of the second-to-last layer of the black box DNN. When fine-tuning P-CBMs’s sparse linear classifier, we fine-tune th...

    2020

  15. [39]

    were by far the most often selected hyperparameters, we recommend using these as the default values of MixCEM’s hyperparameters if there are no resources for fine-tuning its hyperparameters. D.5. Bayes Classifier To determine whether or not all baselines achieve bounded interv...

    2020

  16. [40]

    Salt & Pepper

    Efficiency study showing the inference time per sample (in milliseconds) for all baselines in CUB. Vanilla CBM Hybrid CBM ProbCBM P-CBM Residual P-CBM CEM IntCEM MixCEM Time per Sample (ms) 1.386 1.394 5.687 1.426 1.433 1.437 1.431 1.497 G. Image Noising Details Across all of ...

    2018

  17. [45]

    Random Policy

    Effect of integrating IntCEM’s training procedure into MixCEM. Intervention curves for ID (top) and OOD (bottom) test sets across our CUB-based tasks for learnt policies across intervention-aware models. As a baseline, we include intervention curves for CEM and MixCEM using a ...

    2022

  18. [46]

    (2024) and Marcinkeviˇcs et al

    Finally, for our CIFAR10 and AwA2 loaders, we got inspiration from the public implementation of these loaders by Vandenhirtz et al. (2024) and Marcinkeviˇcs et al. (2024), respectively. Our experiments were run on PyTorch 1.11.0 (Paszke et al.,

    2024

  19. [47]

    For our plots, we used matplotlib 3.5.1 (Hunter,

    and facilitated by PyTorch Lightning 1.9.5 (Falcon, 2019). For our plots, we used matplotlib 3.5.1 (Hunter,

    2019

  20. [100]

    ProbCBM We attempt to closely follow the same hyperparameters for ProbCBMs used in the original work by Kim et al

    for CelebA. ProbCBM We attempt to closely follow the same hyperparameters for ProbCBMs used in the original work by Kim et al. (2023). As such, we (1) always use an Adam (Kingma & Ba,

    2023

  21. [200]

    CEM When training CEMs, we intervene on a concept with probability pint = 0.25 (as suggested by the authors (Es- pinosa Zarlenga et al., 2022))

    Similarly, we selectedλc = 10 for CelebA, AwA2, and AwA2-Incomplete,λc = 1 for CUB-Incomplete and CIFAR10, andλc = 5 for CUB. CEM When training CEMs, we intervene on a concept with probability pint = 0.25 (as suggested by the authors (Es- pinosa Zarlenga et al., 2022)). In thi...

    2022

  22. [299]

    Each sample in this task corresponds to a (3× 299×

    200 22 7 CUB The CUB bird classification image task is constructed from the Caltech-UCSD Birds-200-2011 dataset (Wah et al., 2011). Each sample in this task corresponds to a (3× 299×

    2011

  23. [1999]

    In this work, we apply a common adaptation of Platt Scaling by Guo et al

    is a post-hoc calibration method used to transform the outputs of a probabilistic classifier into well-calibrated probabilities (i.e., probabilities that better represent the model’s true uncertainty). In this work, we apply a common adaptation of Platt Scaling by Guo et al. (...

    2017

  24. [2007]

    Kazhdan, D., Dimanov, B., Jamnik, M., Li`o, P., and Weller, A

    doi: 10.1109/MCSE.2007.55. Kazhdan, D., Dimanov, B., Jamnik, M., Li`o, P., and Weller, A. Now you see me (cme): concept-based model extrac- tion. arXiv preprint arXiv:2010.13233,

    2007 arXiv

  25. [2008]

    Vandenhirtz, M., Laguna, S., Marcinkeviˇcs, R., and V ogt, J. E. Stochastic concept bottleneck models.arXiv preprint arXiv:2406.19272,

    arXiv

  26. [2009]

    Large-scale celeb- faces attributes (celeba) dataset

    Liu, Z., Luo, P., Wang, X., and Tang, X. Large-scale celeb- faces attributes (celeba) dataset. Retrieved August, 15 (2018):11,

    2018

  27. [2011]

    A., Kazhdan, D., Dimanov, B., and Jamnik, M

    Wijaya, M. A., Kazhdan, D., Dimanov, B., and Jamnik, M. Failing conceptually: Concept-based explanations of dataset shift. arXiv preprint arXiv:2104.08952,

    arXiv

  28. [2012]

    form each training sample, and the task is to predict whether all digits add to more than 25% of the maximum sum. We provide the identity of five digits as training concepts (i.e., it is an incomplete task), and at test time, we swap MNIST digits for real-world sampled digits ...

    2011

  29. [2013]

    C., Barbiero, P., Kazhdan, D., Siciliano, F., Ciravegna, G., Silvestri, F., Jamnik, M., and Lio, P

    Magister, L. C., Barbiero, P., Kazhdan, D., Siciliano, F., Ciravegna, G., Silvestri, F., Jamnik, M., and Lio, P. En- coding concepts in graph neural networks. arXiv preprint arXiv:2207.13586,

    arXiv

  30. [2014]

    Learning to intervene on concept bottlenecks

    Steinmann, D., Stammer, W., Friedrich, F., and Kersting, K. Learning to intervene on concept bottlenecks. arXiv preprint arXiv:2308.13453,

    arXiv

  31. [2016]

    Towards automatic concept-based explanations

    10 Avoiding Leakage Poisoning: Concept Interventions Under Distribution Shifts Ghorbani, A., Wexler, J., Zou, J., and Kim, B. Towards automatic concept-based explanations. arXiv preprint arXiv:1902.03129,

    1902 arXiv

  32. [2017]

    Interactive concept bottleneck models

    Chauhan, K., Tiwari, R., Freyberg, J., Shenoy, P., and Dvi- jotham, K. Interactive concept bottleneck models. arXiv preprint arXiv:2212.07430,

    arXiv

  33. [2018]

    and Jaakkola, T

    Alvarez-Melis, D. and Jaakkola, T. S. Towards robust inter- pretability with self-explaining neural networks. arXiv preprint arXiv:1806.07538,

    arXiv

  34. [2019]

    and Gimpel, K

    Hendrycks, D. and Gimpel, K. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint arXiv:1610.02136,

    arXiv

  35. [2020]

    Learning multiple layers of features from tiny images

    Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. Technical Report TR-2009, University of Toronto,

    2009

  36. [2021]

    Marcinkeviˇcs, R., Laguna, S., Vandenhirtz, M., and V ogt, J. E. Beyond concept bottleneck models: How to make black boxes intervenable? arXiv preprint arXiv:2401.13544,

    arXiv

  37. [2022]

    Promises and pitfalls of black-box concept learning models

    Mahinpei, A., Clark, J., Lage, I., Doshi-Velez, F., and Pan, W. Promises and pitfalls of black-box concept learning models. arXiv preprint arXiv:2106.13314,

    arXiv

  38. [2023]

    M., Barker, M., Espinosa Zarlenga, M., Raman, N., Bhatt, U., Jamnik, M., Sucholutsky, I., Weller, A., and Dvijotham, K

    Collins, K. M., Barker, M., Espinosa Zarlenga, M., Raman, N., Bhatt, U., Jamnik, M., Sucholutsky, I., Weller, A., and Dvijotham, K. Human Uncertainty in Concept-Based AI Systems. In Proceedings of the 2023 AAAI/ACM Confer- ence on AI, Ethics, and Society, pp. 869–889,

    2023

  39. [2024]

    A., Sevyeri, L

    Sheth, I., Rahman, A. A., Sevyeri, L. R., Havaei, M., and Kahou, S. E. Learning from uncertain concepts via test time interventions. In Workshop on Trustworthy and Socially Responsible Machine Learning, NeurIPS 2022,

    2022

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.