Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Concept-Based Explainable Artificial Intelligence: Metrics and Benchmarks

T0 review · 3 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Post-hoc concept bottleneck models fail a basic existence-and-location audit: the top concept is absent from the image 44.6% of the time and its activation peak misses the expected region 86.7% of the time.

desk verdict A simple, reusable evaluation suite for concept-based XAI; the existence failure of post-hoc CBMs is credible, but the spatial-localisation claim rests on a best-guess proxy and needs a redo. read the letter →

arxiv 2501.19271 v1 pith:CCLHXUP5 submitted 2025-01-31 cs.AI cs.LG

Halil Ibrahim Aysel , Xiaohao Cai , Adam Prugel-Bennett This is my paper · ORCID
classification cs.AIcs.LG
keywords concept-basedexplainableAIconceptbottleneckmodelspost-hocexplanationactivationmappingexistencemetriclocationCUBbenchmarkspatialalignment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether concept-based explanations actually refer to things visible in the image, and it answers with a benchmark. It introduces three metrics—the concept global importance metric (CGIM), the concept existence metric (CEM), and the concept location metric (CLM)—plus a concept activation mapping (CoAM) method for visualising concept heatmaps. Applying these to post-hoc concept bottleneck models on the Caltech-UCSD Birds (CUB) dataset, it reports that the single most important concept for a correctly classified test image exists in that image only 55.4% of the time, and that its heatmap's peak region contains the expected body-part centre only 13.3% of the time when the region is one-twelfth of the image. If these numbers hold, a widely used explanation method frequently tells users the model relied on a concept that is neither visible nor correctly localised, which is why standardised alignment metrics are needed.

What carries the argument

The engine of the audit is the local importance score $\theta_{jk}\hat{u}_{ij}$, the product of the classifier weight linking concept $j$ to class $k$ and the projected concept value for image $i$; the paper ranks concepts by this product rather than by raw concept values because the class score is a $\theta$-weighted sum. For visualisation, CoAM computes per-concept activation maps $F_{ij} = \frac{1}{d}\sum_{k=1}^{d} c_{jk} E_i(:,:,k)$, weighting each pre-GAP feature-map channel by the corresponding concept-activation-vector component and upsampling to image size. CEM then checks whether the top-$l$ ranked concepts appear in the image's ground-truth concept set, while CLM thresholds the upsampled heatmap into the $\alpha(M_1M_2)/12$ highest-activation pixels and checks whether the expected body-part centre pixel falls inside that region.

What would settle it

Run the same CEM and CLM pipeline on a concept-annotated dataset that provides per-concept segmentation masks instead of 12 body-part centres; if most top concepts' heatmaps overlap their true masks at $\alpha=1$, the reported 13.3% mislocalisation would be an artefact of the body-part proxy, while a similarly low overlap would confirm the paper's spatial-alignment claim.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is a failure pattern in post-hoc concept bottleneck models. Ranking concepts by the local importance score $\theta_{jk}\hat{u}_{ij}$ (classifier weight times projected concept value), the top-1 concept exists in only 55.4% of correctly classified test images; the existence rate falls to 45.8% when the top five concepts are checked. The spatial check is worse: with the activation region set to one-twelfth of the image ($\alpha=1$), the expected body-part centre falls inside the top-activating pixels only 13.3% of the time, and even when half the image is scanned ($\alpha=6$) the centre is missed 41% of the time. Global importance vectors also frequently show weak or negative cosine similarity to human ground truth, such as concept 'black eye colour' at $-0.63$. The paper attributes part of the problem to natural correlations among concepts.

Load-bearing premise

The load-bearing premise is that CUB's 12 body-part centre coordinates, after the paper's 'best guess' mapping of 89 concepts to the nearest body part, are a valid proxy for where each concept actually appears in the image; if that proxy is wrong, the CLM's 13.3% figure does not measure concept location.

Editorial extensions

If this is right

  • Concept-based explanations from post-hoc CBMs should not be presented as spatially grounded without an independent existence or location check.
  • The proposed metrics give the field a reusable yardstick: any concept-based explainer can be scored with CGIM, CEM, and CLM on datasets that carry per-image concept labels and part coordinates.
  • The accuracy the authors reproduce for the best post-hoc CBM (59.1%) is far below the unconstrained network (75.4%), and that accuracy loss does not buy reliable concept alignment on this testbed.
  • Future concept-based models should treat concept existence and localisation as first-class evaluation criteria rather than relying on class accuracy alone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because CGIM, CEM, and CLM are method-agnostic, the same audit could be run on traditional concept bottleneck models and label-free concept bottlenecks, which the paper does not benchmark; that would show whether per-image concept supervision closes the alignment gap.
  • The failure pattern is consistent with concepts acting as correlated proxies: a concept can rank highly because it correlates with the class across the dataset even when it is absent from a particular image, so single-image concept attributions should be treated as hypotheses rather than confirmed evidence.
  • A testable next step is to add a spatial-alignment loss to post-hoc CBM training and measure whether CEM and CLM improve; if they do not, the bottleneck lies in the concept bank itself rather than in the final classifier layer.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes three quantitative metrics for auditing concept-based explanations---CGIM (global concept-importance alignment), CEM (concept existence), and CLM (concept location)---together with a visualisation technique called CoAM that produces concept activation maps. The metrics are applied to post-hoc concept bottleneck models (Yuksekgonul et al., 2023) reproduced on the CUB bird dataset. The authors report that global importance weights correlate only weakly with class-level concept annotations, that the single most important local concept is absent from the input image in a substantial fraction of cases, and that concept activation maps often do not contain the expected concept location. They conclude that post-hoc CBMs can produce unreliable concept attributions and argue that the proposed metrics provide a reusable sanity-check benchmark.

Significance. If the results hold, the paper fills a genuine gap: concept-based XAI currently lacks standardised checks for whether the concepts named by an explainer actually exist in the image and are localised correctly. The proposed metrics are simple, interpretable, and applicable beyond the specific benchmark. The strengths of the paper include a reproducible reproduction protocol with hyperparameter details, full per-concept and per-class CGIM tables in the appendix, and candid discussion of the metrics' own limitations in Appendix B. The CEM evidence, based on per-image ground-truth concept labels, is reasonably robust. The main caveat is that the paper's spatial-misalignment claim rests on CLM, whose ground truth is an acknowledged 'best-guess' mapping from 112 concepts to 12 body-part centres; that premise is currently unvalidated and is load-bearing for the central negative result.

major comments (3)
  1. [Section 4.4.2, Tables 4/5, Appendix B] The CLM result rests on an unvalidated premise. CUB provides centre coordinates for 12 body parts, not for the 112 concepts; Section 4.4.2 maps 89 concepts to the 'nearest' body part, and Appendix B calls this a 'best guess'. If the mapping is wrong for even a moderate fraction of the 89 concepts, the reported CLM values (e.g., 13.3% at alpha=1) do not measure what the paper claims they measure. Please validate the mapping, for example by human agreement on a random sample of concept-part pairs, and report how the CLM scores change under alternative or perturbed mappings. Without this, the second conjunct of the paper's main claim--that saliency maps of present concepts fail to align with expected regions--is not established.
  2. [Section 3.1, Eq. (2), Section 4.4.2, Appendix B] CoAM is a new, unvalidated visualisation. It linearly combines pre-GAP feature maps with CAVs trained on post-GAP features, followed by upsampling from a coarse 7x7 resolution. Appendix B concedes that it is 'debatable whether the heatmaps ... correctly capture the location of the concepts.' The CLM experiment should include a sanity check that CoAM localises known concepts correctly on a subset of images, ideally against manual centre or bounding-box annotations for a few concepts. If CoAM itself is systematically displaced or blurry, the low CLM numbers would be an artefact of the visualisation rather than a property of the post-hoc CBM. This validation is necessary because the paper's quantitative spatial conclusion is entirely mediated by CoAM.
  3. [Table 2] The CLM results are reported without a chance baseline or statistical uncertainty, which matters because the observed effects are close to area-based chance. For alpha=1 the scanned region is 1/12 of the image, so a random hotspot centre has probability 8.3% of being counted as correct; the observed 13.3% is only five percentage points above that. For alpha=6 the chance level is 50% and the observed value is 59%. Please report chance rates for each alpha, bootstrap confidence intervals or permutation tests, and ideally per-concept stratification. The current numbers provide only weak quantitative support for the mislocalisation claim.
minor comments (5)
  1. [Section 3.4 vs Section 4.4.2] The definition of the visual region is inconsistent: Section 3.4 says Omega_ij is obtained by thresholding the heatmap with a threshold tau, while Section 4.4.2 defines Omega_ij as the alpha(M1M2)/12 pixels with the largest intensities. Please reconcile these definitions and state how tau was chosen, or remove tau from the metric definition.
  2. [Table 1] The CEM numbers are point estimates without a chance baseline. Because l=1 selects one concept out of 112, reporting the average fraction of active concepts per image as a reference would help the reader calibrate the 55.4% and 49.3% figures.
  3. [Figure 2 and Tables 6-8] The CGIM distributions would be easier to interpret with an explicit random-alignment reference, since cosine similarity in 200 dimensions has a known null distribution; at minimum, report the mean and standard deviation per CGIM variant in addition to the full per-concept and per-class lists.
  4. [Appendix B] There is a typo in 'plotting histograms of the CBMs results' (should be 'CBM results'), and the paragraph describing Aysel et al. (2023) fits better in Related Work than in the Limitations section.
  5. [Section 4] Please clarify which CUB concept set is used: CUB-200-2011 contains 312 attributes, but the paper uses 112 binary concept labels. State whether these are the same 112 concepts used by Yuksekgonul et al. (2023) and, if so, cite the exact source of that subset.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the metrics are audited against external CUB ground truth; only a minor non-load-bearing self-citation appears in the Limitations.

full rationale

The paper's derivation chain is self-contained and benchmarked against external ground truth. CEM compares the post-hoc CBM's importance ranking (θjk·ûij) with per-image concept labels Λi from CUB; CGIM compares the classifier weights θ (or average concept values) with the class-level ground-truth concept matrix V; CLM compares thresholded CoAM heatmaps with CUB body-part centre coordinates. None of these ground truths is derived from the metric outputs or from the benchmarked model's parameters, and the benchmarked post-hoc CBM (Yuksekgonul et al., 2023) is an independent method reproduced by the authors. The CoAM map F_ij = (1/d) Σ_k c_jk E_i(:,:,k) is a deterministic weighting of pre-GAP feature maps by the CAV; although the spatial average of F_ij is proportional to the concept value ûij, CLM evaluates a different statistic (whether the ground-truth centre pixel falls inside the thresholded high-activation region), so the 13.3% localisation score is not forced by the concept-value ranking. The acknowledged 'best guess' approximation for concept-to-body-part mapping and the admitted uncertainty about whether CoAM heatmaps 'correctly capture the location of the concepts' are validity limitations of the CLM proxy, not circular reductions: the CLM measurement could in principle have been high if the model localised concepts well. The only self-citation to Aysel et al. (2023) appears in the Related Work and Limitations as a comparison/superiority claim and is not load-bearing for any table value, metric definition, or central conclusion. Therefore no circular step is present; the score reflects the minor, non-load-bearing self-citation and the acknowledged proxy limitations, which are correctness risks rather than circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The metrics themselves are parameter-light; the principal free choices are the CLM region scale α, the top-l reporting convention, and the SVM hyperparameters of the reproduced post-hoc CBM. The domain assumptions are CUB ground-truth reliability, the concept-to-body-part mapping, and the spatial validity of upsampled CoAM heatmaps. No new entities are introduced.

free parameters (3)
  • α (CLM region scale) = 1, 3, 6
    Controls the size of the high-activation region Ωij in Eq. (11); CLM scores vary from 13.3% to 59.0% across α, so the headline numbers depend on this hand-chosen threshold.
  • l (number of top concepts) = 1, 3, 5
    CEM and CLM report scores for the first l ranked concepts; the choice is conventional but affects the absolute values reported in Tables 1 and 2.
  • SVM hyperparameters (Np, Nn, λ) = Np=Nn=100, λ=1
    Selected to maximize classifier accuracy in Table 3 (59.1%); the results are reported for this configuration, and accuracy varies from 25.3% to 59.1% across the scan.
assumptions (4)
  • domain assumption CUB per-image concept annotations Λi and centre pixel coordinates Pi are reliable ground truth.
    CEM and CLM rely on these labels and coordinates; any annotation noise or coordinate error propagates directly into the reported scores.
  • ad hoc to paper Each of the 89 evaluable concepts can be mapped to one of the 12 body parts with available centre coordinates.
    In Section 4.4.2 and Limitations, authors assign concepts to the nearest body part as a 'best guess'; CLM scores inherit this approximation.
  • domain assumption Upsampling late-layer concept activation maps to image size preserves enough spatial information for a centre-pixel hit test.
    CoAM produces low-resolution maps that are upsampled; the authors themselves call the fidelity into question in Appendix B.
  • domain assumption The reproduced post-hoc CBM (ResNet-18, SVM CAVs, linear classifier) faithfully represents Yuksekgonul et al. (2023).
    The benchmark's negative findings attach to post-hoc CBMs generally, but only one architecture and one dataset are tested; the authors show only accuracy parity (59.1% vs 58.8%).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Concept-Based Explainable Artificial Intelligence: Metrics and Benchmarks." pith.science (2026). https://pith.science/paper/CCLHXUP5

@misc{pith2026250119271,
  author       = {Pith},
  title        = {Pith review of: Concept-Based Explainable Artificial Intelligence: Metrics and Benchmarks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CCLHXUP5}},
  note         = {Machine review of arXiv:2501.19271}
}
read the original abstract

Concept-based explanation methods, such as concept bottleneck models (CBMs), aim to improve the interpretability of machine learning models by linking their decisions to human-understandable concepts, under the critical assumption that such concepts can be accurately attributed to the network's feature space. However, this foundational assumption has not been rigorously validated, mainly because the field lacks standardised metrics and benchmarks to assess the existence and spatial alignment of such concepts. To address this, we propose three metrics: the concept global importance metric, the concept existence metric, and the concept location metric, including a technique for visualising concept activations, i.e., concept activation mapping. We benchmark post-hoc CBMs to illustrate their capabilities and challenges. Through qualitative and quantitative experiments, we demonstrate that, in many cases, even the most important concepts determined by post-hoc CBMs are not present in input images; moreover, when they are present, their saliency maps fail to align with the expected regions by either activating across an entire object or misidentifying relevant concept-specific regions. We analyse the root causes of these limitations, such as the natural correlation of concepts. Our findings underscore the need for more careful application of concept-based explanation techniques especially in settings where spatial interpretability is critical.

Figures

Figures reproduced from arXiv: 2501.19271 by the authors.

Figure 1
Figure 1. Overview of CAVs, CBMs, post-hoc CBMs and the proposed techniques. Feature extractor ⃝a , concept prediction block ⃝b , CAVs ⃝c , concept bottleneck ⃝d , classifier ⃝e , and our proposed CoAM framework ⃝f . A traditional (without concept bottleneck) classification model consists of ⃝a + ⃝e , and ⃝c is the introduced post-hoc to explain its predictions via CAVs. ⃝a + ⃝b + ⃝d + ⃝e forms the steps for traditional CBMs … view at source ↗
Figure 2
Figure 2. Histograms of the CGIM scores of the post-hoc CBMs. Plots on the left and right columns show the results for classes and concepts, respectively. A full list of the CGIM scores can be found in [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Randomly selected test images from different classes and the top 5 most important concepts for their classification by the post-hoc CBMs. In particular, symbols ✓ and ✗ are for concept existence and absence in the ground-truth label, respectively. of animals, a group of hooved animals often share other com￾mon features (e.g., they are often quadrupeds that feed on grass). Although these other features might help to … view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: presents some examples of class and concept visualisation by using our CoAM. The matching between the concept groups and the body parts for the CUB dataset is given in Tables 4 and 5. Input Concept-wise maps Class maps White breast Nuthatch multi-colour wing (✓) white …

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Survival Concept-Based Learning Models

    cs.LG 2025-02 conditional novelty 6.0 of 10

    SurvCBM and SurvRCM combine concept bottleneck learning with Cox and Beran survival models, and SurvCBM achieves the best C-index and concept F1 on synthetic MNIST and CIFAR experiments.

Reference graph

Works this paper leans on

12 extracted references · 11 canonical work pages · cited by 1 Pith paper

  1. [1]

    This result is very close to the accuracy 58.8% reported in the seminal work (Yuksekgonul et al., 2023). Note that there is more than 15% accuracy loss in comparison to the traditional model, i.e., the one without a concept bottleneck (i.e., a⃝ + e⃝ in Figure 1), for 12 Concept-Based Explainable Artificial Intelligence: Metrics and Benchmarks the sake of ...

    work page 2023

  2. [3]

    best guess

    Classification accuracy of the reproduced post-hoc CBMs with different settings for the parameters λ, Np, and Nn. λ Np = Nn 50 100 0.001 26.7 52.2 0.01 34.1 44.9 0.1 29.1 41.5 1 25.5 59.1 10 25.3 58.7 Traditional model w/o bottleneck 75.4 B. Limitations There are drawbacks to the metrics CEM and CLM that we propose. The CEM can only be used on datasets wh...

    work page 2023

  3. [7]

    The cost-effective solution to CBMs is achieved by only requiring class-wise concept annotations rather than per-image

    offers solutions for both expensive annotation needs and single-level output drawbacks of CBMs. The cost-effective solution to CBMs is achieved by only requiring class-wise concept annotations rather than per-image. Moreover, the multilevel XAI method provides concept-wise heatmaps by-product handling the single-level limitation of CBMs. To be more precis...

    work page 2023

  4. [8]

    addresses this limitation by refining the localisation to better handle multiple instances of objects, offering a more fine-grained interpretation. Further extensions include Score-CAM (Wang et al., 2020), which eliminates the dependency on gradients, instead using the activations themselves to weigh different regions of the input. This addresses some of ...

    work page 2020

  5. [12]

    with reproduced post-hoc CBMs (Yuksekgonul et al., 2023). Class CGIM ρCGIM1 k ρCGIM2 k ρCGIM3 k Class CGIM ρCGIM1 k ρCGIM2 k ρCGIM3 k 1: Black footed Albatross 0.24 −0.20 0.03 51: Horned Grebe 0.27 −0.24 0.24 2: Laysan Albatross 0.30 0.04 0.11 52: Pied billed Grebe 0.32 −0.12 0.32 3: Sooty Albatross 0.28 −0.15 0.17 53: Western Grebe 0.25 −0.04 0.24 4: Gro...

    work page 2023

  6. [2011]

    with reproduced post-hoc CBMs (Yuksekgonul et al., 2023). Concept CGIM ρCGIM1 j ρCGIM2 j ρCGIM3 j Concept CGIM ρCGIM1 j ρCGIM2 j ρCGIM3 j 1: Dagger beak 0.54 0.05 0.41 57: Yellow forehead colour 0.41 0.22 0.47 2: Hooked seabird beak 0.45 0.08 0.34 58: Black forehead colour 0.39 0.25 0.35 3: All-purpose beak 0.37 0.67 0.46 59: White forehead colour 0.52 −0...

    work page 2023

  7. [2018]

    CA Vs represent directions in the latent space of a model corresponding to specific concepts, allowing for sensitivity analysis

    introduced CA Vs to explain model predictions based on high-level human-interpretable concepts. CA Vs represent directions in the latent space of a model corresponding to specific concepts, allowing for sensitivity analysis. By perturbing an input in the direction of a concept vector, TCA V measures how much the model’s prediction depends on that specific...

    work page 2019

  8. [2020]

    They enforce that intermediate representations of the model correspond to human-understandable concepts, such as attributes (e.g., colour, shape, part) of objects in an image

    offers a significantly different approach to interpretability. They enforce that intermediate representations of the model correspond to human-understandable concepts, such as attributes (e.g., colour, shape, part) of objects in an image. By constraining the model to predict based on these explicit concepts, CBMs inherently provide an interpretable mechan...

    work page 2023

Show all 12 references
  1. [2021]

    Chattopadhay, A., Sarkar, A., Howlader, P., and Balasubra- manian, V . N. Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks. In 2018 IEEE Winter Conference on Applications of Com- puter Vision (WACV), pp. 839–847. IEEE,

    2018

  2. [2022]

    Vandenhirtz, M., Laguna, S., Marcinkeviˇcs, R., and V ogt, J. E. Stochastic Concept Bottleneck Models. In ICML 2024 Workshop on Structured Probabilistic In- ference & Generative Modeling ,

    2024

  3. [2023]

    and Zisserman, A

    Simonyan, K. and Zisserman, A. Very deep convolu- tional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556,

    arXiv

  4. [2024]

    Wah, C., Branson, S., Welinder, P., Perona, P., and Belongie, S

    URL https: //openreview.net/forum?id=8jG3Y0xX7b. Wah, C., Branson, S., Welinder, P., Perona, P., and Belongie, S. The caltech-ucsd birds-200-2011 dataset

    2011

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.