{"id":"da051d26-0f69-42fe-8024-76b1c8d71d5f","arxiv_id":"2412.11576","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A concept bottleneck model that derives its concepts from segmentation and detection foundation models instead of text, achieving competitive accuracy and better out-of-distribution robustness with only 50 images per class for concept generation.","lead":"DCBMs build interpretable image classifiers by using segmentation and detection models to turn image regions into concepts, then training a simple linear model on those concepts. The method needs only 50 images per class to build its concept set, which makes it attractive for fine-grained and out-of-distribution tasks where text descriptions are scarce.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Concept activations in Eq. (2) are never validated as concept-presence indicators; the paper's own Fig. 4 shows high activations for concepts absent from the image, so the 'preserving interpretability' claim is not yet established.","rationale":"I read the paper as claiming a data-efficient, interpretable CBM whose concepts are visual regions, with the strongest evidence being Table 1 (CUB 82.4) and Table 2 (lower IN-R gap). The data-efficiency part is supported by the subset ablations, and the accuracy claims are plausible. The load-bearing gap is the interpretability side: the activation function is never checked against actual concept presence, and the paper's own qualitative results show that this can fail. This is not an ad hominem or a disagreement with the CBM community; it is an internal validity question. The Grid Pointing Game does not settle it because it only measures whether the linear layer's Grad-CAM points to the right image, not whether the bottleneck feature itself tracks the concept. The proposed AUC test would directly settle this. The reader's weakest assumption (concept bank representative and linearly separable) is related; my concern is one step later, at the activation-to-presence mapping, so agreement is partial. Since the needed evidence is missing, the existing CONDITIONAL verdict is appropriate; the paper should add this grounding check (or equivalent) before the interpretability claim is accepted.","tokens_in":37876,"tokens_out":10814,"duration_ms":109661,"concrete_test":"Run the same proposal model on held-out test images to obtain region masks. For each concept c_j, define ground-truth presence as whether any test region embedding is closer to c_j than to other centroids. Then compute the ROC-AUC of the CBM activation a_j(x)=<f(x),c_j>/||c_j||^2 for predicting that presence, averaged over the highest-weight concepts. If the mean AUC is near 0.5, Eq. (2) does not encode concept presence and the interpretability claim must be weakened; if AUC is high (e.g., >0.8), the bottleneck is grounded. Also rerun Table 4 with corrected metrics across at least 3 seeds.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central interpretability claim depends on Eq. (2), where a concept activation is the projection of the whole-image CLIP embedding onto a crop-derived cluster centroid. Such a global projection does not measure whether the concept's visual pattern is actually present in the image. The paper's own Figure 4 shows an ambulance correctly classified with top concepts 'med', 'police', 'kit' and 'injection' that are not visible in the input, and the Discussion concedes decisions are based on 'semantically close concepts that are not present in the test image.' The only quantitative localization evidence, the Grid Pointing Game in Table 4, tests Grad-CAM of the trained linear layer on a 2x2 grid, not the correspondence between high activation and concept presence; the reported numbers are also internally inconsistent (Abs 0.6551/0.3445 in Table 4 vs 0.7028/0.4988 in Section 4.4.2). If Eq. (2) activations are not grounded, the model is still a linear probe with a concept dictionary, and the central claim of preserving interpretability is unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Data-efficient Concept Bottleneck Models (DCBMs), which build a concept bottleneck from image regions proposed by segmentation or detection foundation models (SAM, SAM2, Mask R-CNN, DETR, GroundingDINO). The proposals are embedded with a frozen CLIP image encoder, clustered with k-means, and the cluster centroids serve as concepts for a sparse linear classifier. Concepts are optionally named via CLIP text alignment. The authors claim that DCBMs need only 50 images per class for concept generation, yield dataset-specific multi-granularity concepts, perform well on fine-grained tasks and under domain shift, and preserve interpretability by allowing visual concepts to be localized in test images. The evaluation covers ImageNet, Places365, CUB, CIFAR-10/100, ImageNet-R, MiT-States, ClimateTV, AwA2, and CelebA, with extensive ablations and a public code release.","tokens_in":38002,"tokens_out":5312,"duration_ms":49518,"significance":"If the claims were fully established, DCBM would be a practically valuable recipe: foundation-model segmentation or detection plus k-means in CLIP space and a sparse linear layer yields an interpretable CBM from very few images, with released code and unusually thorough ablations. The paper is also honest about limitations, explicitly acknowledging spurious correlations and the possibility of decisions based on concepts not present in the image. However, the central interpretability claim is not yet supported: concept activations in Eq. (2) are never validated as concept-presence indicators, and the only quantitative localization evidence, the Grid Pointing Game, reports internally inconsistent numbers. The contribution is therefore promising but currently overclaimed relative to the evidence.","major_comments":[{"comment":"The central claim that DCBMs 'preserve interpretability' is not established because concept activations in Eq. (2) are projections of the whole-image CLIP embedding onto cluster centroids, and the paper never shows that high activation implies the concept's visual presence in the image. Figure 4 itself shows an ambulance correctly classified with top concepts 'med', 'police', 'kit', and 'injection' that are not visible in the input, and Section 5 concedes that decisions can be based on 'semantically close concepts that are not present in the test image.' The Grid Pointing Game in Section 4.4.2 validates Grad-CAM localization of the linear layer's chosen concepts, not the correspondence between Eq. (2) activations and concept presence. I request a direct validation of Eq. (2) as a presence indicator, for example through concept-presence annotations or intervention experiments showing that activation is high only when the concept is actually present, or a revised claim that restricts interpretability to a linear probe with a named visual dictionary.","section":"Section 3.3, Eq. (2); Sections 4.3, 4.4.2, and 5"},{"comment":"The quantitative localization evidence is internally inconsistent and therefore not usable as support. The text reports 'Abs: 0.7028 vs. 0.4988' while Table 4 lists Abs values of 0.6551 and 0.3445. The text also lists three metrics (Gini Index, relative percentage, maximum score), whereas Table 4 has columns labeled Gini, Percentage, and Abs, with no definition of the difference between 'Percentage' and 'Abs' or of 'maximum score.' No error bars or confidence intervals are reported. These numbers should be reconciled, the metrics precisely defined, and the evaluation rerun or reported with variance.","section":"Section 4.4.2 and Table 4"},{"comment":"The claim that 'On both datasets, DCBMs outperform the linear probes' is contradicted by Table 3: on ClimateTV, DCBM-GDINO achieves 81.8% versus the linear probe's 84.5%. Similarly, the 'within 6%' statement in Section 4.2.1 conflicts with the 'within 5%' statement in Section 5, given that the ImageNet gap in Table 1 is exactly 6.0%. These contradictions should be corrected, and the performance comparisons should be accompanied by error bars or significance statements, since many differences in Table 1 and Table 3 are smaller than typical run-to-run variation.","section":"Section 4.2.1 and Table 3"},{"comment":"The paper motivates itself against parallel visual-concept CBMs (Kowal et al., 2024; Sun et al., 2024; Zhu et al., 2024) and cites them in Section 2, but Table 1 includes only text-based CBMs and DN-CBM. Without a quantitative comparison to at least one of these closest visual-concept methods, the claim of 'extensively evaluate' and the positioning against parallel visual-concept CBMs are incomplete. Please add such comparisons where feasible, or explicitly state and justify why they are not comparable.","section":"Related Work and Table 1"}],"minor_comments":[{"comment":"Please reconcile the 'within 6%' and 'within 5%' statements; Table 1 shows an ImageNet gap of 6.0%, so the Discussion's 'within 5%' is inaccurate.","section":"Section 4.2.1 vs. Section 5"},{"comment":"The sentence 'Percentage and absolute scores are lower bounded at 0.25' is unclear because the text earlier defines only Gini, relative percentage, and maximum score; please define every metric precisely.","section":"Section 4.4.2"},{"comment":"The caption reads 'effectively (Step 3)' without a verb; please rephrase to complete the sentence.","section":"Figure 2 caption"},{"comment":"The value '82.' for DCBM-MASK-RCNN with CLIP ViT-L/14 appears to be missing a digit; please correct it.","section":"Table 13 in Appendix D.2"},{"comment":"The appendix states that 'Each DCBM was trained with 1024 clusters,' whereas the main text uses k = 2048; please clarify that the ablations use a different cluster count and explain any implications.","section":"Appendix D, first paragraph"},{"comment":"The text 'A W A2' appears to be a typo for 'AwA2'.","section":"Appendix F.2"},{"comment":"The phrase 'as in(Wang et al., 2023)' is missing a space before the citation.","section":"Section 5"}],"recommendation":"major_revision","confidential_remarks":"This is a promising empirical paper with transparent ablations, released code, and an honest limitations section. The main risk is overclaiming interpretability: the concept activation mechanism in Eq. (2) is not validated as a concept-presence signal, and the paper's own Figure 4 and Section 5 undercut the claim. The GPG inconsistency and the absence of parallel visual-CBM baselines further weaken the evaluation. I support major revision rather than rejection, because the core pipeline is simple, reproducible, and likely to be useful once the interpretability evidence is strengthened and the internal inconsistencies are resolved."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid, useful empirical paper. The core idea—replace LLM/text concept sets with segmentation/detection proposals, cluster them in CLIP space, and train a sparse linear layer on 50 images per class—is simple, clearly specified, and well evaluated. Code is provided. The ablations on images-per-class, clustering, centroid choice, backbones, and proposal methods are unusually thorough. The OOD transfer result (lower ImageNet-to-ImageNet-R gap than DN-CBM) is interesting and worth following up.\n\nThe soft spots are real but mostly addressable. The biggest is the gap between the interpretability claim and the evidence. Equation (2) defines a concept activation as a global projection of the image embedding onto a cluster centroid; that does not by itself establish that the concept's visual pattern is present in the image. The paper's own Figure 4 shows an ambulance classified via 'med', 'police', 'kit', and 'injection'—none visible in the image—and the Discussion concedes decisions can rest on semantically close concepts absent from the test image. The Grid Pointing Game results are presented as localization evidence, but they evaluate Grad-CAM of the linear layer, not the correspondence between high concept activation and concept presence. So the 'preserving interpretability' phrasing in the abstract is stronger than what is demonstrated. The authors are transparent about spurious correlations, which I credit, but the central interpretability claim should either be tempered or backed by a direct concept-presence evaluation.\n\nThere are also several small inconsistencies: the Discussion says 'within 5%' while Table 1 shows a 6-point ImageNet gap; the GPG absolute numbers differ between Table 4 and Section 4.4.2; and parallel visual-concept CBMs (VCC, Explain Any Concept, Zhu et al.) are cited but not compared. Main results have no error bars. None of this is fatal; the empirical core is probably sound.\n\nWho should read it: anyone building CBMs for fine-grained or data-sparse domains, and people working on visual concept discovery. I would send it to review. The requested revisions are concrete: fix the number mismatches, add comparisons to the parallel works, and either weaken the interpretability claim or add a direct test of whether high activations imply visual presence.","headline":"Useful data-efficient CBM with thorough ablations; interpretability claim needs grounding before it fully lands.","tokens_in":38647,"tokens_out":2913,"would_cite":true,"duration_ms":26341,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper shows that a concept bottleneck model can be built from image regions proposed by segmentation or detection models, needing only 50 images per class for concept generation, and that this visual concept bank is accurate enough to…","keywords":["concept bottleneck models","visual concepts","segmentation foundation models","detection foundation models","data-efficient concept generation","fine-grained classification","out-of-distribution generalization","concept localization"],"falsifier":"On the CUB bird dataset used in the paper, run DCBM twice, once with part-sensitive segment proposals and once with only whole-image crops as proposals, keeping cluster counts, centroids, and training identical; if the whole-image variant matches the segment variant in accuracy, the claim that image-region concepts are doing the work would fail. A complementary check is to compute the Grid Pointing Game score after ablating all concepts whose Grad-CAM peak in test images falls outside their own proposal bounding box; if the localization score stays near the 0.25 random baseline, the localization claim fails.","tokens_in":37568,"feed_emoji":"🖼️","tokens_out":10336,"duration_ms":85139,"temperature":0.7,"pith_summary":"Concept Bottleneck Models usually get their concepts from large language model text descriptions or large image corpora, which makes them hard to use when data is scarce. The paper claims that a CBM can instead define concepts as image regions cut out by off-the-shelf segmentation or detection models, cluster those regions in CLIP embedding space, and train a sparse linear layer over the cluster centroids. With only 50 images per class, this produces a concept bank that yields competitive accuracy, including the best CBM accuracy in its comparison on the fine-grained CUB dataset (82.4% with the ViT-L/14 backbone) and smaller accuracy losses on ImageNet-R than a task-agnostic text-named CBM. The significance is that interpretability no longer depends on a predefined vocabulary: the concepts are dataset-specific, can be localized in test images, can be named after the fact, and can be removed by a text prompt if they are spurious.","feed_headline":"50 images per class build an interpretable visual classifier","feed_subtitle":"A concept bank built from image segments, not text, keeps fine-grained accuracy and survives domain shifts.","key_machinery":"The central object is a concept bank $C=\\{c_j\\}_{j=1}^{k}$ of $k$ cluster centroids in the CLIP embedding space, each centroid being the median embedding of a cluster of image-region crops proposed by a segmentation or detection foundation model. Concept activations are computed as the normalized projection $a(x_i)_j = \\langle f(x_i), c_j\\rangle / \\|c_j\\|_2^2$, and a sparse linear layer $t(x_i)=\\omega^{\\top}a(x_i)$ with $\\ell^1$ regularization turns these activations into class predictions. The machinery is: foundation-model proposals give multi-granularity visual concepts; k-means clustering collapses redundant crops into a few thousand centroids; and the frozen CLIP encoder plus linear layer keeps the whole pipeline inspectable at every step.","core_discovery":"The paper's discovery is that a concept bottleneck model does not need text descriptions, large concept corpora, or task-agnostic pretraining to be accurate and interpretable. If the image is first decomposed by a segmentation or detection foundation model into region proposals, and those proposals are embedded, clustered, and reduced to a few thousand centroids, then a single sparse linear layer over the centroid activations can classify competitively. The paper reports that this holds across several generic segmenters and a promptable detector, with 2048 clusters as the default, and that the approach is strongest where predefined text concepts are weakest: fine-grained classes such as CUB (82.4% with ViT-L/14, the best CBM in its comparison) and out-of-distribution renditions, where every DCBM variant keeps a smaller IID-to-OOD error gap than the task-agnostic DN-CBM. The visual concepts are also localizable, as measured by Grad-CAM and the Grid Pointing Game, and they can be named post hoc by matching cluster centroids to text in CLIP space.","pith_inferences":["The paper leaves implicit that the frozen concept bank doubles as a fixed visual vocabulary for post-hoc explanation: project any black-box model's embeddings onto the same centroids to obtain concept attributions without retraining the pipeline.","A stress test beyond the paper's ablations would vary segment size thresholds far beyond the tested range; if accuracy stays flat across very different proposal distributions, the data-efficiency claim is robust, and if it changes sharply, the choice of segmentation model is the real load-bearing component.","The discussion sketches interpretable regression through concept activations as spatial extent; this is testable, for instance by predicting lesion severity from how strongly a single damage concept activates across an image."],"forward_implications":["DCBM achieves 82.4% top-1 accuracy on CUB with the ViT-L/14 backbone, the best among the CBMs compared, indicating that visual part-level concepts fit fine-grained classification.","On ImageNet-R, DCBM variants show IID-to-OOD error gaps of 22.4–27.4 percentage points, versus 38.8 for DN-CBM, indicating that visual dataset-specific concepts transfer to out-of-distribution renditions better than task-agnostic text-named concepts.","Building the concept bank from 50 images per class cuts concept-generation data by about 96% on ImageNet compared with the task-agnostic baseline and needs no extra image-caption corpus or pretraining.","On MiT-States and ClimateTV, DCBM outperforms the linear probe (e.g., 42.8–43.3 vs. 37.3 on MiT-States and 85.6–87.9 vs. 84.5 on ClimateTV), suggesting the pipeline adapts to novel domains quickly.","Because concepts are visual and clustered, unwanted concepts can be removed after clustering via a text prompt, giving users a way to suppress spurious correlations (e.g., removing stone concepts from a gull classifier leaves accuracy unchanged or slightly better)."],"supporting_citations":[{"why":"defines the concept bottleneck formulation that the method inherits.","marker":"Koh et al., 2020"},{"why":"supplies the frozen CLIP encoder whose embedding space carries all concepts.","marker":"Radford et al., 2021"},{"why":"provides the SAM proposal generator used for concept crops.","marker":"Kirillov et al., 2023"},{"why":"provides the SAM2 proposal generator used for concept crops.","marker":"Ravi et al., 2025"},{"why":"provides the promptable detector that steers proposals toward part-like regions.","marker":"Liu et al., 2024"},{"why":"provides the Mask R-CNN detector used as a proposal baseline.","marker":"He et al., 2017"},{"why":"supplies the normalized projection used to compute concept activations.","marker":"Yuksekgonul et al., 2023"},{"why":"is the task-agnostic baseline compared on accuracy and OOD gap, and supplies the naming-corpus idea.","marker":"Rao et al., 2024"},{"why":"defines the Grid Pointing Game used to verify concept localization.","marker":"Bohle et al., 2021"},{"why":"provides Grad-CAM attributions used to localize concepts in test images.","marker":"Selvaraju et al., 2019"}],"fun_headline_variants":["Concept models from image regions, data-efficient and OOD-robust","Replace text with segments for affordable, localizable concepts","Image-segment concepts: accurate, localizable, and domain-shift safe","Fine-grained interpretability without text: segment-based concepts","Concise concept models from segmented images, no text needed"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the image regions cut out by a pretrained segmenter or detector from as few as 50 images per class, after clustering, capture the visual distinctions the target classes differ on; if the proposals miss the discriminative parts, the linear classifier has no way to recover the lost accuracy.","fun_headline_variants_meta":{"raw":{"variants":["Concept models from image regions, data-efficient and OOD-robust","Replace text with segments for affordable, localizable concepts","Image-segment concepts: accurate, localizable, and domain-shift safe","Fine-grained interpretability without text: segment-based concepts","Concise concept models from segmented images, no text needed"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000281,"raw_usage":{"total_tokens":1645,"prompt_tokens":909,"completion_tokens":736,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":525,"completion_tokens_details":{"reasoning_tokens":650}},"tokens_in":525,"tokens_out":736,"duration_ms":7264,"temperature":1.0,"reasoning_tokens":650,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:47:04.967850+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On the CUB bird dataset used in the paper, run DCBM twice, once with part-sensitive segment proposals and once with only whole-image crops as proposals, keeping cluster counts, centroids, and training identical; if the whole-image variant matches the segment variant in accuracy, the claim that image-region concepts are doing the work would fail. A complementary check is to compute the Grid Pointing Game score after ablating all concepts whose Grad-CAM peak in test images falls outside their own proposal bounding box; if the localization score stays near the 0.25 random baseline, the localization claim fails.","supporting_citations":[{"cited_title":"W., Nguyen, T., Tang, Y","cited_arxiv_id":null,"evidence_quote":"defines the concept bottleneck formulation that the method inherits."},{"cited_title":"C., Lo, W.-Y., et al","cited_arxiv_id":null,"evidence_quote":"provides the SAM proposal generator used for concept crops."},{"cited_title":"V., Carion, N., Wu, C.-Y., Girshick, R., Dollar, P., and Feichtenhofer, C","cited_arxiv_id":null,"evidence_quote":"provides the SAM2 proposal generator used for concept crops."},{"cited_title":"Grounding dino: Marrying dino with grounded pre-training for open-set object detection","cited_arxiv_id":null,"evidence_quote":"provides the promptable detector that steers proposals toward part-like regions."},{"cited_title":"Post-hoc concept bottleneck models","cited_arxiv_id":null,"evidence_quote":"supplies the normalized projection used to compute concept activations."},{"cited_title":"Discover-then-name: Task-agnostic concept bottlenecks via automated concept discovery","cited_arxiv_id":null,"evidence_quote":"is the task-agnostic baseline compared on accuracy and OOD gap, and supplies the naming-corpus idea."},{"cited_title":"Convolutional dynamic alignment networks for interpretable classifications","cited_arxiv_id":null,"evidence_quote":"defines the Grid Pointing Game used to verify concept localization."},{"cited_title":"R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D","cited_arxiv_id":null,"evidence_quote":"provides Grad-CAM attributions used to localize concepts in test images."}],"review_version":1}