Pith. sign in

REVIEW 4 major objections 6 minor 12 references

V2C-CBM: Building Concept Bottlenecks with Vision-to-Concept Tokenizer

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read V2C-CBM shows that concept bottleneck classifiers can be built from common words and unlabeled images alone, matching or beating LLM-guided concept bottlenecks and even a CLIP linear probe on ImageNet.

desk verdict A clever training-free CBM recipe, but the ImageNet win is confounded by using ImageNet's own training images as the unlabeled pool. read the letter →

arxiv 2501.04975 v1 pith:OMDABTBW submitted 2025-01-09 cs.CV

Hangzhou He , Lei Zhu , Xinliang Zhang , Shuang Zeng , Qian Chen , Yanye Lu This is my paper · ORCID
classification cs.CV
keywords conceptbottleneckmodelsinterpretablemachinelearningvision-languageCLIPimagequantizationdiscoveryfew-shotclassificationunlabeledimages
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that concept bottleneck models—classifiers that first name human-readable visual concepts and then combine them linearly—can be built without large language models or expert annotations. The proposed V2C tokenizer uses CLIP and a vocabulary of common English words, plus unlabeled images selected by class name, to turn any image into a short list of visual concepts. On ten classification benchmarks, the resulting V2C-CBM matches or exceeds LLM-guided concept bottlenecks and, on ImageNet, also exceeds a linear probe on full CLIP features. The payoff is a training-efficient, inherently interpretable model whose concepts are concise and visual, such as "black head" or "white bear."

What carries the argument

The V2C tokenizer is the mechanism: it maps an image feature vector to the K concept texts whose CLIP embeddings are nearest in Euclidean distance (Eq. 5), after a concept-filtering step that keeps only the most frequent, visually grounded words for each class. The vocabulary starts from common English words, extended by adjective-noun bigrams and relational trigrams, and is filtered using augmented views of class-selected unlabeled images. The tokenizer itself is training-free; only the class-concept weight matrix $W$ of the linear head is learned.

What would settle it

Take the Food-101 setup and swap the unlabeled image pool for images that contain none of the target classes, such as landscapes; the paper's logic implies the filtered concept vocabulary will no longer contain food-relevant words, and accuracy should drop sharply, approaching the 'without food' ablation's 28.9-point one-shot gap.

Watch

Extended reading notes

Core claim

The central claim is that the visual concepts needed for a concept bottleneck can be discovered from images themselves rather than generated by a language model. The paper shows that a codebook of common words, filtered by how often each word is the nearest CLIP text embedding to class-selected unlabeled images, yields a bottleneck that is both interpretable and accurate. Specifically, V2C-CBM outperforms or matches GPT-3-supervised CBMs (LaBo, CDM, DCLIP) and the sparse-autoencoder-based DN-CBM across CIFAR, fine-grained, texture, remote-sensing, and medical datasets, and reaches 84.1% on ImageNet versus 83.9% for a linear probe. The method also closes the few-shot gap to LaBo once four or more labeled examples per class are available.

Load-bearing premise

The premise that breaks the method is that the everyday words most often matched to unlabeled example images of a class actually capture what visually separates that class from others; if they do not, the linear classifier built on those word scores cannot match a full black-box probe.

Editorial extensions

If this is right

  • Concept bottlenecks for new classification tasks can be constructed from a frozen CLIP model and unlabeled web images, with no LLM querying and no tokenizer training.
  • The accuracy gap between inherently interpretable CBMs and black-box linear probes can be closed or inverted on large datasets: 84.1% versus 83.9% on ImageNet.
  • In few-shot settings, the method trails LLM-guided LaBo at 1–2 shots but overtakes it from 4 shots onward, so the value of LLM prior knowledge shrinks as labeled data grows.
  • The discovered concepts are short visual phrases (e.g., "purple center," "black head") rather than verbose or non-visual LLM descriptions, making the bottleneck easier for a person to inspect and intervene on.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the vocabulary is just common words and the tokenizer is frozen, the same codebook could be reused across datasets; a testable extension is to specialize the word list per domain (medical, remote sensing) and see whether accuracy improves further.
  • The ImageNet result uses ImageNet training images as the unlabeled pool, which makes the ImageNet evaluation partly transductive; on a truly out-of-distribution target the margin over the linear probe may shrink, and this could be tested with an unlabeled pool from a disjoint source.
  • The nearest-token selection rule suggests a natural intervention interface: editing or removing a concept (e.g., "white") should change predictions in predictable directions, offering a cheap faithfulness check the paper does not run.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes V2C-CBM, a concept bottleneck model that avoids LLM-generated concepts by building a vocabulary from common English words and using an auxiliary unlabeled image pool. A Vision-to-Concept (V2C) tokenizer quantizes CLIP image features into the nearest concept-text embeddings; per-class concepts are selected by frequency over class-selected unlabeled images; and a linear layer maps the concept similarity scores to class predictions. Experiments on ten datasets compare with LaBo, CDM, DCLIP, DN-CBM, and linear probing, including few-shot results for 1 to 16 shots. The central claim is that V2C-CBM matches or outperforms LLM-supervised CBMs and even outperforms black-box linear probing on ImageNet.

Significance. The proposal is genuinely appealing: concept bottlenecks are constructed without LLMs, without concept annotations, and without training a sparse autoencoder, and the authors release code. The ablations on vocabulary size, unlabeled pool size, and initialization are useful. If the ImageNet claim were clean, this would be a notable result for interpretable classification. However, the headline ImageNet comparison is confounded by the transductive use of the ImageNet training set as the unlabeled pool, so the significance as stated is not yet established. The non-ImageNet results are still informative, but the paper's strongest claim needs a cleaner experimental test.

major comments (4)
  1. [Implementation Details; Eq. (1)-(3); Table 1] For ImageNet, the unlabeled pool U is randomly sampled from the ImageNet training set. Because U_q is formed by selecting images most similar to each class-name text feature (Eq. 3), the per-class concept codebook is derived from images of the target classes themselves. The ImageNet result (84.1 vs 83.9 for LP, 84.0 for LaBo) is therefore measured in a transductive setting and does not support the claim that concepts are discovered from generic unlabeled web images. Please rerun the ImageNet experiment with a genuinely external unlabeled pool (e.g., a non-overlapping web corpus such as a YFCC or LAION subset) or explicitly restrict U to exclude target-domain images. If the advantage disappears, the conclusion should be revised accordingly.
  2. [Conclusion; Table 1] The conclusion states that V2C-CBM 'has surpassed LLM-guided CBMs across various datasets and even outperformed black-box linear probing methods on the ImageNet dataset,' but this overstates the evidence. In Table 1, Ours is below LaBo on Aircraft (60.7 vs 61.3) and Flower (98.8 vs 99.3), and the ImageNet advantage over LP (84.1 vs 83.9) is 0.2 points with overlapping standard deviations. The abstract's phrase 'matched or outperformed' is supported by the table; the conclusion should be softened to match the actual comparisons.
  3. [Evaluation; Table 16] The method is described as training-efficient, but the final configuration uses per-dataset, per-shot hyperparameters (learning rate, batch size, max epochs, and K = 25 or 50) selected by grid search on validation sets. This is a substantial tuning surface, especially for a method whose concept discovery is claimed to be training-free. Please report the number of grid points, whether the same tuning protocol was applied to the LaBo and linear-probing baselines, and whether the ImageNet result is robust to a single default hyperparameter setting rather than the per-dataset tuned values.
  4. [Concept Filtering; Implementation Details] All experiments use a single source for the unlabeled pool U, namely the ImageNet training set. For ImageNet this is the transductive confound described above; for the other datasets it means the method has been tested with only one, highly diverse auxiliary corpus. The paper claims U 'can be obtained easily from large-scale web images,' but no experiment varies the source of U. Please add at least one experiment with a different external unlabeled source to show that the concept discovery mechanism is not specific to ImageNet statistics.
minor comments (6)
  1. [Method, Concept Vocabulary] There is a typo in 'such aspart of' (missing space after 'as').
  2. [Method, Concept Vocabulary] The phrase 'as exampled in Figure 2' should read 'as exemplified in Figure 2'.
  3. [Method, Concept Filtering, Eq. (4)] The notation sim(F^c_concept, x^q_u) is overloaded: the left side suggests a similarity between a concept feature and a single image, while the right side uses the augmented image set A(x^q_u); please define A and clarify that the maximum or mean over augmentations is used.
  4. [Tables 8 and 14] The abbreviations 'Oursp' and 'Oursr' are used in tables but only defined in the caption of Table 8; please define them in each table caption or in a common notation list.
  5. [Supplement, Table 10] There is a typo in the prompt template: 'a jpeg corupted photo' should be 'a jpeg corrupted photo'.
  6. [Abstract and Introduction] The phrase 'training efficient' should be qualified, since the pipeline requires a 200k-image unlabeled pool and per-dataset hyperparameter search; a sentence on wall-clock time or compute would help readers calibrate the claim.

Circularity Check

1 steps flagged · score 3.0 of 10

Concept 'discovery' reduces to frequency over class-selected images; ImageNet result is confounded by in-distribution unlabeled pool, though final accuracy is not fully tautological.

  1. self definitional [Quantization Unlabeled Image Set (Eq. 3); Concept Filtering; Implementation Details]
    "Specifically, we select the images with highest similarity scores from U for each class k to form U q ... then we save M most frequent concepts for each U q k as the final concept vocabulary ... For the unlabeled image set U, we randomly sample images from the ImageNet training set, and the default number of the unlabeled images is 200k."

    U^q_k is defined by Eq. 3 as the images with the highest cosine similarity to the class-name feature F^base_k. The codebook C_k is then defined as the M most frequent concepts over U^q_k. Consequently, every concept in C_k is, by construction, frequent in the very images chosen for class k; the paper's 'discovery' of interpretable visual concepts (e.g., Table 3) is a deterministic summary of the selection rule rather than an empirical finding. For ImageNet, U is drawn from the target training set, so the concept bottleneck is built from in-distribution class images; the claimed superiority over linear probing and LLM baselines on ImageNet is therefore confounded by this extra in-domain information.

full rationale

The paper's central numerical claim (Table 1, ImageNet 84.1 vs LP 83.9, LaBo 84.0) is measured on held-out test images and the linear head W is trained on labels (Eq. 7), so the classification result does not reduce by identity to the inputs. The few-shot results likewise are real empirical measurements. The main circularity concern is narrower: the 'discovery' of visual concepts is definitional. U^q_k is selected by Eq. 3 as the images closest to the class-name feature, and the codebook C_k is then defined as the most frequent concepts over that very set. Therefore Table 3's top concepts (e.g., 'white' for ice bear) are guaranteed by the selection rule; they are summary statistics, not an out-of-sample discovery. This circularity is amplified for ImageNet by the implementation detail that U is sampled from the ImageNet training set, so the 'auxiliary unlabeled web images' are in fact target-domain images from the same classes being evaluated. That makes the comparison to LP and LLM baselines a transductive/evaluation confound rather than a formal circularity of the loss function. The self-citations (Zhu et al.) are background related work and not load-bearing; no uniqueness theorem is imported. Overall the accuracy pipeline is not equivalent to its inputs, so the paper is only partially circular in its concept-discovery narrative.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical or conceptual entities beyond the algorithmic V2C tokenizer and its codebook, which are standard engineering constructs. The main assumptions are about the semantic validity of CLIP nearest-neighbor matches and the representativeness of the unlabeled image pool; the main tuned choices are detailed in the free parameters list.

free parameters (5)
  • Number of unlabeled images |U| = 200,000
    Default size of the auxiliary unlabeled image set; ablation (Table 4) shows accuracy increases with size, so 200k is a tuned choice.
  • Top-M words per class and concept bottleneck size N_C = M=500, N_C=50
    M=500 most frequent concepts kept per class and 50 concepts per class used in the bottleneck; these are chosen by validation performance.
  • K (concepts per image in tokenizer) = 5
    Number of top concepts used to update word frequency per image; set in Implementation Details.
  • Base vocabulary size = 10,000
    Top 10k English words by frequency; ablation (Table 15) shows robustness to this choice.
  • Per-dataset/per-shot training hyperparameters = varies (Table 16)
    Learning rate, batch size, and max epochs are grid-searched on validation sets for each dataset and shot count, a large set of tuned choices.
assumptions (4)
  • domain assumption CLIP's shared embedding space aligns common English words with visual concepts accurately enough for nearest-neighbor concept retrieval.
    The V2C tokenizer (Eq. 5) and concept filtering (Eq. 4) rely on cosine/Euclidean distances between CLIP text and image features being semantically meaningful.
  • domain assumption The most frequent common words among top-K neighbors of class-related images form a discriminative and sufficient concept vocabulary for classification.
    The entire method is built on frequency-based concept selection; no theoretical guarantee is given.
  • domain assumption Auxiliary unlabeled images sampled from the ImageNet training set are representative for all target datasets, including ImageNet itself.
    U is sampled from ImageNet train split; for ImageNet classification this creates a transductive setup, for other datasets it assumes cross-domain transferability.
  • domain assumption A linear layer over concept similarity scores is capable of matching a linear probe on full CLIP features.
    The CBM head (Eq. 6) restricts the classifier to a sparse concept bottleneck, and the paper's performance parity with LP relies on this assumption holding.

how reviews work

0 comments
Cite this review

Pith. "Pith review of V2C-CBM: Building Concept Bottlenecks with Vision-to-Concept Tokenizer." pith.science (2026). https://pith.science/paper/OMDABTBW

@misc{pith2026250104975,
  author       = {Pith},
  title        = {Pith review of: V2C-CBM: Building Concept Bottlenecks with Vision-to-Concept Tokenizer},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OMDABTBW}},
  note         = {Machine review of arXiv:2501.04975}
}
read the original abstract

Concept Bottleneck Models (CBMs) offer inherent interpretability by initially translating images into human-comprehensible concepts, followed by a linear combination of these concepts for classification. However, the annotation of concepts for visual recognition tasks requires extensive expert knowledge and labor, constraining the broad adoption of CBMs. Recent approaches have leveraged the knowledge of large language models to construct concept bottlenecks, with multimodal models like CLIP subsequently mapping image features into the concept feature space for classification. Despite this, the concepts produced by language models can be verbose and may introduce non-visual attributes, which hurts accuracy and interpretability. In this study, we investigate to avoid these issues by constructing CBMs directly from multimodal models. To this end, we adopt common words as base concept vocabulary and leverage auxiliary unlabeled images to construct a Vision-to-Concept (V2C) tokenizer that can explicitly quantize images into their most relevant visual concepts, thus creating a vision-oriented concept bottleneck tightly coupled with the multimodal model. This leads to our V2C-CBM which is training efficient and interpretable with high accuracy. Our V2C-CBM has matched or outperformed LLM-supervised CBMs on various visual classification benchmarks, validating the efficacy of our approach.

Figures

Figures reproduced from arXiv: 2501.04975 by the authors.

Figure 1
Figure 1. Problems in previous LLM-generated concept bot [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Method overview: (a) construct quantization unlabeled image set U q using class-related base features F k base of class k, (b) adopt the most common words as the base concept vocabulary and use bigrams and trigrams to extend the vocabulary, (c) filter out non-visual and irrelevant concepts using U q to form the codebook for V2C tokenizer, (d) build V2C-CBM with vision-oriented concept bottlenecks generated by V2C to… view at source ↗
Figure 3
Figure 3. Few-shot classification accuracy (x-axis denotes the number of shots and y-axis denotes test accuracy). [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 11 canonical work pages

  1. [1]

    a photo of a {Nk} aircraft

    work page

  2. [2]

    a jpeg corupted photo of a {Nk}

    work page

  3. [3]

    a photo of a large {Nk}

    work page

  4. [5]

    · · · · · · Table 10: Text prompts used for extracting base features

    is a type of {Nk}. · · · · · · Table 10: Text prompts used for extracting base features. Prompt Templates

    work page

  5. [6]

    a jpeg corupted photo of a {Nk} aircraft

    work page

  6. [7]

    a photo of a large {Nk} aircraft

    work page

  7. [8]

    · · · · · · Table 11: Text prompts used for Aircraft dataset

    is a type of {Nk} aircraft. · · · · · · Table 11: Text prompts used for Aircraft dataset. Prompt Templates for DTD

    work page

  8. [9]

    a photo of a {Nk} texture

    work page

Show all 12 references
  1. [10]

    a jpeg corupted photo of a {Nk} texture

  2. [11]

    a photo of a large {Nk} texture

  3. [12]

    · · · · · · Table 12: Text prompts used for DTD dataset

    is a type of {Nk} texture. · · · · · · Table 12: Text prompts used for DTD dataset. Dataset Method number of shots 1 2 4 8 16 LP 51.8 65.3 72.3 77.1 81.6 Average LaBo 63.0 67.7 71.5 75.1 79.3 Ours 58.1 64.1 71.1 75.8 79.7 LP 28.3 35.1 41.6 50.3 56.4 Aircraft LaBo 32.7 37.7 41....

  4. [2020]

    Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsi- ble AI. Inf. Fusion, 58: 82–115. Bach, F. 2010. Convex Analysis and Optimization with Sub- modular Functions: a Tutorial. arXiv:1010.4207. Bossard, L.; Guillaumin, M....

    2023 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.