{"id":"1bc09116-564c-4b57-b723-c98ab9c64613","arxiv_id":"2606.19882","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"MM-CBM adds dual concept bottleneck layers to CLIP to enable interpretable multimodal vision tasks, reporting up to 51.26% average accuracy gains over prior CBMs across four benchmarks.","lead":"The paper introduces Multimodal Concept Bottleneck Models that add dual concept bottleneck layers to CLIP embeddings for both images and text. This aims to support interpretable zero-shot classification and retrieval while keeping accuracy near black-box levels.","discovery_kind":"unclear","skeptic_critique":{"model":"grok-4.3","headline":"Dual CBLs may permit non-concept leakage if concept vocabulary is insufficient to span CLIP embedding variance","rationale":"The reader's weakest assumption directly identifies the same point. Because the review was performed on the abstract alone, the full manuscript would be needed to confirm whether the authors already performed the leakage or ablation checks; absent those checks the empirical claims remain conditional on the alignment assumption holding.","tokens_in":1657,"tokens_out":351,"duration_ms":20707,"concrete_test":"Extract the exact concept vocabulary and CBL weight matrices from the released code or appendix; recompute zero-shot classification accuracy on one benchmark (e.g., the one showing the 51.26 % gain) after (a) freezing the CBL and (b) replacing the CBL with an identity map of identical output dimension. If accuracy under (b) exceeds accuracy under (a) by more than the reported ~5 % margin, leakage is occurring.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline performance claim (51.26 % avg. gain, within ~5 % of black-box) and the zero-shot interpretability claim both rest on the assertion that the two CBLs map image and text embeddings onto a shared, fixed concept basis while discarding all other signals. If the chosen concept set is incomplete relative to the variance present in CLIP embeddings for a given benchmark, residual non-concept dimensions can still be exploited by the downstream classifier or retrieval head. The abstract gives no indication that the authors measured mutual information between the post-CBL activations and the original embeddings, nor that they performed a controlled ablation that isolates concept-only versus full-embedding performance on the zero-shot tasks.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript proposes the Multimodal Concept Bottleneck Model (MM-CBM) that extends standard Concept Bottleneck Models to CLIP by introducing dual Concept Bottleneck Layers (CBLs). These layers map both image and text embeddings onto a shared set of predefined concepts, enabling interpretable zero-shot classification and image retrieval while claiming to mitigate non-concept information leakage. Across four standard benchmarks the method is reported to deliver up to 51.26 % average accuracy improvement relative to prior CBMs and to remain within ~5 % of black-box CLIP performance.","tokens_in":1800,"tokens_out":452,"duration_ms":12383,"significance":"If the dual-CBL construction demonstrably discards all non-concept signals and the reported gains survive standard controls for concept selection and statistical significance, the work would meaningfully extend the CBM paradigm into the multimodal zero-shot regime. The combination of interpretability with competitive zero-shot retrieval and classification would be a useful addition to the vision-language literature.","major_comments":[{"comment":"Abstract: the headline claim of 51.26 % average accuracy improvement and the assertion that dual CBLs prevent non-concept leakage are presented without any description of concept vocabulary construction, training procedure, or error bars. Because these details are load-bearing for both the performance and the interpretability claims, the abstract alone does not allow verification that the gains survive ordinary controls.","section":"Abstract"},{"comment":"The central modeling assumption (that the chosen concept set spans essentially all variance in the CLIP embeddings) is not accompanied by any reported measurement of mutual information between post-CBL activations and the original embeddings, nor by an ablation that isolates concept-only versus full-embedding performance on the zero-shot tasks. Without such evidence the leakage concern raised in the skeptic note remains unaddressed.","section":"Method / Experiments"}],"minor_comments":[{"comment":"Notation for the two CBLs and the shared concept basis should be introduced with explicit equations rather than prose only.","section":"Method"},{"comment":"The four benchmarks are not named in the abstract; a table listing per-dataset numbers, concept counts, and black-box baselines would improve clarity.","section":"Experiments"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback. We respond point-by-point to the major comments below, agreeing to revisions that improve clarity and evidence while defending the core methodological choices where they are supported by the manuscript.","responses":[{"response":"We agree the abstract is too concise for standalone verification. In the revision we will add a brief clause describing the concept vocabulary (derived from standard vision-language datasets aligned via CLIP text encoder) and the dual-CBL training objective. Error bars from repeated runs will be reported for all accuracy figures, and the 51.26 % figure will be explicitly defined as the mean relative improvement over prior CBM baselines across the four benchmarks.","revision_made":"yes","referee_comment":"[Abstract] Abstract: the headline claim of 51.26 % average accuracy improvement and the assertion that dual CBLs prevent non-concept leakage are presented without any description of concept vocabulary construction, training procedure, or error bars. Because these details are load-bearing for both the performance and the interpretability claims, the abstract alone does not allow verification that the gains survive ordinary controls."},{"response":"The dual-CBL construction projects both image and text embeddings onto an explicit shared concept space, which removes non-concept dimensions by design rather than by post-hoc filtering. While mutual-information statistics were not computed in the submitted version, the observed performance gap of only ~5 % relative to black-box CLIP on zero-shot tasks supplies supporting evidence that task-relevant information is retained. We will add an ablation comparing concept-bottleneck versus full-embedding performance and include mutual-information measurements between post-CBL and original embeddings in the revised experiments section.","revision_made":"partial","referee_comment":"[Method / Experiments] The central modeling assumption (that the chosen concept set spans essentially all variance in the CLIP embeddings) is not accompanied by any reported measurement of mutual information between post-CBL activations and the original embeddings, nor by an ablation that isolates concept-only versus full-embedding performance on the zero-shot tasks. Without such evidence the leakage concern raised in the skeptic note remains unaddressed."}],"tokens_in":1343,"tokens_out":459,"duration_ms":31628,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The punchline is that MM-CBM adds dual concept bottleneck layers to CLIP to support interpretable zero-shot classification and retrieval, and it reports keeping performance close to the black box while beating prior CBMs by a lot.\n\nWhat is new is the application of CBMs to a multimodal backbone with separate layers for vision and language embeddings. The paper does well in demonstrating that this can be done without a huge drop in accuracy on standard benchmarks. The 51% average improvement and the ~5% gap to black-box are the kind of numbers that would get attention if the experiments are solid.\n\nThe soft spots are around the details that are missing from the abstract. Concept selection, how the bottlenecks are trained, and whether statistical significance was checked are not described. This makes it difficult to judge if the gains are robust or if they depend on particular choices. The stress-test concern about non-concept leakage is a fair one to raise; without measurements like mutual information between post-bottleneck features and original embeddings, it's possible that some predictive power comes from outside the intended concepts. The work also seems to skip citing any prior multimodal CBM efforts, which makes the novelty look more incremental than it might be.\n\nThis paper is for people who want to add interpretability to CLIP-style models for tasks like zero-shot classification. A reader interested in practical interpretability methods for vision-language models would get value from seeing how the dual layers are implemented and what the results look like on retrieval as well as classification.\n\nI would recommend sending it to peer review. The core idea is testable and addresses a real need, even if the current presentation leaves some questions about the strength of the evidence.","headline":"MM-CBM extends CBMs to CLIP via dual bottlenecks but needs more method details and leakage checks to back the big claims.","tokens_in":2299,"tokens_out":416,"would_cite":false,"duration_ms":28000,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Multimodal Concept Bottleneck Models add dual layers to CLIP so image and text embeddings align to shared concepts for interpretable zero-shot tasks.","keywords":["concept bottleneck models","multimodal learning","CLIP","interpretability","zero-shot classification","image retrieval","vision-language models"],"falsifier":"A test set of zero-shot classification or retrieval examples whose correct answers require concepts absent from the predefined set; if MM-CBM accuracy falls more than five percent below black-box performance or loses interpretability, the alignment claim fails.","tokens_in":2560,"feed_emoji":"","tokens_out":624,"duration_ms":20615,"temperature":0.7,"pith_summary":"The paper addresses two limits of standard Concept Bottleneck Models: they cannot generalize past a fixed set of classes and they risk using predictive signals outside the intended concepts. MM-CBM places dual Concept Bottleneck Layers inside CLIP to map both image and text embeddings onto the same predefined concepts. This change supports zero-shot classification and image retrieval that remain traceable to the chosen concepts. On four benchmarks the resulting model improves accuracy by as much as 51.26 percent on average while staying within roughly five percent of ordinary black-box performance.","feed_headline":"Dual concept layers let CLIP handle interpretable zero-shot tasks","feed_subtitle":"MM-CBM aligns image and text embeddings to shared concepts, lifting accuracy up to 51% while staying near black-box levels.","key_machinery":"Dual Concept Bottleneck Layers that project CLIP image and text embeddings into a shared space of predefined concepts.","core_discovery":"MM-CBM utilizes dual Concept Bottleneck Layers to align both the image and text embeddings into interpretable features. This allows us to perform new vision tasks like zero-shot classification or image retrieval in an interpretable way. Compared to existing methods, MM-CBM achieves up to 51.26% accuracy improvement on average across four standard benchmarks while maintaining high accuracy, staying within ~5% of black-box performance.","pith_inferences":["The same dual-layer pattern could be inserted into other vision-language models that produce separate image and text embeddings.","If the alignment holds, the set of concepts could be grown after training without retraining the backbone.","Direct measurement of concept leakage on held-out tasks would give a quantitative check on the no-leakage premise."],"forward_implications":["Zero-shot classification decisions become traceable to individual concept activations.","Image retrieval can rank results by matching concept vectors rather than raw embeddings.","Accuracy on standard benchmarks rises by up to 51 percent relative to earlier CBMs.","Performance remains within five percent of non-interpretable CLIP baselines."],"fun_headline_variants":["MM-CBM aligns CLIP with dual concept layers for zero-shot","Dual CBLs align CLIP embeddings for interpretable zero-shot vision","MM-CBM adds concept bottlenecks to CLIP for interpretable tasks","Interpretable zero-shot via MM-CBM in CLIP embeddings"],"cache_read_input_tokens":64,"weakest_assumption_plain":"The dual concept bottleneck layers align image and text embeddings to shared concepts without significant information leakage or loss of generalization beyond the training concepts.","fun_headline_variants_meta":{"raw":{"variants":["MM-CBM aligns CLIP with dual concept layers for zero-shot","Dual CBLs align CLIP embeddings for interpretable zero-shot vision","MM-CBM adds concept bottlenecks to CLIP for interpretable tasks","Interpretable zero-shot via MM-CBM in CLIP embeddings"]},"model":"grok-4.3","cost_usd":0.005515,"raw_usage":{"total_tokens":2624,"prompt_tokens":621,"num_sources_used":0,"completion_tokens":74,"cost_in_usd_ticks":55149500,"prompt_tokens_details":{"text_tokens":621,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1929,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":621,"tokens_out":74,"duration_ms":14411,"temperature":1.0,"reasoning_tokens":1929,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-26T18:10:20.158095+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A test set of zero-shot classification or retrieval examples whose correct answers require concepts absent from the predefined set; if MM-CBM accuracy falls more than five percent below black-box performance or loses interpretability, the alignment claim fails.","supporting_citations":[],"review_version":1}