{"id":"0763f4d8-5304-440a-9429-14d268b3f4ed","arxiv_id":"2505.06003","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"An inherently interpretable image classifier that selects sparse, semantically meaningful superpixel regions per instance and adaptively determines how many regions to keep based on prediction confidence.","lead":"This paper introduces P2P, an image classifier that makes decisions by selecting and masking meaningful image regions (superpixels) instead of individual pixels. The authors show that P2P keeps the accuracy of a full black-box model while highlighting only the parts of the image that matter, and it adapts how much image content it keeps based on its own confidence.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No control rules out the mask-pattern shortcut: P2P's jointly trained classifier could be predicting from the shape of the binary mask rather than the content of the selected regions, which would void the faithfulness claim.","rationale":"P2P's entire contribution rests on the conjunction of meaningful grouping, faithfulness, and accuracy. Of the three, faithfulness is the least secured: the reported fidelity metrics compare the model's prediction on x_m with predictions after inserting/deleting the same selected pixels, but they never test whether the classifier would still make the same prediction if the content of those regions were replaced while the mask shape was held fixed. The paper's own related-work discussion identifies this shortcut for REAL-X, and the missing safeguard makes the concern concrete rather than speculative. The superpixel-semantics concern raised by the reader is real, but even perfectly meaningful superpixels would not save the claim if the classifier uses the mask silhouette as the predictive signal. The proposed content-scramble experiment is a single check that settles the issue. The rest of the evaluation (ten seeds, ablations, BAM controls, code release) is solid and should be credited; the problem is a missing control, not an observed failure, so the verdict remains conditional rather than rejection.","tokens_in":19284,"tokens_out":10341,"duration_ms":119227,"concrete_test":"On COCO-10 and ImageNet-9, take P2P's inferred mask m for each test image and build a scrambled input x'_m by permuting pixel values within each selected region (or copying the selected region content from a different image of a different class), keeping unselected regions zero. Measure top-1 agreement between P2P's predictions on x_m and x'_m. If agreement stays far above chance (e.g., >80%) while the scrambled content is unrecognizable, the classifier is using the mask pattern, not the selected content, and the faithfulness claim is refuted; if agreement collapses to chance, the concern is resolved. As a secondary check, retrain the classifier with REAL-X-style random masking and repeat; if the shortcut disappears, the joint training is the cause.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Sections 5 and 6 claim P2P is inherently interpretable because insertion/deletion fidelity is steep and localization is high (Tables 2-3, Figures 3, 6-7). The load-bearing assumption is that the classifier's prediction is driven by the pixel content inside the selected superpixels. But P2P's classifier is trained jointly with the selector and always receives x_m = m⊙x, so the binary mask pattern itself (which regions are zeroed) is part of the input; a ViT can exploit the silhouette of selected regions as a class cue. The paper itself acknowledges this failure mode for REAL-X in Section 2 ('the selector can encode predictions within its mask') and notes REAL-X's random-masking safeguard, but P2P adopts neither random masking nor any control separating mask shape from region content. Localization and insertion/deletion fidelity cannot distinguish the two mechanisms: a model predicting from a cat-shaped mask still yields high localization and steep insertion/deletion curves even if the pixels inside the mask are irrelevant. If this shortcut is active, the central claim that P2P 'selects meaningful features that faithfully lead to a strong predictive performance' is false: the explanation would be the geometry of the mask, not the image content it highlights.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes P2P, an instance-wise feature-selection method for image classification that makes predictions from a sparse binary mask over superpixel regions. A frozen SLIC proposer partitions each image; a selector predicts per-region selection parameters, modeled jointly through a logit-normal distribution with a covariance matrix built from learnable embeddings; Gumbel-Softmax sampling yields a differentiable binary mask; and a classifier predicts from the masked input. The method uses a thresholded sparsity loss and a dynamic inference-time threshold that increases the retained fraction until classifier certainty exceeds a user-set value. Experiments on CIFAR-10, ImageNet, ImageNet-9, COCO-10, and the BAM datasets compare accuracy, localization, and insertion/deletion fidelity against DiET, REAL-X, RB-AEM, B-cos, COMET, and control baselines. The paper reports accuracy comparable to a black-box classifier, the best localization on COCO-10 and ImageNet-9, and the steepest fidelity curves, and concludes that P2P is inherently interpretable.","tokens_in":19472,"tokens_out":11062,"duration_ms":115773,"significance":"If the central claims are borne out, P2P is a useful contribution: it demonstrates that grouping pixels into regions and modeling their joint selection can produce sparse, object-focused masks with accuracy competitive with full-image classifiers, and the dynamic thresholding idea directly addresses the fixed-sparsity limitation of prior feature-selection methods. The evaluation is thorough in several respects: results are reported over ten seeds with standard deviations, code is released, the COMET-1 control gives a concrete faithfulness check for COMET, the BAM semi-synthetic datasets provide a localization check with known ground truth, and ablations cover the superpixel algorithm and the certainty threshold. The proof of positive semi-definiteness of the covariance in Appendix A is elementary and correct. However, the faithfulness evidence does not exclude a mask-pattern shortcut, and the perceptual meaningfulness of the superpixel grouping is asserted rather than validated; both issues bear directly on the headline interpretability claim.","major_comments":[{"comment":"The claim that P2P faithfully uses the selected regions is not established because there is no control for the mask-pattern shortcut. Since the classifier is trained jointly with the selector and always receives x_m = m⊙x, it can predict from the spatial arrangement of zero and nonzero regions; a ViT can exploit the silhouette of the mask as a class cue. Insertion and deletion fidelity (Figures 3 and 7) cannot separate content-based from pattern-based prediction because both mechanisms make the curves steep. The paper itself identifies this failure mode for REAL-X in Section 2 and notes that REAL-X counters it with random masking, but P2P adopts no analogous safeguard. Please add one or more of the following controls and report results: (i) train or evaluate a classifier on the binary mask alone; (ii) replace the contents of the selected regions with class-independent noise or a constant and measure accuracy; (iii) train P2P with random masking and compare. If either control shows that the mask pattern alone drives accuracy, the central faithfulness claim in Sections 5 and 6 should be withdrawn or substantially weakened.","section":"Section 5, Figures 3 and 7; Section 2"},{"comment":"The premise that SLIC superpixels are 'perceptually meaningful atomic regions' is not validated. Section 3 relies on this premise to argue that region-level masking yields human-understandable explanations, but the only related experiment, Appendix B Table 5, swaps SLIC for Watershed and finds similar accuracy and localization; this tests algorithm invariance, not whether the regions correspond to the parts humans use for recognition. Please provide a direct check, for example overlap with human part or object annotations or a user study comparing the interpretability of P2P masks against pixel-level masks, or explicitly narrow the claim to 'contiguous regions' rather than 'perceptually meaningful parts.'","section":"Section 3; Appendix B, Table 5"}],"minor_comments":[{"comment":"On BAM Scene, COMET achieves slightly higher localization (98.49 ± 0.63) than P2P (98.18 ± 0.27); the sentence 'Clearly, P2P also excels in this controlled setup' should acknowledge this exception or report the numbers explicitly.","section":"Appendix B, Table 3"},{"comment":"The statement that P2P removes 'up to 80%' of image content is an average over certain datasets; for ImageNet the average masked fraction is about 52% (Table 4). Please make the dataset-specific nature of this claim explicit.","section":"Section 5; Table 4"},{"comment":"The claimed benefit of the logit-normal covariance for selection is not isolated. Please add an ablation with a diagonal covariance, or no covariance, to show how much of the accuracy or localization gain comes from modeling part relationships.","section":"Section 3; Section 5"},{"comment":"The sparsity levels τ reported in Table 1 appear to be rounded averages of P2P's dynamic masks (compare Table 4), while the text describes them as fixed budgets. Please clarify how τ is set for P2P and for each baseline so the comparison is unambiguous.","section":"Section 4; Table 1"}],"recommendation":"major_revision","confidential_remarks":"The missing mask-pattern control is the deciding issue for the paper's central faithfulness claim. The rest of the empirical work is solid and the method is well within scope; I would be prepared to accept after the authors either add the proposed controls or explicitly restrict the claim to exclude pattern-based prediction."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: P2P is a genuinely useful entry in the interpretable-ML toolbox. Grouping features into superpixel regions and adding dynamic thresholding is a solid, practical idea, and the empirical work is better than what this subfield usually ships. But the paper's headline claim—that P2P is inherently interpretable because its predictions are faithfully driven by the selected regions—is not actually demonstrated. The reason is the mask-pattern shortcut.\n\nWhat's new and good: the method couples a frozen superpixel proposer with a Gumbel-Softmax selector over region-level logit-normal probabilities, and it learns a part-covariance that is visualized. The dynamic thresholding is a real addition: during training τ is sampled and fed to the model, and at inference the mask is grown until the classifier reaches confidence δ. Clean way to handle instance-varying sparsity. The experiments are broad—CIFAR-10, ImageNet, ImageNet-9, COCO-10, plus the BAM semi-synthetic sets—with ten seeds and error bars. Accuracy is close to the black-box upper bound, localization beats the baselines, and the insertion/deletion curves are steep. The ablation replacing SLIC with Watershed is a nice touch; it shows the specific superpixel algorithm is not the source of the gains. The code is linked, which also helps.\n\nSoft spots. The big one is that there is no control separating mask shape from region content. The classifier is trained jointly with the selector and always sees the binary mask applied to the input. A ViT can pick up on the silhouette of kept regions, and if that is what drives the prediction, then the selected regions are not the actual explanation. The paper cites REAL-X for exactly this failure mode—'the selector can encode predictions within its mask'—but then does not adopt REAL-X's random-masking safeguard, nor any alternative control. Localization and fidelity metrics cannot distinguish the two mechanisms. A simple experiment would settle this: take each P2P mask and fill the selected regions with random or blurred content, or with content from another image, and measure how much the prediction changes relative to the original mask. If the mask pattern alone carries the information, the faithfulness claim collapses. Without that, 'inherently interpretable' is a belief, not a result.\n\nMinor issues. The dynamic thresholding is underspecified: how τ is fed to the selector, and the step size of the inference-time search, are not fully described. The localization metric is only a proxy and the authors acknowledge it. The superpixel-meaningfulness assumption is untested, though the Watershed ablation reduces the concern.\n\nBottom line: the method is worth publishing, but the central interpretability claim overreaches as written. I'd send it to reviewers and ask for the content-scramble control. That is an addressable fix, not a redo.","headline":"A useful region-based feature-selection method whose faithfulness claim lacks a control for the mask-shape shortcut—worth reviewing, needs one fix.","tokens_in":20078,"tokens_out":3031,"would_cite":true,"duration_ms":29519,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Region-based image masking yields interpretable predictions that rival black-box accuracy.","keywords":["inherent interpretability","instance-wise feature selection","superpixels","binary masking","dynamic thresholding","faithfulness","insertion fidelity","image classification"],"falsifier":"Run P2P with a region proposer whose regions are random contiguous blobs of the same size distribution as the superpixels. If accuracy, localization, and insertion fidelity stay essentially unchanged, then the semantic meaningfulness of the regions is not doing the work; if they collapse, the superpixel premise is confirmed. Alternatively, a human annotation study can check whether the regions P2P keeps align with human object-part segmentations on the same images.","tokens_in":18997,"feed_emoji":"🔍","tokens_out":6824,"duration_ms":65133,"temperature":0.7,"pith_summary":"P2P is a classifier that explains its own prediction by masking out most of the image, but it does so over superpixel regions rather than individual pixels. The paper argues that pixel-level sparsification is meaningless because evenly spaced or darkening masks preserve accuracy without removing information, whereas regions correspond to perceptual parts a human can grasp. P2P learns which regions to keep per instance, models relationships among parts, and dynamically lets the classifier request more regions until it reaches a user-set confidence level. The reported results aim to show that these region masks keep accuracy near a full black-box model while removing up to 80% of image content, locate target objects better than prior feature-selection methods, and are faithful in the sense that predictions actually depend on the kept regions.","feed_headline":"Superpixel masking matches black-box accuracy while hiding 80% of an image","feed_subtitle":"P2P learns to keep only the object parts a human would point to, and asks for more pixels only when it is unsure.","key_machinery":"The machine is a region-level binary mask over SLIC superpixels. For each image, a frozen superpixel proposer partitions pixels into $D$ regions, an importance predictor outputs per-region logit-normal parameters $\\mu$ and covariance $\\Sigma$, with $\\Sigma_{jk}=E_j\\cdot E_k$ for learnable region embeddings so it is positive semi-definite; a Gumbel-Softmax sample binarizes the mask while remaining differentiable, and a thresholded sparsity loss $\\mathcal{L}_m = -\\log(1-\\bar p)$ for $\\bar p>\\tau$ controls how many pixels remain. Dynamic thresholding at inference sets $\\tau$ as the smallest value at which classifier confidence reaches $\\delta$, making sparsity instance-specific.","core_discovery":"The central claim is that inherently interpretable image predictions can be obtained by instance-wise grouped feature selection in the space of semantically meaningful regions. To carry this out, P2P freezes a superpixel proposer, predicts a selection probability per region from pixel-level features, draws a binary mask with the Gumbel-Softmax trick, and classifies only the unmasked regions; a non-diagonal logit-normal distribution over region probabilities encodes relationships among parts. Rather than fixing one sparsity level, P2P samples thresholds during training and at inference increases the threshold stepwise until the classifier's confidence reaches $\\delta$, so the model effectively asks for more evidence when needed. On CIFAR-10, COCO-10, ImageNet, and ImageNet-9 the paper reports accuracy comparable to the upper-bounding black-box while retaining only 20–50% of pixels depending on dataset, the best localization among baselines on COCO-10 and ImageNet-9, and the steepest insertion and deletion fidelity curves, which it reads as evidence that predictions are based on the selected regions and hence inherently interpretable.","pith_inferences":["The paper never directly tests whether the superpixel proposer matches human part decomposition; a user study asking humans to name or segment the retained regions would settle whether the interpretability claim is perceptual or only structural.","Because the paper's ablation only swaps SLIC for Watershed, it does not separate grouping from semantic meaning; comparing P2P against random contiguous region partitions of matched granularity would isolate whether the regions' content, not just their contiguity, drives the gains.","The same grouped-feature-selection recipe should transfer to audio or text, where atomic segments such as phonemes, words, or spans play the role of superpixels, offering a direct test of the method's generality.","The embedding-based covariance could be read as a part-object scene graph; clustering the embeddings instead of coloring them would let P2P output named part groupings, an extension the paper only gestures at."],"forward_implications":["Pixel-level instance-wise feature selection is shown to be insufficient: evenly spaced masks that remove 80% of pixels keep near-black-box accuracy, so optimization must be carried out over regions to enforce informative sparsity.","If P2P's results hold, an interpretable classifier can be both faithful and accurate, which weakens the usual accuracy-interpretability trade-off for this setup.","Dynamic thresholding gives users a direct control: lowering the confidence threshold $\\delta$ produces sparser explanations, and raising it yields more evidence, so sparsity can be tuned per application without retraining.","Steep insertion and deletion curves mean faithfulness can be demonstrated empirically for masked-input models, providing a template for evaluating future inherently interpretable vision methods.","Visualizing the kept regions exposes potential dataset shortcuts such as pillows or teddy bears signaling 'bed', making the method a tool for detecting spurious correlations in datasets."],"supporting_citations":[{"why":"Supplies the SLIC superpixel region proposer that defines the atomic features P2P masks.","marker":"Achanta et al. (2012)"},{"why":"Introduces instance-wise feature selection and motivates maximizing likelihood under sparsity, the objective P2P reformulates.","marker":"Chen et al. (2018)"},{"why":"REAL-X baseline; also warns that selectors can encode predictions in masks, which P2P's region masking is designed to avoid.","marker":"Jethani et al. (2021)"},{"why":"COMET baseline with continuous-valued masks; its inverted-mask variant motivates P2P's binary masking and faithfulness analysis.","marker":"Zhang et al. (2025)"},{"why":"DiET baseline whose gradient-based mask convergence is compared on real-world datasets.","marker":"Bhalla et al. (2024)"},{"why":"RB-AEM baseline introducing a geometric prior for pixel selection, compared for localization and fidelity.","marker":"Ganjdanesh et al. (2022)"},{"why":"Defines inherent interpretability as faithful by design, the property P2P claims to satisfy.","marker":"Rudin (2019)"},{"why":"Provides the 'inherent interpretability is a claim until proven otherwise' framing that motivates fidelity evaluation.","marker":"Jacovi & Goldberg (2020)"},{"why":"Supplies insertion and deletion fidelity metrics used to measure whether predictions depend on highlighted regions.","marker":"Petsiuk (2018)"},{"why":"Provides the semi-synthetic BAM datasets used to measure localization in a controlled setting without confounding co-occurring features.","marker":"Yang & Kim (2019)"}],"fun_headline_variants":["Superpixel masks match black-box accuracy, hide up to 80%","Adaptive superpixel masks: black-box accuracy with 80% fewer pixels","P2P: keep only key object parts, ask for more when unsure","Instance-wise superpixel selection: match black-box, stay interpretable","Trim pixels to the meaningful few—P2P keeps accuracy high"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"That SLIC superpixels partition an image into perceptually meaningful atomic regions, so that switching a region off removes one understandable feature; if superpixels instead split or merge object parts arbitrarily, P2P's masks would be no more meaningful than pixel masks.","fun_headline_variants_meta":{"raw":{"variants":["Superpixel masks match black-box accuracy, hide up to 80%","Adaptive superpixel masks: black-box accuracy with 80% fewer pixels","P2P: keep only key object parts, ask for more when unsure","Instance-wise superpixel selection: match black-box, stay interpretable","Trim pixels to the meaningful few—P2P keeps accuracy high"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001057,"raw_usage":{"total_tokens":4405,"prompt_tokens":887,"completion_tokens":3518,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":503,"completion_tokens_details":{"reasoning_tokens":3419}},"tokens_in":503,"tokens_out":3518,"duration_ms":24851,"temperature":1.0,"reasoning_tokens":3419,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:51:04.921899+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run P2P with a region proposer whose regions are random contiguous blobs of the same size distribution as the superpixels. If accuracy, localization, and insertion fidelity stay essentially unchanged, then the semantic meaningfulness of the regions is not doing the work; if they collapse, the superpixel premise is confirmed. Alternatively, a human annotation study can check whether the regions P2P keeps align with human object-part segmentations on the same images.","supporting_citations":[{"cited_title":"Have we learned to explain?: How interpretability methods can learn to encode predictions in their interpretations","cited_arxiv_id":null,"evidence_quote":"REAL-X baseline; also warns that selectors can encode predictions in masks, which P2P's region masking is designed to avoid."},{"cited_title":"Comprehensive attribution: Inherently explainable vision model with feature detector","cited_arxiv_id":null,"evidence_quote":"COMET baseline with continuous-valued masks; its inverted-mask variant motivates P2P's binary masking and faithfulness analysis."},{"cited_title":"Discriminative feature attributions: bridging post hoc explainability and inherent interpretability","cited_arxiv_id":null,"evidence_quote":"DiET baseline whose gradient-based mask convergence is compared on real-world datasets."},{"cited_title":"Interpretations steered network pruning via amortized inferred saliency maps","cited_arxiv_id":null,"evidence_quote":"RB-AEM baseline introducing a geometric prior for pixel selection, compared for localization and fidelity."}],"review_version":1}