{"id":"d4e5ea0d-7131-4b9b-afe1-5a1c217dad91","arxiv_id":"2501.13818","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Using concept vectors as bias representations, the framework semi-automatically labels and localizes spurious artifacts and partially unlearns them, though success varies by architecture and artifact type.","lead":"This paper extends the Reveal2Revise interpretability workflow with semi-automated bias annotation: concept vectors retrieve medical images or ECG traces that contain spurious artifacts and produce spatial masks for those artifacts. Tests on melanoma, endoscopy, chest X-ray, and ECG models show the workflow reliably detects and often reduces artifact reliance, with weaker mitigation for Vision Transformers and for diffuse artifacts.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ViT mitigation is unsupported: Appendix Table 11 shows CAV-based methods leave ViT biased-test accuracy near Vanilla (e.g., HyperKvasir 0.42→0.42), and the only substantial ViT gain uses RRR with ground-truth masks, not the proposed semi-automated pipeline.","rationale":"The reader's weakest_assumption concerned the linear-CAV representation; that is a genuine limitation, acknowledged in Sec. 7, and the retrieval/localization experiments show it often holds in practice. A more directly falsifiable gap is the architecture coverage in the central claim. The abstract names ViT as a success, but the only evidence in the appendix shows that the paper's own CAV-based mitigation methods are ineffective for ViT, while the method that works (RRR with ground-truth masks) is not the semi-automated contribution and is not evaluated with the automatically computed masks for ViT. This is a mismatch between the headline claim and the reported table, not a matter of style or consensus. If a rerun with automated ViT masks shows significant biased-accuracy gains, the concern dissolves; if not, the abstract's 'successfully mitigate ... ViT' assertion is false as stated. I would keep CONDITIONAL rather than REJECT because the CNN mitigation results, retrieval AUROCs, and localization metrics are substantial and reproducible, and the required fix is to narrow the architecture claim or supply the missing ViT experiment with automated masks.","tokens_in":35171,"tokens_out":5233,"duration_ms":48627,"concrete_test":"Run the three controlled ViT mitigation experiments (ISIC2019-microscope, HyperKvasir-timestamp, CheXpert-brightness) end-to-end with 5 random seeds, feeding the paper's SHAP/SLIC-based automated localization masks into RRR instead of ground-truth masks, and report mean ± 95% CI for clean and biased accuracy. Also report biased accuracy for RR-ClArC, P-ClArC, and rP-ClArC with the same seeds. If no method from the proposed pipeline exceeds Vanilla biased accuracy by more than 0.05 on any dataset, the mitigation claim should be restricted to VGG16/ResNet50 and ViT reported as a negative result.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The abstract's central claim is that the extended framework 'successfully identify[ies] and mitigat[es] these biases in VGG16, ResNet50, and contemporary Vision Transformer models.' The main-text mitigation table (Tab. 1) reports only ResNet50. The ViT results in Appendix A.7.4 (Tab. 11) show that the paper's focus methods—RR-ClArC, P-ClArC, rP-ClArC—hardly move biased-test accuracy for ViT: ISIC2019-microscope clean/biased 0.81/0.27 (Vanilla) vs 0.81/0.31 (RR-ClArC); HyperKvasir-timestamp 0.93/0.42 vs 0.93/0.42; CheXpert-brightness 0.80/0.25 vs 0.80/0.31. The only ViT method that substantially improves biased accuracy is RRR with ground-truth masks (ISIC2019 0.27→0.59; HyperKvasir 0.42→0.54), which requires manual pixel-level masks and is not the proposed semi-automated CAV annotation pipeline; RRR with automated masks is not reported for ViT. Moreover, RR-ClArC drives ViT TCAVsens to 0.00 while biased accuracy stays flat, showing that erasing a CAV direction can zero the sensitivity metric without improving robustness. Thus the strong claim of successful ViT mitigation is currently unsupported: at most, the framework detects biases in ViTs and mitigates them only when given ground-truth masks.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reviews and extends the Reveal2Revise framework for detecting and mitigating spurious correlations in medical deep learning models. The extension adds semi-automated bias annotation: CAV-based biased-sample retrieval, iterative refinement of the bias representation, and spatial bias localization, followed by mitigation with RRR and ClArC variants. The authors evaluate on four medical datasets across two modalities (image and ECG) using VGG16, ResNet50, and ViT architectures, with both real-world and artificially controlled artifacts. The central claim is that the extended framework successfully identifies and mitigates these biases across all three vision architectures.","tokens_in":35532,"tokens_out":4137,"duration_ms":40199,"significance":"The paper has clear practical value if the claims are properly scoped. The strengths are real: controlled artifacts with ground-truth masks, externally anchored artifact labels for real-world biases, quantitative retrieval and localization evaluations on multiple layers and architectures, and released code and annotations. The bias identification and retrieval results are largely convincing and would be useful to practitioners. However, the headline claim of successful mitigation in ViTs is not supported by the reported numbers, and part of the mitigation evaluation is circular because the TCAV/artifact-relevance metrics are computed along the same CAV direction that the mitigation methods erase. These issues weaken the central claim as written, although they are addressable by revising the claims and adding independent evaluation.","major_comments":[{"comment":"The claim that the framework \"successfully identify[ies] and mitigat[es] these biases in VGG16, ResNet50, and contemporary Vision Transformer models\" is not supported for ViTs. In Table 11, the CAV-based methods RR-ClArC, P-ClArC, and rP-ClArC leave biased-test accuracy essentially unchanged: HyperKvasir stays at 0.42 for RR-ClArC, ISIC2019 moves only from 0.27 to 0.31, and CheXpert from 0.25 to 0.31. The only substantial ViT improvement comes from RRR with ground-truth masks (ISIC2019 0.27→0.59, HyperKvasir 0.42→0.54), which requires manual pixel-level annotations and is not the proposed semi-automated CAV annotation pipeline. The abstract and conclusions should be revised to claim ViT detection and localization, with mitigation demonstrated only when ground-truth masks are provided.","section":"Abstract; §6.5; Appendix A.7.4 (Table 11)"},{"comment":"The mitigation evaluation is partly circular: TCAV sensitivity and artifact relevance are computed along the same CAV direction that RR-ClArC and P-ClArC explicitly erase. Table 11 shows RR-ClArC driving ViT TCAVsens to 0.00 while biased accuracy remains flat, which indicates that the metric may be measuring removal of the direction rather than behavioral robustness. The paper should include an independent evaluation, such as a CAV computed from a held-out layer or annotation source, or behavioral insertion/deletion tests, before claiming successful ViT mitigation.","section":"§6.5; Appendix A.7.4 (Eq. 12, Table 11)"},{"comment":"The proposed semi-automated feature-level annotations are not actually connected to the main mitigation results. Table 1 in the main text uses RRR with ground-truth masks, and the automated heatmap and binarized masks appear only in the appendix with substantially smaller gains: for ResNet50 on ISIC2019, biased accuracy is 0.44 (hm) and 0.43 (bin) versus 0.51 with ground truth, and for HyperKvasir 0.77 versus 0.82. Automated masks are not reported for ViT at all. The claim that the feature-level annotation pipeline reduces labeling effort in the full mitigation loop therefore needs to be either demonstrated with automated masks in the main results or explicitly qualified.","section":"§6.5; Appendix A.7.4 (Tables 9 and 10)"},{"comment":"The spatial localization results are weaker than the retrieval results: for the microscope artifact, IoU is below 0.16 for all layers and architectures because the models attend only to the border of the circle. The paper acknowledges this in the limitations section, but the low IoU has direct consequences for the automated-mask mitigation pipeline discussed in the previous comment. The authors should state more explicitly that RRR with automated masks is currently only practical for well-localized artifacts such as the timestamp, not for artifacts where the model's concept representation differs from the human-annotated region.","section":"§6.4; Appendix A.7.3 (Table 7)"}],"minor_comments":[{"comment":"There are several typos and small errors, including \"n a medical context\" and \"sub-popluations\" in §2, \"overivew\" in Appendix A.1, \"miscroscope\" in Table 7, \"clearn dataset\" in Appendix A.7.4, \"perspetive\" in Appendix A.7.3, and \"in int terms\" in Appendix A.7.4. These should be corrected.","section":"Throughout"},{"comment":"The retrieval plots would be easier to read if the best-performing layer per artifact were marked directly on the figure, since the text refers to layer choice as crucial and Figures 7 and 9 show that the optimal layer differs between retrieval and localization.","section":"§6.3; Figure 7"},{"comment":"Equation (2) writes R_l(x) = a_l(x) ∘ h_l without specifying whether a_l is the full spatial activation tensor or a pooled representation; because the paper uses max-pooled 1D activations for CAVs, the dimensions of the element-wise product should be stated explicitly.","section":"§3.3; Eq. (2)"},{"comment":"The limitation paragraph on layer selection is well placed, but since the paper identifies layer choice as an additional hyperparameter and shows that retrieval and localization prefer different layers, a short paragraph in the main text on how a practitioner would choose the layer in a real deployment would strengthen the practical contribution.","section":"§7; Bias Modeling"}],"recommendation":"major_revision","confidential_remarks":"The paper's core detection and retrieval contributions are solid, and the code and controlled ground truths are valuable. The main issue is that the abstract and conclusions overstate the mitigation results for ViT models; the appendix data directly contradict the headline claim. A revision that narrows the claims and adds a non-circular mitigation evaluation would make the paper suitable for publication. The paper fits the scope of a venue interested in interpretability and safe medical AI."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know about this paper is that the detection and retrieval part is genuinely useful, but the mitigation claim in the abstract is stronger than what the appendix shows. The authors extend Reveal2Revise with CAV-based sample retrieval and spatial localization, and they evaluate it on four medical datasets across image and ECG data. The retrieval experiments are the strongest part: CAVs generally beat single neurons for ranking artifact samples, with high AUROC/AP on both real-world and controlled artifacts. The localization results are honest—they report IoU and artifact relevance, and they acknowledge that for the microscope artifact the models only capture the border of the circle. Code and annotations are released, which is real value.\n\nThe soft spots are in mitigation and in the scope of the claims. The main table only reports ResNet50. The ViT results in Appendix Table 11 show that the proposed CAV-based methods (RR-ClArC, P-ClArC, rP-ClArC) barely move biased-test accuracy for ViT—HyperKvasir goes 0.42 to 0.42, ISIC2019 0.27 to 0.31. The only substantial ViT gain comes from RRR with ground-truth masks, which is not the semi-automated pipeline the paper is proposing. So the abstract's claim of successful mitigation in 'contemporary Vision Transformer models' is not supported. Also, the mitigation evaluation is partly circular: TCAV and artifact relevance are computed along the same CAV direction that the mitigation methods erase, so driving TCAV to zero can happen without improving robustness, which is exactly what happens for ViT.\n\nThe authors do flag the linear-separability assumption and layer selection as limitations, so this is not a hidden flaw. But it should change how the results are reported. No error bars or multiple seeds are reported, which makes it hard to judge how stable the ResNet50 mitigation gains are. For a paper in this area, that is a moderate weakness.\n\nWho is this for? Researchers working on shortcut detection and interpretability-based model debugging will find the empirical comparison of CAVs vs neurons across layers and datasets useful, and the released code and annotations are a practical contribution. It deserves a serious peer review, but it needs a revised abstract and a more careful discussion of what the proposed pipeline can and cannot do for ViTs. I would ask for those revisions rather than desk rejecting.","headline":"A useful empirical integration of CAV-based bias annotation into Reveal2Revise, but the headline claim about ViT mitigation is not supported by the paper's own appendix.","tokens_in":36062,"tokens_out":2396,"would_cite":true,"duration_ms":21143,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single learned direction in latent space lets a medical AI pipeline detect, localize, and unlearn spurious artifacts with little expert labeling.","keywords":["spurious correlations","shortcut learning","concept activation vectors","interpretability","bias mitigation","medical imaging","ECG time series","explainable AI"],"falsifier":"Take a controlled artifact with ground-truth masks, such as the inserted timestamp in HyperKvasir, train CAVs on every layer of a fixed model, and test both ranking and localization on held-out samples; the central claim is falsified if some artifact admits no layer whose CAV separates artifact samples from clean samples above chance while its localization mask overlaps the ground truth no better than randomly.","tokens_in":34969,"feed_emoji":"🩺","tokens_out":7733,"duration_ms":68253,"temperature":0.7,"pith_summary":"Deep neural networks used for medical decisions can latch onto spurious artifacts—band-aids, rulers, pacemakers, brightness shifts, or inserted noise—that correlate with a diagnosis without causing it. This paper extends the Reveal2Revise bias-correction loop, which detects such shortcuts, models them, and unlearns them, with semi-automated annotation tools. A concept activation vector (CAV), a direction in the model's latent space that separates artifact-bearing from clean activations, ranks unlabeled samples by bias score to surface more artifact examples and produces spatial masks that localize the artifact. The authors demonstrate the extended loop on melanoma, gastrointestinal, chest radiograph, and electrocardiogram (ECG) datasets using VGG16, ResNet50, and Vision Transformer models, reporting that detected biases are identified, localized, and mitigated with reduced artifact reliance. The practical payoff is that bias correction becomes far less dependent on expensive expert labeling.","feed_headline":"Semi-automated shortcut buster rescues medical AI from spurious cues","feed_subtitle":"A bias-direction ranking finds hidden artifacts in skin, GI, chest X-ray, and ECG models, then unlearns them.","key_machinery":"The load-bearing object is the Concept Activation Vector (CAV), defined as a direction in a chosen layer's activation space that separates samples containing the artifact from samples without it. It works as a ranker, as an iteratively refinable bias model, and as a localizer: projecting activations onto the CAV yields a bias score, while using the element-wise product of activations and the CAV as the initialization for layer-wise relevance propagation turns the direction into input-space heatmaps and binary masks. The choice of layer is part of the machinery, and the paper treats it as a hyperparameter that strongly affects retrieval and localization quality.","core_discovery":"The paper's central claim is that the two least-automated steps of the Reveal2Revise life cycle—finding which samples carry a spurious artifact and marking where the artifact sits—can be driven by a single linear bias representation. Starting from a small seed set of artifact samples, a CAV trained at a chosen layer gives a bias score for every other sample; ranking by that score retrieves previously unlabeled artifact images, and iterating with brief human inspection refines the direction. The same CAV, when combined with layer-wise relevance propagation, localizes the artifact in input space as a soft or binary mask. The paper reports that this closes the loop across four medical datasets and two modalities: the identified biases are unlearned with gradient-penalty or activation-projection corrections, accuracy on biased test sets rises, and artifact relevance and concept-sensitivity (TCAV) scores fall while clean-test accuracy is largely preserved.","pith_inferences":["A natural extension the authors leave implicit is automatic layer selection—for example, choosing the layer whose CAV maximizes retrieval or localization quality on a validation set—which would remove the extra hyperparameter they flag in their limitations.","The bias-score ranking could be deployed as a standalone dataset-auditing tool before any model correction, quantifying artifact prevalence and exposing labeling errors in medical training data.","If the linear-direction assumption fails for heavily entangled concepts, the same loop would need non-linear concept regions or sparse-autoencoder disentanglement; the paper itself points to sparse autoencoders as future work.","Outside medicine, the same CAV annotation loop should apply to any domain where device-specific artifacts correlate with labels, such as satellite imagery or industrial inspection, though non-localizable artifacts like color shifts would remain harder to mask."],"forward_implications":["A small seed set of artifact samples can be expanded to a full dataset annotation by ranking unlabeled samples with CAV bias scores, reducing the expert labeling burden in the reveal and modeling stages.","Spatial masks produced by CAV localization can feed input-gradient-based correction methods such as RRR, removing the need for hand-drawn pixel-level artifact masks.","In the reported experiments, CAV-based mitigation methods such as RR-ClArC improve accuracy on biased test sets while keeping clean-test accuracy close to the baseline model's.","The framework transfers beyond images to time-series medical data, where ECG models trained with an inserted static-noise artifact show the same detect-and-mitigate pattern."],"supporting_citations":[{"why":"Introduced the original Reveal2Revise life cycle that this paper extends with sample- and feature-level annotation steps.","marker":"[57]"},{"why":"Introduced CAVs and the TCAV sensitivity score used to represent and measure bias reliance.","marker":"[47]"},{"why":"Introduced the ClArC bias-mitigation family and latent-space relevance clustering used throughout the loop.","marker":"[3]"},{"why":"Introduced SpRAy, the spectral-relevance outlier detection used as the data-perspective reveal step.","marker":"[48]"},{"why":"Provided layer-wise relevance propagation, the attribution method behind heatmaps, localization, and relevance-based bias scores.","marker":"[4]"},{"why":"Distinguished SVM-CAVs from Pattern-CAVs, motivating which bias direction is used for retrieval versus mitigation.","marker":"[58]"},{"why":"Provided the RRR input-gradient penalty used to unlearn localizable artifacts.","marker":"[63]"},{"why":"Provided PCX prototypes used to reveal spurious inlier behavior when biases are too dominant to appear as outliers.","marker":"[28]"},{"why":"Provided DORA, the model-perspective outlier-neuron detection used in bias identification.","marker":"[17]"}],"fun_headline_variants":["Semi-auto bias ranker finds and unlearns medical AI shortcuts","Interpretability-driven bias busting for safer medical AI","Rank spurious samples, erase shortcuts in medical deep nets","Semi-auto interpretability finds and unlearns medical AI bias"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline assumes a spurious artifact can be captured as one linear direction in some layer of the network's latent space, and that a suitable layer can be found.","fun_headline_variants_meta":{"raw":{"variants":["Semi-auto bias ranker finds and unlearns medical AI shortcuts","Interpretability-driven bias busting for safer medical AI","Rank spurious samples, erase shortcuts in medical deep nets","Semi-auto interpretability finds and unlearns medical AI bias"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000452,"raw_usage":{"total_tokens":2277,"prompt_tokens":948,"completion_tokens":1329,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":564,"completion_tokens_details":{"reasoning_tokens":1258}},"tokens_in":564,"tokens_out":1329,"duration_ms":8792,"temperature":1.0,"reasoning_tokens":1258,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T15:35:09.735217+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a controlled artifact with ground-truth masks, such as the inserted timestamp in HyperKvasir, train CAVs on every layer of a fixed model, and test both ranking and localization on held-out samples; the central claim is falsified if some artifact admits no layer whose CAV separates artifact samples from clean samples above chance while its localization mask overlaps the ground truth no better than randomly.","supporting_citations":[{"cited_title":"Reveal to re- vise: An explainable ai life cycle for iterative bias correction of deep models","cited_arxiv_id":null,"evidence_quote":"Introduced the original Reveal2Revise life cycle that this paper extends with sample- and feature-level annotation steps."},{"cited_title":"Interpretability beyond feature attribu- tion: Quantitative testing with concept activa- tion vectors (tcav)","cited_arxiv_id":null,"evidence_quote":"Introduced CAVs and the TCAV sensitivity score used to represent and measure bias reliance."},{"cited_title":"Unmasking clever hans predictors and assessing what ma- chines really learn","cited_arxiv_id":null,"evidence_quote":"Introduced SpRAy, the spectral-relevance outlier detection used as the data-perspective reveal step."},{"cited_title":"Navigating neural space: Revisiting concept activation vectors to over- come directional divergence","cited_arxiv_id":null,"evidence_quote":"Distinguished SVM-CAVs from Pattern-CAVs, motivating which bias direction is used for retrieval versus mitigation."},{"cited_title":"Right for the right reasons: training differentiable models by constraining their explanations","cited_arxiv_id":null,"evidence_quote":"Provided the RRR input-gradient penalty used to unlearn localizable artifacts."},{"cited_title":"Understand- ing the (extra-) ordinary: Validating deep model decisions with prototypical concept-based expla- nations","cited_arxiv_id":null,"evidence_quote":"Provided PCX prototypes used to reveal spurious inlier behavior when biases are too dominant to appear as outliers."},{"cited_title":"Dora: Exploring outlier representations in deep neural networks","cited_arxiv_id":null,"evidence_quote":"Provided DORA, the model-perspective outlier-neuron detection used in bias identification."}],"review_version":1}