REVIEW 4 major objections 6 minor 1 cited by
Prompting the Unseen: Detecting Hidden Backdoors in Black-Box Models
T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read BPROM catches hidden backdoors in black-box models by measuring how badly visual prompting adapts them to a clean task.
desk verdict BPROM is a useful black-box detector with a shaky mechanism: the claimed low-accuracy-from-subspace-inconsistency signal lacks a clean baseline and is contradicted by the paper's own Table 9, but the broad empirical results still deserve referee time. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the visual prompt itself: a trainable pixel perturbation that adapts a frozen source classifier to a target task. The paper's named mechanism is class subspace inconsistency — the misalignment between the poisoned source model's class subspaces and those of a clean external dataset — which manifests as low accuracy of the prompted model and as a distinctive pattern in the concatenated confidence vectors. That pattern is what the meta-classifier (a random forest) learns to associate with backdoors, and the method's efficiency comes from training shadow models with a single backdoor type rather than a large zoo of attacks.
What would settle it
Train an all-to-one backdoored model whose feature geometry is regularised to keep the target class subspace separated from other classes (for example, by a contrastive loss on clean features during poisoning); if BPROM's AUROC collapses toward 0.5 while the attack's success rate stays above 98%, the class-subspace-inconsistency premise is falsified. A simpler check the authors already concede is an all-to-all backdoor, which they report the method fails to detect.
Extended reading notes
Core claim
The central discovery is the class subspace inconsistency: visual prompting, which the paper reads as mapping class subspaces from a source domain to a target domain, fails when the source model is backdoored. In an all-to-one backdoored model, the target class subspace is pushed to share boundaries with every other class, so a prompt learned to align the poisoned source with a clean external dataset produces a prompted model with degraded accuracy. The paper operationalises this as a detector: train clean and backdoored shadow models on a small reserved dataset, apply visual prompting to each shadow model and to the suspicious model with an unrelated clean dataset, concatenate the resulting confidence vectors on $q$ samples, and train a random-forest meta-classifier to separate the two groups. The claim is that this signal is stable across backdoor types, architectures, and dataset scales, and that it needs far fewer shadow models than prior meta-learning detectors such as MNTD. On ResNet18 and MobileNetV2, across CIFAR-10, GTSRB, Tiny-ImageNet, and ImageNet, the reported average AUROC is at least 0.979, with many settings at 1.0.
Load-bearing premise
The detector assumes that every all-to-one backdoor distorts the model's feature space in the same way — pushing the target class adjacent to all other classes — so that visual prompting always produces a comparably large accuracy drop that the meta-classifier can learn to recognise.
Editorial extensions
If this is right
- Black-box model-level backdoor detection becomes feasible with only 1% to 10% of the test set as a clean reference and roughly 20 shadow models trained with a single backdoor type.
- Detection transfers across architectures: a meta-classifier trained on ResNet18 shadows detects backdoors in MobileNetV2, MobileViT, and Swim Transformer models without retraining.
- The prompt-accuracy drop scales with trigger size and poison rate, so stronger attacks become easier, not harder, to detect.
- The method remains effective against adaptive attacks including extremely low poison rates (0.2%) and clean-label attacks such as SIG and LC, with AUROC at or near 1.0.
Reading between the lines
- The same probe could be used to detect other kinds of feature-space corruption, such as class-imbalance distortion or poorly calibrated fine-tuning, wherever the corruption makes class subspaces in the source model hard to align with a clean target.
- The dependence on a single shadow attack type suggests the meta-classifier is learning the shape of the inconsistency rather than the trigger, which would explain the cross-attack transfer; a direct test would be to train shadows on one attack and hold out a completely unseen attack family.
- Because the signal is an accuracy gap, a natural next step is to search over visual prompt families or initialization to amplify the gap, potentially turning a binary detector into a per-class diagnostic that identifies the target class.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes BPROM, a black-box model-level backdoor detector that applies visual prompting to a suspicious frozen model using an unrelated clean dataset, then trains a random-forest meta-classifier on the confidence vectors of prompted shadow models. The authors argue that backdoored models exhibit 'class subspace inconsistency' between the poisoned source classes and the clean target classes, which causes the prompted model's accuracy to drop, and that this drop is the discriminative signal. Experiments cover 9 backdoor attacks on CIFAR-10, GTSRB, Tiny-ImageNet, and ImageNet with ResNet18, MobileNetV2, MobileViT, and Swin Transformer, reporting high AUROC/F1 on standard settings and claiming strong performance against adaptive and clean-label attacks, while admitting failure on all-to-all backdoors.
Significance. If the central mechanism were established, BPROM would be a practically useful black-box detector: it requires only a small reserved clean dataset, a handful of shadow models (20 vs. MNTD's 256), and an unrelated external dataset for prompting. The experimental matrix is broad, with many baseline comparisons and consistent high AUROC on the default attack configurations. The authors are also transparent about the all-to-all limitation. However, the paper's core explanatory claim—that low prompted-model accuracy caused by class subspace inconsistency is what drives detection—is asserted rather than derived, and the paper's own Table 9 appears to contradict it. The reported clean-label and low-poison-rate results further suggest the meta-classifier may be using a different signal than the stated mechanism, so the significance currently rests on an unverified interpretation.
major comments (4)
- [Section 6.3, Tables 9 and 4] The reported results contradict the paper's core mechanism. For CIFAR-10 Blend at 5% poison rate, Table 9 gives ASR 0.996 and AUROC 0.607, while Table 4 reports prompted accuracy 0.5297 for the same setting; at 10% and 20% poison rates, prompted accuracy is lower (0.4772 and 0.3985) and AUROC is higher (0.933 and 1.000). If low prompted accuracy due to class subspace inconsistency were the discriminative signal, the 5% condition—which already shows substantially reduced accuracy—should be at least as detectable as the 0.3% poison-rate setting in Table 5, where AUROC is 1.000. The AUROC drop at 5% suggests the meta-classifier is not learning the claimed accuracy signal and may be overfitting to the specific poison configuration of the shadow models (BadNets at 0.3% per Table 13). This undermines the central claim that BPROM leverages prompted-model accuracy; please either reconcile this result or revise the mechanistic claim.
- [Section 5.1, Tables 3 and 4] The mechanism claim that 'low classification accuracy of the prompted model indicates backdoors' is never validated against clean models. Tables 3 and 4 report prompted accuracy only for backdoored models (e.g., 0.31–0.53 on CIFAR-10), and Section C's PCA shows separation but no accuracy comparison. Without the distribution of prompted accuracy for clean models under the same visual-prompting procedure and the same DS/DT pairs, the paper does not demonstrate that the proposed signal is discriminative. Please add clean-model prompted accuracy for the same settings and show the margin used by the meta-classifier; this is necessary to support the paper's abstract and Section 5.1 statements.
- [Section 5.1 and Section 7] The paper asserts that class subspace inconsistency is a universal consequence of all-to-one poisoning, citing Wang et al. (2019). However, the evidence in Tables 3 and 4 covers only Blend and Adap-Blend, Section C covers Trojan and Adap-Blend, and the conclusion explicitly concedes that all-to-all backdoors fail; Table 12 also shows degraded performance on clean-label attacks (SIG/LC). The claim 'BPROM effectively detects all-to-one backdoors' is therefore broader than the evidence. Please either restrict the claim to the attack families tested or provide a characterization of the conditions (e.g., target-class geometry perturbation) under which the accuracy drop actually occurs.
- [Section 6.4, Table 11] The adaptive-attack experiment reports AUROC = 1.0 at 0.2% poison rate where the backdoor is weak (ASR 0.709), while Table 9 shows AUROC = 0.607 at 5% Blend with ASR 0.996 and prompted accuracy 0.53. This pattern indicates that the detector's decision is not monotonically related to the claimed 'class subspace inconsistency' accuracy drop; the meta-classifier appears to respond to some other property of the confidence vectors. Please report the prompted accuracy and the clean-model baseline for the Table 11 settings to clarify what signal the meta-classifier is actually using.
minor comments (6)
- [Tables 24 and 25] The architecture name 'Swim Transformer' should be 'Swin Transformer' in both table captions and the text in Section B.3.
- [Author affiliation] The affiliation contains a typo: 'Univeraity' should be 'University'.
- [Table 1 and Section 6.1] The baseline is referred to both as 'ScaleUp' (Table 1) and 'SCALE-UP' (elsewhere); please use one consistent name.
- [Algorithm 1, line 12] The prompt subscript is dropped: it should be 'construct f̃_i = f_i ◦ V(·|θ_i)' to match the notation in Section 5.2.
- [Section 5.1] The reference to 'Theorem 1 in Yang et al. (2021)' is not stated; please either state the theorem or explain how the alignment-loss term is 'amplified by the inconsistency in infected models', since this is currently an unsupported assertion.
- [Section 6.3 and Table 13] Please clarify which attack configurations are used for shadow-model training versus suspicious-model evaluation in the generalization experiments; the current text says the meta-classifier is trained on BadNets shadow models, but Table 9 varies poison rates for Blend, making it hard to assess distribution shift.
Circularity Check
No significant circularity: the detection pipeline is an empirically calibrated shadow-model meta-classifier, and the central mechanism is supported by external citations and the paper's own held-out experiments.
full rationale
BPROM's derivation chain is not circular. The central premise—that poisoned models distort class-subspace geometry so that visual prompting to a clean target dataset yields degraded confidence vectors—is imported from external work (Wang et al. 2019; Yang et al. 2021) and is additionally tested by the paper's own PCA visualizations (Appendix C) and trigger-size/poison-rate accuracy tables (Tables 3 and 4). No equation in the paper defines class subspace inconsistency in terms of the detector's output, and no fitted constant is relabeled as a prediction. The meta-classifier is trained on a held-out set of shadow models (Algorithm 1) and then evaluated on separately trained suspicious models, including attacks not used in meta-training (e.g., a BadNets-trained meta-classifier evaluated on Blend, Trojan, WaNet, Dynamic, Adap-Blend, and Adap-Patch) and on different architectures and datasets; this is standard external calibration, not a construction-equivalent reduction. The one self-citation (Li et al. 2023b, Section 3) is a background enumeration of VP extensions and is not load-bearing. The manuscript's own limitations are explicit: the conclusion states that BPROM struggles with all-to-all backdoors, and Section 6.4 notes that clean-label attacks may preserve class subspaces and hinder detection. The strongest concerns are evidentiary rather than circular: Table 9's Blend 5% row (ASR 0.996, prompted accuracy 0.53, AUROC 0.607) and the absence of a clean-model prompted-accuracy baseline weaken the mechanistic story, but these are correctness and robustness issues, not circularity.
Assumptions & free parameters
free parameters (6)
- Shadow model count M, split n/(M-n) =
20 (10 clean, 10 backdoor)
- Poisoning rate and trigger parameters for shadow backdoor datasets =
BadNets 0.3% (CIFAR-10), 1% (GTSRB); Table 13 defaults
- Reserved clean dataset size alpha =
10% for main results; 1% and 5% ablations
- External visual prompting dataset DT =
STL-10 main; SVHN and CIFAR-100 ablations
- Number of query samples q =
not specified
- CMA-ES prompt optimization hyperparameters =
not reported
assumptions (5)
- domain assumption Backdoor-poisoned models have target-class subspaces adjacent to all other class subspaces in feature space.
- domain assumption Visual prompting can adapt a frozen clean model to an external target domain with acceptable accuracy.
- ad hoc to paper A random forest on concatenated prompted confidence vectors separates clean from backdoored models.
- domain assumption All-to-one poisoning increases feature-space distortion in a way that depends monotonically on trigger size and poison rate.
- domain assumption Defender has a small reserved clean dataset DS from the same distribution as the suspicious model's training data.
Cite this review
Pith. "Pith review of Prompting the Unseen: Detecting Hidden Backdoors in Black-Box Models." pith.science (2026). https://pith.science/paper/XLHOHO2Y
@misc{pith2026241109540,
author = {Pith},
title = {Pith review of: Prompting the Unseen: Detecting Hidden Backdoors in Black-Box Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/XLHOHO2Y}},
note = {Machine review of arXiv:2411.09540}
}
read the original abstract
Visual prompting (VP) is a new technique that adapts well-trained frozen models for source domain tasks to target domain tasks. This study examines VP's benefits for black-box model-level backdoor detection. The visual prompt in VP maps class subspaces between source and target domains. We identify a misalignment, termed class subspace inconsistency, between clean and poisoned datasets. Based on this, we introduce \textsc{BProm}, a black-box model-level detection method to identify backdoors in suspicious models, if any. \textsc{BProm} leverages the low classification accuracy of prompted models when backdoors are present. Extensive experiments confirm \textsc{BProm}'s effectiveness.
Figures
Figures from the paper (2 more)
Forward citations
Cited by 1 Pith paper
-
On Understanding, Identifying, and Mitigating Vulnerabilities in Agentic Large Language Models
A PRISMA-based survey of 85 papers shows agentic LLM security research is attack-heavy and perception-focused, leaving action-layer and code-execution risks understudied.
Reference graph
Works this paper leans on
-
[1]
Visual prompting: Modify- ing pixel space to adapt pre-trained models
Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Visual prompting: Modify- ing pixel space to adapt pre-trained models. arXiv:2203.17274,
-
[3]
Sentinet: Detecting localized universal attacks against deep learning systems
Edward Chou, Florian Tram`er, and Giancarlo Pellegrino. Sentinet: Detecting localized universal attacks against deep learning systems. 2020 IEEE Security and Privacy Workshops (SPW), pp. 48–54,
work page 2020
-
[9]
Ahmed Salem, Rui Wen, Michael Backes, Shiqing Ma, and Yang Zhang
doi: 10.1007/s11263-015-0816-y. Ahmed Salem, Rui Wen, Michael Backes, Shiqing Ma, and Yang Zhang. Dynamic backdoor attacks against machine learning models. In 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P),
-
[10]
URL https://arxiv.org/abs/1912.02771. 14 This paper has been accepted by IEEE/IFIP DSN 2025 Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y . Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. IEEE Symposium on Security and Privacy (SP),
arXiv 1912
-
[11]
Zhenting Wang, Juan Zhai, and Shiqing Ma. Bppattack: Stealthy and efficient trojan attacks against deep neural networks via image quantization and contrastive adversarial learning. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),
work page 2022
-
[12]
Section A details the implementation and configurations of the experiments
15 This paper has been accepted by IEEE/IFIP DSN 2025 APPENDIX OF BPROM: B LACK -BOX MODEL -LEVEL BACKDOOR DETECTION VIA VISUAL PROMPTING This appendix provides additional details and experimental results supporting our main findings. Section A details the implementation and configurations of the experiments. Section B presents BP ROM’s evaluation on diff...
work page 2025
-
[13]
• Poison rate: The proportion of training data with the trigger pattern
For each attack, we specify parameters related to the backdoor trigger insertion, including poison rate and cover rate. • Poison rate: The proportion of training data with the trigger pattern. A higher poison rate increases the attacker’s influence on the model’s behavior but also raises the detection risk. • Cover rate: The proportion of data with the tr...
work page 2017
-
[17]
(2018) 0.952 0.047 0.952 0.115 0.240 0.952 0.551 SS (Tran et al.,
0.986 0.896 0.895 0.883 0.865 0.985 0.926 SentiNet Chou et al. (2018) 0.952 0.047 0.952 0.115 0.240 0.952 0.551 SS (Tran et al.,
work page 2018
Show all 22 references
-
[18]
0.992 0.980 0.982 0.612 0.877 0.380 0.804 BP ROM(10%) 1.000 1.000 1.000 1.000 1.000 1.000 1.000 input images. Table 22 presents the results of BPROM on three feature-based backdoor methods: Refool (Yunfei Liu, 2020), BPP (Wang et al., 2022), and Poison Ink (Zhang et al., 2022)...
2020
-
[19]
We analyze the impact of the reserved clean dataset size (DS) on BPROM’s performance
CIFAR-10 1.000 1.000 Impact of Reserved Clean Dataset Size. We analyze the impact of the reserved clean dataset size (DS) on BPROM’s performance. As shown in Table 23, BPROM maintains high AUROC across different DS sizes (1%, 5%, and 10% of the CIFAR-10 and GTSRB test sets). E...
2018
-
[20]
4https://github.com/vtu81/backdoor-toolbox 16 This paper has been accepted by IEEE/IFIP DSN 2025 • SCAn (Tang et al., 2021): Threshold for abnormal score = 0.5
• SS (Tran et al., 2018): Number of removed samples = min(1.5 × |Dpoison|/|D|, 0.5 × class size). 4https://github.com/vtu81/backdoor-toolbox 16 This paper has been accepted by IEEE/IFIP DSN 2025 • SCAn (Tang et al., 2021): Threshold for abnormal score = 0.5. • SPECTRE (Hayase ...
2018
-
[21]
(2018) cifar10 0.5002 0.3902 0.3745 0.5145 0.6154 0.3801 0.3977 0.4532 gtsrb 0.4925 0.4987 0.4925 0.4925 0.4966 0.4961 0.4929 0.4945 SCAn (Tang et al.,
cifar10 1.000 0.9962 1.000 0.969 0.9999 0.9961 0.9643 0.9893 gtsrb 0.9994 0.9732 0.9993 0.7782 0.9601 0.8731 0.8993 0.9261 CT (Qi et al., 2023c) cifar10 0.9941 0.9379 0.9843 0.7475 0.9892 0.9439 0.8815 0.9255 gtsrb 0.9727 0.9718 0.9744 0.9677 0.5652 0.9684 0.9289 0.9070 SS Tra...
2018
-
[22]
The same meta-model is also used to classify clean (green dots) and Adap-Blend-infected models (red dots) Qi et al
cifar10 0.4752 0.4752 0.5961 0.4754 0.4754 0.5015 0.4752 0.4963 gtsrb 0.7383 0.4995 0.7383 0.4995 0.4995 0.4995 0.4991 0.5677 BPROM (10%) cifar10 1.0000 1.0000 0.9667 1.0000 1.0000 1.0000 1.0000 0.9952 gtsrb 1.0000 1.0000 1.0000 1.0000 0.9655 0.9655 1.0000 0.9901 dots) and bac...
2023
-
[23]
(2018) cifar10 0.3877 0.3745 0.3753 0.2747 0.3749 0.3749 0.3913 0.3648 gtsrb 0.4961 0.4925 0.4946 0.4987 0.4925 0.4925 0.4925 0.4942 SCAn (Tang et al.,
cifar10 1.0000 0.9563 0.9594 0.5707 0.9999 0.8564 0.8125 0.8793 gtsrb 0.9283 0.8835 0.9194 0.6767 0.8301 0.8531 0.8794 0.8529 CT (Qi et al., 2023c) cifar10 0.868 0.9968 0.9994 0.5142 0.9919 0.8766 0.9758 0.8890 gtsrb 0.9125 0.867 0.9967 0.4279 0.9991 0.8638 0.7455 0.8304 SS Tr...
2018
-
[2011]
datasets, attacked with 9 different backdoor attacks, and then defended with state-of-the-art methods. B.1 A CCURACY AND ATTACK SUCCESS RATE We report the clean accuracy (ACC) of the infected models on benign test samples without triggers and the attack success rate (ASR), whi...
2017
-
[2015]
Neural attention distillation: Erasing backdoor triggers from deep neural networks
Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Neural attention distillation: Erasing backdoor triggers from deep neural networks. In International Conference on Learning Representations (ICLR), 2021a. Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu,...
2025
-
[2019]
Ranasinghe, and Hyoungshick Kim
11 This paper has been accepted by IEEE/IFIP DSN 2025 Yansong Gao, Yeonjae Kim, Bao Gia Doan, Zhi Zhang, Gongxuan Zhang, Surya Nepal, Damith C. Ranasinghe, and Hyoungshick Kim. Design and evaluation of a multi-domain trojan detection method on deep neural networks. IEEE Transa...
2025
-
[2020]
To investigate the impact of inconsistency between the numbers of classes in DS and DT , we conducted experiments using CIFAR-100 as DS and STL-10 as DT
Adap- Blend (Qi et al., 2023b) Adap- Patch (Qi et al., 2023b) A VG F1 1.000 1.000 1.000 0.967 1.000 1.000 1.000 0.995 AUROC 1.000 1.000 1.000 0.967 1.000 1.000 1.000 0.995 Experiments on CIFAR-100. To investigate the impact of inconsistency between the numbers of classes in DS...
2025
-
[2021]
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. arXiv preprint arXiv:1512.03385,
-
[2022]
doi: 10.1109/TNNLS.2022. 3182979. Yizhe Li, Yu-Lin Tsai, Chia-Mu Yu, Pin-Yu Chen, and Xuebin Ren. Exploring the benefits of visual prompting in differential privacy. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023b. Yuezun Li, Yiming Li,...
2022 doi
-
[2023]
Revisiting the assump- tion of latent separability for backdoor defenses
Xiangyu Qi, Tinghao Xie, Yiming Li, Saeed Mahloujifar, and Prateek Mittal. Revisiting the assump- tion of latent separability for backdoor defenses. InInternational Conference on Learning Represen- tations, 2023a. URL https://api.semanticscholar.org/CorpusID:259298183. Xiangyu...
2025
-
[2024]
Targeted backdoor attacks on deep learning systems using data poisoning
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Xiaodong Song. Targeted backdoor attacks on deep learning systems using data poisoning. ArXiv, abs/1712.05526,
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.