{"id":"5da660d5-62a7-4467-bfea-73679a7f32c5","arxiv_id":"2411.09776","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A stage-and-risk-based decision rule predicts whether pairs of ML defenses conflict, with reported balanced accuracy of 90% on eight prior combinations and 81-86% on 30 new ones.","lead":"This paper proposes DefCon, a flowchart-style rule that predicts whether two machine learning defenses can be combined without one destroying the other. It also contributes a systematic map of 55 defense pairs and reports 81-90% balanced accuracy on 38 combinations.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"DefCon's Table 4 labels are not implementation-stable: C7 (DtWM.Pre + DiffPriv.In) is alignment for radioactive watermarks but conflict for BadNets backdoors, so the 90%/81% accuracy is tied to the specific implementations tested.","rationale":"The reader's weakest assumption is the hand-assigned Table 4 labels; I refine this to a concrete instability. The paper's own C7 is the clearest evidence: the ground-truth combination (DtWM.Pre + DiffPriv.In) is aligned for the radioactive-watermark variant used by Szyller & Asokan, but DefCon's labels (DtWM.Pre uses backdoors; DiffPriv.In protects against backdoors) make it a predicted conflict. The authors count this as a false negative and move on, but the implication is deeper: the labels are not properties of the defense type; they are properties of a specific implementation. Because all 30 unexplored combinations use BadNets-style backdoors for DtWM.Pre/MdlWM.Pre, the 81%/86% accuracy is an evaluation of the rule on those implementations, not on the general categories DefCon claims to handle. The §7.5 assertion that other implementations 'should not affect our conclusions' is contradicted by the C7 discussion, which says the opposite for a different DtWM variant. A concrete two-variant rerun of C7 would settle the matter. If the variants behave differently, the paper should either restrict the claim to the tested implementations or require variant-level labels (trigger type, perturbation type, etc.) as inputs to DefCon. Either way, the central 'without running defenses' claim is currently overstated. I do not think this warrants outright rejection: the conflict rule works for the tested backdoor implementations, and the paper is transparent about C7. But the conditional verdict is right, and the requested test should be added. I also considered the §7.3 threshold issue but did not make it primary, because the empirical conflicts in Table 5 are large and would likely survive a significance-based threshold; the label-instability issue is more decisive.","tokens_in":40438,"tokens_out":13953,"duration_ms":141249,"concrete_test":"Run C7 (DtWM.Pre followed by DiffPriv.In) with the same datasets, model, and DiffPriv hyperparameters as §6.4, using (a) BadNets 5x5 backdoor watermarks as in this paper's DtWM.Pre, and (b) the radioactive/adversarial-example watermark implementation from Szyller & Asokan (2023) used as the C7 ground truth. Measure ϕRSD and ϕdp for both. If (a) yields RSD well below the 100% single-defense baseline (conflict) while (b) preserves RSD near 100% (alignment), then DefCon's single Table 4 entry for DtWM.Pre cannot be correct for both variants, and the 90%/81% accuracy is implementation-specific. If both variants behave identically, the label is stable and this concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central accuracy claims assume each defense has a single, implementation-independent tuple (stage, change type, risk-used, risk-protected) in Table 4. The paper's own C7 contradicts this. In §7.2, C7 (DtWM.Pre + DiffPriv.In) is taken from Szyller & Asokan as an effective combination (alignment), because their DtWM variant uses radioactive/adversarial-example watermarks that are inliers and not suppressed by DPSGD. But DefCon's Table 4 labels DtWM.Pre as using backdoors (S3=Yes) and DiffPriv.In as protecting against that risk (S4=Yes), so DefCon predicts conflict; this is exactly the one false negative that reduces the 90% accuracy to 7/8. The same table is then used to predict the 30 unexplored combinations, where DtWM.Pre is implemented as BadNets 5x5 backdoors (§6.4), a variant that is indeed suppressed by poison/DP defenses. Thus the reported 81%/86% validates DefCon only for the specific implementations in §6.4, not for 'defenses' as general categories. The paper's claim in §7.5 that 'other implementations can be used and should not affect our conclusions' is not supported by its own C7 discussion. A practitioner who knows only that their defense is 'dataset watermarking' cannot determine from Table 4 whether a later defense (e.g., DiffPriv.In or PoisnRob.In) will suppress the watermark: that depends on whether the watermark is a backdoor trigger, a radioactive perturbation, or another scheme. Without this variant-level information, the non-invasive screening rule is under-specified, so the headline claim that practitioners can identify effective combinations without running them together is not established for the general defense categories claimed.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper systematizes prior work on combining ML defenses and proposes DefCon, a rule-based technique that predicts whether two defenses can be combined without modifying them. DefCon's flowchart (S1-S4) records each defense's pipeline stage, whether it makes global/local/no changes, whether it uses a risk, and whether it protects against a risk. The authors report a balanced accuracy of 90% on eight combinations taken from prior work and 81% on 30 previously unexplored combinations, rising to 86% after hyperparameter tuning for one false-positive combination. They also provide illustrative three-defense combinations and argue that DefCon satisfies accuracy, scalability, non-invasiveness, and generality.","tokens_in":40763,"tokens_out":3998,"duration_ms":41886,"significance":"If the accuracy and generality claims were fully supported, DefCon would be a useful practical screening tool: a practitioner could decide whether two off-the-shelf defenses can be combined without empirical evaluation. The paper's main strengths are its broad systematization of the defense-combination literature, its effort to evaluate 30 previously unexplored combinations, the public availability of code, and the explicit treatment of multi-way combinations. However, the headline accuracy figures are weaker than presented: the 90% figure is measured on the same prior-work combinations used to iteratively design DefCon, and the 86% figure is obtained only after post hoc hyperparameter tuning of a single combination. More importantly, the paper's own C7 discussion shows that the DefCon labels in Table 4 are implementation-dependent, contradicting the generality claim in Section 7.5. The empirical evaluation is still a real contribution, but the central claims need to be substantially qualified.","major_comments":[{"comment":"The Table 4 tuple for DtWM.Pre (S3=Yes) and DiffPriv.In (S4=Yes) predicts a conflict, but C7 is classified as an effective combination in prior work because the radioactive/adversarial-example watermarks used by Szyller and Asokan are inliers not suppressed by DPSGD. This is the single false negative that lowers the eight-combination accuracy to 7/8. In the 30-combination evaluation, DtWM.Pre is implemented with BadNets 5x5 backdoor watermarks (§6.4), which are indeed suppressed by poison/DP defenses. Consequently, the 81%/86% accuracy validates DefCon only for the specific implementations tested, not for 'dataset watermarking' as a general defense category. The statement in §7.5 that 'other implementations can be used and should not effect our conclusions' is therefore not supported by the paper's own evidence. The authors need either to make the variant level part of the defense representation or to explicitly restrict the claims to the tested implementations.","section":"§7.2, C7; §6.4; §7.5"},{"comment":"The 90% accuracy is not an independent validation. Section 5.3 states that the authors 'iterate over the design of DefCon using prior work from §4,' and Section 7.2 then evaluates DefCon on eight combinations drawn from that same prior work. This makes the eight-combination result in-sample. The authors should present the 90% figure as a training-set accuracy or, preferably, derive the rules without using the eight C1-C8 combinations and then test on those combinations as a held-out set. The 30-combination evaluation in §7.3 is less circular, but the headline comparison against the naïve technique should not be based on the in-sample 90% figure.","section":"§5.3; §7.2"},{"comment":"The empirical ground-truth rule is a single arbitrary threshold: a combination is called a conflict if 'at least one dataset has at least one metric marked as orange or red,' where orange means worse than the single-defense baseline but better than no defense. This rule can classify a combination as conflicting because of one degraded metric among several, and it takes the worst outcome across two datasets with different random seeds. The accuracy numbers in §7.3 are directly contingent on this labeling rule, but no sensitivity analysis is provided. The authors should justify the threshold, report results under alternative rules (e.g., requiring at least two metrics to fail, or treating orange as non-conflict), and state how sensitive the 81%/86% figures are to this choice.","section":"§7.3"},{"comment":"The 86% figure in the Abstract is post hoc. Section 7.3 reports 81% on the 30 unexplored combinations; Section 7.4 then tunes hyperparameters for the three false positives (C17, C32, C35), finds one configuration for C32 that removes the conflict, and re-computes accuracy as 86%. This is an exploratory search for a successful configuration, not an out-of-the-box prediction accuracy. If the 86% figure is kept, the paper must state explicitly that it is achieved after hyperparameter tuning of one combination, and the Abstract should not present it as the accuracy of DefCon as a screening rule. The same issue applies to the contribution list in Section 1.","section":"§7.4; Abstract"}],"minor_comments":[{"comment":"In several rows the value reported for 'wmacc' is identical to the value reported for 'ASR' (e.g., C36 wmacc 10.18±0.46, C37 wmacc 71.31±14.10, C38 wmacc 58.65±19.23). This appears to be a copy-and-paste error in the table and should be corrected, because the empirical conflict labels for these combinations should depend on the actual watermark accuracy.","section":"Table 5, C36-C38"},{"comment":"The appendix is titled 'Formal Analysis of DefCon,' but only the consistency claim is a proof; soundness is argued by citing empirical accuracy, and completeness is conditional on an assumption stated in the text. The authors should either provide a real formal statement with assumptions made explicit or rename the appendix to avoid overclaiming.","section":"Appendix B"},{"comment":"The Abstract says '86% in 30 unexplored combinations,' whereas §7.3 reports 81% and §7.4 obtains 86% only after tuning a single combination. Please align the abstract and the contribution list with the methodology actually used.","section":"Abstract vs. §7.3/§7.4"},{"comment":"Table 1 uses the same symbol for alignment, conflict, and unexplored combinations, relying on color/text that may not be distinguishable in black-and-white printing. Distinct glyphs for the three interaction types would improve readability.","section":"Table 1"},{"comment":"The three-defense combinations C39-C44 are reported on only one dataset for some rows (e.g., C42-C44 have no FMNIST column), and the text says these are illustrative. This is acceptable, but the table should clearly mark where a value is omitted by design and where it was not run, so that readers do not misinterpret the absence as a failed experiment.","section":"Table 7"}],"recommendation":"major_revision","confidential_remarks":"The paper contains a real evaluation effort and the 30-combination experiment is valuable, but the central accuracy claims are currently overstated. The C7 contradiction is not a peripheral issue: it shows that the Table 4 tuple is not implementation-stable, which directly undermines the generality claim in Section 7.5. I believe the paper can be repaired by restricting the claims to tested implementations and by clearly separating in-sample development accuracy from out-of-sample accuracy, but those fixes require substantive rewriting of the abstract, the contribution list, and Sections 7.2-7.5."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"DefCon is a useful screening heuristic, and the systematization around it is genuinely the most valuable part of the paper. But the accuracy headline outruns the evidence: the 90% is measured on the same eight combinations used to iteratively design the flowchart, the 86% on the 30 unexplored pairs is achieved only after post-hoc hyperparameter tuning of one false positive (C32), and both figures rest on hand-assigned labels (Table 4) that are not stable across implementations.\n\nWhat is actually new: a simple non-invasive decision procedure (S1–S4) that clearly beats mutually-exclusive placement in their evaluation (90% vs 40%; 81% vs 36% before tuning), a useful systematization of 55 pairwise combinations with interaction types, and a commendably transparent evaluation of 30 previously unexplored pairs plus five three-way combinations. The code is public. The paper also states most of its own limitations in §8, which is more than many papers do.\n\nThe soft spots, in proportion:\n- Circularity: §5.3 says DefCon was iterated using prior work from §4, and §7.2 then uses those same eight combinations as the 90% test. That number is fitting, not prediction. The only out-of-sample estimate is the 81% (pre-tuning) on the 30 combinations.\n- Post-hoc tuning: the 86% figure in the abstract is 81% plus a grid-search fix for C32. That is not itself fatal, but the abstract should say so, or report 81% as the primary number.\n- Implementation-dependence: the stress-test point is right. Table 4 assigns DtWM.Pre a single tuple (Local, Yes, No), but the paper's own C7 discussion shows the aligned result from Szyller & Asokan used radioactive watermarking (inliers, not suppressed by DP), while the 30-combination evaluation implements DtWM as BadNets backdoors. So the 81%/86% validates DefCon only for the tested implementations; §7.5's claim that other implementations would not affect the conclusions is contradicted by C7.\n- The ground-truth rule for the 30 combinations (any defense metric on any dataset below its single-defense baseline => conflict) is transparent but self-defined, and the accuracy is relative to that rule.\n\nNone of this kills the core idea. DefCon is a reasonable heuristic for defense variants you have correctly labeled, and a badly needed systematization. It just is not, as claimed, a general non-invasive predictor for defense categories. A practitioner who knows only that they use 'dataset watermarking' cannot tell from Table 4 whether DPSGD will suppress it.\n\nThis is a paper for ML-security researchers who want a structured map of defense interactions and a cheap screening baseline. It also makes a good case study in evaluation circularity for a reading group.\n\nRecommendation: serious peer review, certainly — it deserves referee time. For a next version, report the pre-tuning 81% as the main out-of-sample accuracy, clearly label the tuned 86%, and either mark Table 4 entries with the specific implementation tested or restrict the generalization claims to those implementations. With that, the contribution is solid.","headline":"Useful screening heuristic and systematization, but the headline accuracy is inflated—the 90% is on the design set and Table 4's labels are implementation-specific, so the generalization claim doesn't hold.","tokens_in":41331,"tokens_out":4824,"would_cite":true,"duration_ms":47447,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Def\\Con claims a four-step flowchart can tell whether two machine-learning defenses conflict, matching 90% of known combinations and 81% of 30 previously untested ones.","keywords":["machine learning defenses","defense combination","conflict prediction","adversarial training","watermarking","differential privacy","group fairness","model fingerprinting"],"falsifier":"Take a defense pair with a known empirical outcome, assign the Table 4 attributes, and check whether Def\\Con's prediction matches; repeating this with a defense not in the paper, such as certified robustness or a language-model watermarking scheme, would test whether the two conflict mechanisms generalize. A more direct check is to perturb the labels—for example, marking MdlWM.Pre as global instead of local—and observe whether the accuracy on the paper's own 30 combinations shifts, which would show the result is label-sensitive rather than mechanism-driven.","tokens_in":40207,"feed_emoji":"🛡️","tokens_out":4650,"duration_ms":45176,"temperature":0.7,"pith_summary":"The paper claims that whether two defenses can be combined without modifying either can be decided from three facts about each defense: its pipeline stage, whether its changes are global, local, or none, and whether it uses or protects against a particular risk. It packages this into Def\\Con, a flowchart that flags a combination as conflicting only when a later defense would override the first defense's changes or mitigate the risk the first defense relies on. On eight combinations previously studied, Def\\Con's predictions match the reported result 90% of the time, versus 40% for the common rule of placing defenses in different pipeline stages. On 30 previously unstudied combinations the authors evaluated, it matches empirical outcomes 81% of the time, rising to 86% after hyperparameter tuning. If this holds, practitioners can screen defense combinations cheaply before paying for the experiments.","feed_headline":"Def\\Con predicts clashing ML defenses at 90% accuracy","feed_subtitle":"It matches known outcomes on 90% of prior combinations and 81% of 30 new ones—without retraining defenses.","key_machinery":"The central object is the Def\\Con flowchart, a four-step decision procedure that maps each defense to a stage (pre-, in-, or post-training), a change type (global, local, or none), a risk it uses, and a risk it protects against. The flowchart answers four questions in sequence: same stage, whether the second defense makes only local or no changes, whether the first defense uses a risk, and whether the second defense protects that risk. This turns combination effectiveness into a binary prediction without running the defenses together, and it extends to three or more defenses by decomposing the combination into pairwise checks and pruning conflicting permutations.","core_discovery":"The central claim is that the two underlying conflict mechanisms, not stage separation, are what make defense combinations fail: a later-stage defense can mitigate a risk that an earlier-stage defense uses as its mechanism, or a later-stage defense can globally override changes made by the earlier defense. Def\\Con predicts alignment when a same-stage second defense makes only local or no changes, when different-stage defenses do not involve a protected risk, or when the second defense does not protect against the risk the first one uses. It predicts conflict when a later global change overrides an earlier defense or when the second defense protects against a risk used by the first. On this basis the paper reports a balanced accuracy of 90% on eight prior combinations, 81% on 30 unexplored combinations, and 86% after tuning false-positive cases, with the naïve stage-based technique at 40% and 36% respectively.","pith_inferences":["A testable extension is to replace the hand-assigned Table 4 labels with automatically derived property cards for each defense; if such labels can be produced reliably, Def\\Con becomes a cheaper advisory tool for new defenses.","The two conflict mechanisms likely transfer to other modalities, such as language models, where fine-tuning, pruning, and watermarking occupy similar pipeline stages, but that transfer needs its own empirical validation.","The paper's conflict rule treats any metric falling below its single-defense baseline as a conflict, so Def\\Con may over-reject combinations with small, acceptable degradations; a graded notion of conflict could improve practical usefulness.","The taxonomy of conflict causes is explicitly incomplete; extending it with mechanisms such as lp-norm mismatches or utility-effect entanglement could raise accuracy on the false-positive cases the paper observed."],"forward_implications":["Practitioners can pre-screen pairwise and multi-way defense combinations without training runs, reserving expensive empirical evaluation for combinations Def\\Con predicts as aligned.","The 30 previously unexplored combinations can serve as a benchmark for future conflict-prediction techniques, since the paper supplies per-defense metrics and ground-truth labels for each.","Because Def\\Con is non-invasive, existing deployed defenses can be combined without reimplementation, which lowers the adoption cost for real systems needing simultaneous protection.","The same flowchart extends beyond two defenses by decomposing multi-way combinations into pairwise checks, and the paper demonstrates effective triple combinations on five illustrative cases."],"supporting_citations":[{"why":"Supplies the two underlying conflict mechanisms and the eight prior-work combinations (C1-C8) used as ground truth for the 90% accuracy result.","marker":"Szyller & Asokan, 2023"},{"why":"Source of the mutually exclusive placement (naïve) technique and of combination C1's ground-truth alignment.","marker":"Yaghini et al., 2023"},{"why":"Provides the ground-truth alignment for combination C8 (differential privacy with explanations).","marker":"Patel et al., 2022"},{"why":"Supplies the DPSGD implementation used as DiffPriv.In across the evaluated combinations.","marker":"Abadi et al., 2016"},{"why":"Supplies the dataset-inference implementation used as Fngrprnt.Post in the evaluation.","marker":"Maini et al., 2021"},{"why":"Supplies the radioactive-data watermarking implementation for DtWM.Pre and explains why C7 aligns with DiffPriv.In.","marker":"Sablayrolles et al., 2020"},{"why":"Supplies the BadNets backdoor implementation used for MdlWM.Pre, DtWM.Pre, and poison generation in PoisnRob defenses.","marker":"Gu et al., 2017"},{"why":"Supplies the pruning implementation used as PoisnRob.Post in the empirical evaluation.","marker":"Zheng et al., 2022b"},{"why":"Supplies the TRADES adversarial training implementation used as EvsnRob.In.","marker":"Zhang et al., 2019"}],"fun_headline_variants":["Def\\Con picks safe ML defense combos, 90% accurate","Clash-free ML defenses: Def\\Con predicts conflicts at 90%","Combine ML defenses without clash: Def\\Con hits 90%","Def\\Con: 90% accurate at spotting conflicting defenses","ML defense combo tester: Def\\Con nails 90% of cases"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The accuracy of Def\\Con rests on the hand-assigned labels in Table 4—each defense has one pipeline stage, one change type, and binary risk-use and risk-protection relations—and on the rule that a combination conflicts if any defense metric on any dataset falls below its single-defense baseline; if those labels or that threshold are wrong or incomplete, the reported accuracy is an artifact of the labeling scheme.","fun_headline_variants_meta":{"raw":{"variants":["Def\\Con picks safe ML defense combos, 90% accurate","Clash-free ML defenses: Def\\Con predicts conflicts at 90%","Combine ML defenses without clash: Def\\Con hits 90%","Def\\Con: 90% accurate at spotting conflicting defenses","ML defense combo tester: Def\\Con nails 90% of cases"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000213,"raw_usage":{"total_tokens":1421,"prompt_tokens":942,"completion_tokens":479,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":558,"completion_tokens_details":{"reasoning_tokens":385}},"tokens_in":558,"tokens_out":479,"duration_ms":4843,"temperature":1.0,"reasoning_tokens":385,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:19:15.546069+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a defense pair with a known empirical outcome, assign the Table 4 attributes, and check whether Def\\Con's prediction matches; repeating this with a defense not in the paper, such as certified robustness or a language-model watermarking scheme, would test whether the two conflict mechanisms generalize. A more direct check is to perturb the labels—for example, marking MdlWM.Pre as global instead of local—and observe whether the accuracy on the paper's own 30 combinations shifts, which would show the result is label-sensitive rather than mechanism-driven.","supporting_citations":[],"review_version":1}