{"id":"f11e1f55-9ba0-48cf-b833-58140c35a9dc","arxiv_id":"2501.11795","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"The paper defines 'effective' poisoning so that effective attacks force an empty intersection of conformal prediction sets, then presents experiments with a new test that observes that intersection.","lead":"This paper proposes a mathematical definition of effective data poisoning and introduces a conformal prediction based test that it says can provably catch any attack matching that definition. The authors demonstrate the test on CIFAR-10 and GTSRB with competitive false positive and false negative rates.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 3.39's bound is unsupported: its premise (P effectively poisoned wrt D) makes D and P dependent, while Lemma 3.38's (1−ε)^2 bound requires independence, which also cannot hold when both sequences share the test point.","rationale":"The reader's weakest_assumption correctly identifies the independence premise in Lemma 3.38 as load-bearing, and I agree that it is violated by the paper's own attack model. My stress-test sharpens this concern: the independence is not merely unrealistic, it is logically impossible when the two sequence-valued random variables share the test point (X_{n+1},Y_{n+1}), because that component would be measurable with respect to both sigma-algebras and hence almost surely constant. This makes Corollary 3.26 and Lemma 3.38 vacuous as stated. Furthermore, Theorem 3.39 is asserted without proof, and its hypothesis 'P is an effectively poisoned dataset' is undefined; Definition 3.33 only defines pointwise effectiveness at a particular (x,y). Even granting an appropriate interpretation, the proof would need to show that pointwise empty intersection implies the sequence-level separability event has probability below (1−ε)^2, and the only route offered is the independence lemma, which cannot apply. The paper's experimental results are plausible and competitive as heuristics, but they do not repair the theoretical gap; the central claim of provable detection with the stated bound is unsupported. I see no reason to change the reader's REJECT verdict.","tokens_in":22347,"tokens_out":11747,"duration_ms":109925,"concrete_test":"Re-run the authors' CIFAR10 experiment (poison rate 0.1%, conformal threshold ε=0.1) and compute the empirical frequency of p∩(T,Tr,x) ≤ 0.1 over the 500 held-out poisoned samples. Lemma 3.38 predicts this frequency is at most 1−(1−0.1)^2 = 0.19; Table 3 implies it is about 97.8% (FNR=2.2%). If the recomputed frequency is close to 97.8% rather than ≤19%, the independence-based bound does not hold for the dependent pair (T,Tr), confirming that Theorem 3.39 cannot be established via Lemma 3.38.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Theorem 3.39 asserts that D and P are conformally (ε,ε,(1−ε)^2)-separable whenever P is effectively poisoned w.r.t. D. The only quantitative argument available for this bound is Lemma 3.38/Corollary 3.26, which requires ⟨D1,(X_{n+1},Y_{n+1})⟩ and ⟨D2,(X_{n+1},Y_{n+1})⟩ to be independent sequence-valued random variables. This premise is unsatisfiable in two ways. First, in the threat model (Definition 3.11), P is defined as D_C ⊎ t(D_P) via a Markov kernel from the very same D; the paper's experiments use Tr = T with poison applied, so D and P are functions of the same source and hence dependent. Second, even abstracting from the attack, if the two sequences share the last component (X_{n+1},Y_{n+1}), that component is measurable w.r.t. both σ-algebras, so independence forces it to be almost surely constant. The product bound P(Γ^ε)P(Γ^ε) ≥ (1−ε)^2 therefore cannot be invoked. Additionally, Theorem 3.39 is stated without proof, and its hypothesis 'P is an effectively poisoned dataset' is never formally defined; Definition 3.33 only gives pointwise empirical effectiveness at a particular (x,y). The step from pointwise empty intersection (Lemma 3.37) to the sequence-level event Γ^ε ∩ Γ^ε in Definition 3.24 is not provided. Thus the central guarantee reduces to the tautology that an attack defined to produce empty conformal intersection is detectable by a test for empty intersection.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a probabilistic formalization of trigger-based dataset poisoning attacks using exchangeability and Markov kernels, and introduces a Conformal Separability Test that compares conformal prediction sets of a clean dataset and a poisoned dataset. The central theoretical claim (Theorem 3.39) is that if a dataset P is 'effectively poisoned' with respect to an exchangeable dataset D, then D and P are conformally (ε, ε, (1−ε)^2)-separable, meaning the conformal prediction sets at a test point are disjoint with probability at least (1−ε)^2. The authors also present experiments on CIFAR-10 and GTSRB patch attacks and on the Witches' Brew clean-label attack, reporting false-negative and false-positive rates competitive with or better than prior work.","tokens_in":22701,"tokens_out":4501,"duration_ms":42837,"significance":"If the main theorem were valid, the paper would make a substantial contribution: an information-theoretic, polynomial-time defense against dataset poisoning with a finite-sample detection guarantee would be a significant advance over optimization-based defenses that require exponential time. The paper also contains a helpful high-level framing of poisoning via exchangeable sequence-valued random variables, and the experimental comparison with state-of-the-art proactive defenses is a useful data point. However, the central mathematical guarantee is not established: Theorem 3.39 is asserted without proof, the independence premise needed for the (1−ε)^2 bound is unsatisfiable in the paper's own attack model, and the definition of 'effective poisoning' is so close to the detection event that the main theorem risks being a tautology. The paper also does not provide a valid symmetric non-conformity score for the stochastic training procedures used in the experiments. These issues are load-bearing, so the paper in its current form cannot be accepted.","major_comments":[{"comment":"Theorem 3.39, which is the paper's main theoretical claim, is stated without proof. The preceding text says only that the result 'summarizes the observations in this section,' but the available ingredients are pointwise: Lemma 3.37 shows that if an attack is empirically (1−r)-effective at (x, y), then Γ^r_n(D)(tx) ∩ Γ^r_n(P)(tx) = ∅. No argument is supplied that the pointwise condition, or an expectation over (x, y), yields the sequence-level event Γ^ε_{n,A} ∩ Γ^ε_{n,A} of Definition 3.24 with the claimed probability bound. In addition, the hypothesis 'P is an effectively poisoned dataset with respect to D' is never formally defined; Definition 3.33 only defines empirical (1−r)-effectiveness at a particular point (x, y) for given D and P. The main theorem is therefore unsupported as stated.","section":"§3.4, Theorem 3.39"},{"comment":"The probability bound (1−ε)^2 requires the two sequence-valued random variables ⟨D1, (X_{n+1}, Y_{n+1})⟩ and ⟨D2, (X_{n+1}, Y_{n+1})⟩ to be independent. In the threat model of Definition 3.11, however, P is obtained from D as the composite of the Markov kernels †, t^k×id, and ⊎, so P is a function of D; the experiments in Section 4 likewise compare T with Tr, where Tr is T with poison applied. Thus D and P are dependent, and the independence premise is not satisfied. Even abstracting from the attack mechanism, the two sequences share the final component (X_{n+1}, Y_{n+1}), which is measurable with respect to both σ-algebras; for independence, that shared component must be almost surely constant. The product bound in Lemma 3.38 therefore cannot be invoked, and Theorem 3.39's quantitative claim is unsupported.","section":"§3.4, Lemma 3.38 and Corollary 3.26"},{"comment":"The definition of empirical (1−r)-effectiveness is formulated directly in terms of the p-values that define the conformal sets: condition [re.1] states pD(tx, ty) < min(r, pD(tx, y)), and condition [re.2] states max(τ^P(tx) − ty) ≤ r < pP(tx, ty). Lemma 3.37 then derives the empty-intersection event Γ^r_n(D)(tx) ∩ Γ^r_n(P)(tx) = ∅ from exactly those conditions. Consequently, the statement that an effective attack is detectable by an empty-intersection test is close to a restatement of the definition of effectiveness. The nontrivial content that is missing is a distributional model and a probability bound for the event that an attack satisfying Definition 3.33 occurs; without that, Theorem 3.39 does not provide an independent detection guarantee.","section":"§3.4, Definition 3.33 and Lemma 3.37"},{"comment":"The experiments use the non-conformity score A(D, x, y) = −entropy(softmax(M_D(x))) / (s_{x,y} ln s_{x,y}), where M_D is trained with Adam, weight decay, and learning-rate schedulers. Such a training procedure is stochastic and is not invariant to permutations of the training set D, so A is not a symmetric non-conformity score as required by Definition 3.15. The paper itself notes in Example 3.17 that stochastic gradient descent or batching would require a convergence result to yield a non-conformity score. No such result is supplied for the training procedure used in Section 4, so the empirical FNR/FPR tables do not inherit the conformal validity guarantee of Theorem 3.22.","section":"§4, Experimental Setup and non-conformity score"}],"minor_comments":[{"comment":"The word 'conformal' is misspelled as 'confomal' in the statement of Corollary 3.26.","section":"§3.3, Corollary 3.26"},{"comment":"In the sentence 'The probability of p∩(D1, D2, ) ≤ ϵ', the argument x is missing; it should be p∩(D1, D2, x) ≤ ϵ.","section":"§3.4, Lemma 3.38"},{"comment":"Reference [53] has the year '2956'; this should be 1956.","section":"References"},{"comment":"There are several formatting and spelling errors, including 'c´adl´ag' with broken accents, 'na¨ıve', 'IMDB' instead of 'IMDb', and 'D, Pare' missing a space; a careful proofreading pass is needed.","section":"Throughout"}],"recommendation":"reject","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Angela—quick take on 2501.11795. The paper is not dead on arrival: the formalization of trigger attacks via Markov kernels and exchangeable sequences is a real step forward, and the Conformal Separability Test is a sensible, polynomial-time heuristic that performs competitively on patch attacks and the Witches' Brew attack. That part deserves credit.\n\nThe problem is the advertised theorem. Theorem 3.39 says that an 'effectively poisoned' dataset is conformally (ε,ε,(1−ε)^2)-separable from the clean one. It is stated without proof, and the only quantitative route offered is Lemma 3.38/Cor 3.26, whose (1−ε)^2 bound requires the two sequence-valued random variables ⟨D1,(X_{n+1},Y_{n+1})⟩ and ⟨D2,(X_{n+1},Y_{n+1})⟩ to be independent. That premise is not satisfied in the paper's own threat model: P is built from D as D_C ⊎ t(D_P), so D and P are dependent. And even ignoring the attack, if both sequences share the same last component, that component is measurable with respect to both σ-algebras; independence would force it to be almost surely constant. So the product bound cannot be invoked. The stress-test note is right on both counts.\n\nThere is also a circularity concern that is not just a stylistic quibble. 'Empirically (1−r)-effective' (Def 3.33) is defined via p-value inequalities on both the clean and poisoned datasets; Lemma 3.37 then derives empty conformal intersection. The main theorem's hypothesis essentially builds in the conclusion. The step from pointwise effectiveness to the sequence-level event in Theorem 3.39 is not supplied. So the 'provable detection of effective attacks' claim is closer to a restatement than a discovery.\n\nThe experiments are decent and honestly reported, but they do not repair the theory. One small thing: the non-conformity score uses stochastically trained networks, so the exchangeability premise for the score itself is shaky, and at r=0 the test flags 9.8% of CIFAR poisoned hold-out items as positive even though Tr=T—worth a closer look.\n\nWho is this for? People working on practical poison detection will want the Conformal Separability Test and the experimental comparisons. The theoretical guarantee as stated should not be cited until the independence issue is fixed and Theorem 3.39 actually gets a proof. I'd send it to a serious referee: the kernel/exchangeability framework is novel enough and the flaw is identifiable enough that a rewrite could yield a solid contribution. But my own verdict remains skeptical of the central claim.","headline":"A genuinely novel conformal framework for poison detection whose headline provable guarantee collapses on the independence premise; the experiments are competitive but do not rescue the theory.","tokens_in":23220,"tokens_out":2019,"would_cite":false,"duration_ms":19467,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62G15","62F03","68T05"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proves that any effectively poisoned dataset must be statistically separable from a clean dataset, and gives a polynomial-time test that detects the separation with a bounded error rate.","keywords":["data poisoning","conformal prediction","trigger attack","exchangeability","conformal separability test","proactive defense","backdoor attacks","information-theoretic security"],"falsifier":"Measure the empty-intersection rate $\\hat{P}(p_\\cap(D,P,x)\\le \\epsilon)$ for any attack that satisfies the paper's own definition of empirical effectiveness, and compare it with $1-(1-\\epsilon)^2$; finding a rate above this bound (for example above 0.19 at $\\epsilon=0.1$) would disprove Theorem 3.39 as stated.","tokens_in":22121,"feed_emoji":"🛡️","tokens_out":8692,"duration_ms":84904,"temperature":0.7,"pith_summary":"Dataset poisoning is usually framed as an optimization problem, which makes detection as hard as solving the attack. This paper formalizes a trigger poison attack as a stochastic transformation of an exchangeable training set and asks what an effective attack necessarily does to the geometry of conformal prediction sets. Its central claim is that effectiveness forces separation: when a poisoned dataset is compared with a clean dataset at a triggered input, the probability that both conformal prediction sets contain the same label is at least $(1-\\epsilon)^2$, so an empty intersection is a provably rare event that exposes the poison. The resulting Conformal Separability Test runs in polynomial time, makes no assumptions about how the attacker constructs the poison, and carries a finite-sample detection guarantee. Experiments on patch attacks over CIFAR-10 and GTSRB and on the Witches' Brew clean-label attack report false-negative and false-positive rates competitive with a proactive-defense baseline.","feed_headline":"Effective data poisoning leaves a provably detectable trace","feed_subtitle":"A polynomial-time conformal separability test flags poisoned datasets with a bounded error rate and no attack-specific assumptions.","key_machinery":"The carrying object is the conformal prediction $p$-value function $p_{n,A}$, which scores how typical a label is for a given input relative to a dataset, and Algorithm 1's Conformal Separability Test, which computes $p_\\cap(D_1,D_2,x) = \\max_y \\min(p_{D_1}(x,y), p_{D_2}(x,y))$ as the empirical probability that the two datasets' conformal sets overlap at $x$. The machinery pairs this with the notion of an expiry vector $\\tau^D(x)$, the per-label $p$-values that determine when each label leaves the prediction set. Lemma 3.37 converts empirical effectiveness into an empty intersection, and Corollary 3.26 plus Theorem 3.25 convert exchangeability and independence of the two sequences into the lower bound $(1-\\epsilon)(1-\\gamma)$ on that intersection; Theorem 3.27 invokes Glivenko-Cantelli to justify estimating the bound from finite samples.","core_discovery":"The paper's central discovery is that 'effective' can be defined so that it mathematically implies detectability. An attack is empirically $(1-r)$-effective when, on triggered inputs, the target label becomes the only label in the poisoned conformal prediction set while it was absent from the clean one at confidence above $r$. Lemma 3.37 shows such an attack forces $\\Gamma^r_n(D)(tx) \\cap \\Gamma^r_n(P)(tx) = \\emptyset$; Lemma 3.38 shows that, for independent exchangeable samples, this empty intersection has probability at most $1-(1-\\epsilon)^2$; Theorem 3.39 assembles these into the statement that $D$ and $P$ are conformally $(\\epsilon,\\epsilon,(1-\\epsilon)^2)$-separable. The authors take this to mean that the act of effectively poisoning a dataset guarantees the attack can be detected with better-than-guessing probability, even when the adversary is computationally unbounded, and that no poisoned dataset and clean dataset can be IID as sequence-valued random variables.","pith_inferences":["Editorial: The strongest point to stress-test is the independence premise in Lemma 3.38, since the poisoned set is defined as a transformation of the clean set; verifying whether the bound survives the resulting dependence is a direct next step.","Editorial: The sharp jump in attack success between 0.002% and 0.01% poison rates suggests a threshold effect; if confirmed, defenses might aim to keep poison below the threshold rather than remove all poison.","Editorial: The paper's conclusion explicitly leaves open the possibility of poison items that are identically distributed to clean items while still breaking exchangeability, so the theorem's scope is exchangeability violation rather than all distributional equivalence.","Editorial: The paper only reports empirical false-positive rates, so a deployment-grade guarantee would require a separate analysis of the test's behavior when both datasets are clean."],"forward_implications":["Effective attacks are detectable with a guaranteed bound on missed detections, so a defender can remove poisoned items with a quantifiable residual risk.","The guarantee is independent of the attacker's computational power; the paper claims it covers even non-computable poison functions.","The test is polynomial-time, unlike optimization-based provable detection, which the paper notes requires exponential time in the worst case.","In the paper's experiments, filtering with the test reduced the Witches' Brew attack's success rate from 35% to 3.5% at a 3% false-positive rate.","The results imply an effective poison can never make poisoned data exchangeable with clean data, so 'effective' and 'statistically invisible' are incompatible for this attack model."],"supporting_citations":[{"why":"Supplies the conformal prediction validity theorem (probability at least $1-\\epsilon$) on which the separability bound rests.","marker":"[41]"},{"why":"Provides the proof of conformal validity cited for Theorem 3.22.","marker":"[42]"},{"why":"Supplies the Dvoretzky-Kiefer-Wolfowitz inequality used to justify the finite-sample empirical test in Theorem 3.27.","marker":"[53]"},{"why":"Provides the optimization-based provable detection approach and patch-generation method used as baseline and experimental template.","marker":"[4]"},{"why":"Defines the Witches' Brew clean-label attack used in experiments and the claim about supervision that the paper tests.","marker":"[3]"},{"why":"Provides the proactive backdoor-poison detection baseline whose false-positive and false-negative rates are compared.","marker":"[6]"},{"why":"Shows neural-network conformal predictions often have non-singleton sets, motivating the max-min intersection test.","marker":"[44]"}],"fun_headline_variants":["Effective poisoning ensures detection, provably","To poison effectively is to be caught, provably","Effective attacks are provably detectable by definition","Poisoning effectively guarantees a detection test"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result's probability bound depends on treating the clean and poisoned datasets as independent random sequences, but a real poison attack constructs the poisoned set from the clean set, so that independence is not automatic and the paper does not prove it.","fun_headline_variants_meta":{"raw":{"variants":["Effective poisoning ensures detection, provably","To poison effectively is to be caught, provably","Effective attacks are provably detectable by definition","Poisoning effectively guarantees a detection test"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000167,"raw_usage":{"total_tokens":1184,"prompt_tokens":800,"completion_tokens":384,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":416,"completion_tokens_details":{"reasoning_tokens":328}},"tokens_in":416,"tokens_out":384,"duration_ms":3992,"temperature":1.0,"reasoning_tokens":328,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T17:51:37.550790+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the empty-intersection rate $\\hat{P}(p_\\cap(D,P,x)\\le \\epsilon)$ for any attack that satisfies the paper's own definition of empirical effectiveness, and compare it with $1-(1-\\epsilon)^2$; finding a rate above this bound (for example above 0.19 at $\\epsilon=0.1$) would disprove Theorem 3.39 as stated.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the conformal prediction validity theorem (probability at least $1-\\epsilon$) on which the separability bound rests."},{"cited_title":"Learning by transduction,","cited_arxiv_id":null,"evidence_quote":"Provides the proof of conformal validity cited for Theorem 3.22."},{"cited_title":"Asymptotic minimax charac- ter of the sample distribution function and of the classical multinomial estimator.,","cited_arxiv_id":null,"evidence_quote":"Supplies the Dvoretzky-Kiefer-Wolfowitz inequality used to justify the finite-sample empirical test in Theorem 3.27."},{"cited_title":"Rethinking backdoor at- tacks,","cited_arxiv_id":null,"evidence_quote":"Provides the optimization-based provable detection approach and patch-generation method used as baseline and experimental template."},{"cited_title":"Towards a proactive ml approach for detecting backdoor poison samples,","cited_arxiv_id":null,"evidence_quote":"Provides the proactive backdoor-poison detection baseline whose false-positive and false-negative rates are compared."}],"review_version":1}