{"id":"134af9f2-5626-4b93-b085-86477742d2c9","arxiv_id":"2607.19430","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"An undefended multi-agent pipeline's apparent perfect attack-safety is mostly the cloud provider's hidden filter, silently re-sourcing to model alignment on other backends; embedding gates restore deterministic application-owned tool-poisoning blocking but fail on adaptive paraphrase.","lead":"An undefended multi-agent LLM pipeline that reports perfect safety on tool- and memory-poisoning attacks is actually protected by the cloud provider's hidden filter (54 of 60 blocks on Azure), and that protection silently re-sources to the model's own alignment on other backends. A cheap six-gate filter moves tool-poisoning blocking into the application layer, while the paper shows exactly where such filters fail against rewritten attacks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'no server-side filter' premise for Anthropic in §6.3 contradicts §1 and is unmeasured; Table 4's model-handled attribution may actually be provider-side blocking, undercutting the complete re-sourcing claim.","rationale":"The reader's CONDITIONAL verdict is sound. The most load-bearing issue is not internal arithmetic (though the GSM8K 0.867 vs 0.833 discrepancy and the Prompt Injection cell inconsistencies are real) but the premise that Anthropic has no comparable provider-side filter. Section 1 names Anthropic's acceptable-use classifier as a provider-side, opaque, stochastic filter outside developer control, while §6.3 asserts the opposite to interpret Table 4. If Anthropic's API applies any server-side safety step, 'model-hdl. 30/30' is not evidence for model alignment; it is the same unknown mechanism the paper claims to expose. The central 'mechanism substitution' story survives only as 'Azure has a detectable filter'; the stronger 'completely re-sources to model behavior' does not. This is externally checkable by instrumenting raw API responses on the exact Tool Poisoning set. The defense's own deterministic 30/30 IB-3 result is independent of this concern, as Corollary B.5 shows, so the paper's practical contribution is not at risk; the attribution contribution is. The reader identified the same weakest assumption, and our conditional recommendation is unchanged.","tokens_in":28010,"tokens_out":5012,"duration_ms":137934,"concrete_test":"Use the released trace corpus or a fresh 30-trace Tool Poisoning run with raw API response logging: send the exact poisoned tool strings used in §6.3 directly to Claude Sonnet 4.5 and Haiku 4.5 through the same client, with all client-side gates disabled, and classify each raw response by whether it returns generated text or a pre-response safety rejection/block (HTTP/content-filter error, empty output with safety reason, or Anthropic's documented policy refusal). If any of the 60 responses is blocked before generation, Table 4's model-hdl. count is inflated and the no-server-filter premise is false. If none are, the Section 1 listing must still be reconciled with the operational definition of 'server-side filter' used for Azure.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central mechanism-substitution result depends on the contrast Azure-has-filter/Anthropic-has-none. Section 6.3 and Table 4 attribute zero ASR on Sonnet/Haiku entirely to 'model-hdl.' (30/30) by asserting these backends have 'no comparable server-side content filter.' But Section 1 explicitly names 'Anthropic's acceptable-use classifier' as a provider-side, opaque, stochastic filter outside developer control. No measurement of Anthropic API-level filtering is reported; the trace schema apparently has no 'prov' category on non-Azure rows, so any such filter would be silently absorbed into 'model-hdl.' If Anthropic's API filters even a subset of the Tool Poisoning probes, the clean '100% model-handled' story collapses: the outcome is still zero ASR, but the 'silently re-sources to alignment' claim misidentifies the mechanism, and the paper's own attribution methodology is what claims to make mechanism observable. This is not a consensus disagreement; it is an unverified assumption that contradicts a sentence in the same paper. The same-vendor counterfactual (two Anthropic sizes) also cannot separate model alignment from shared provider-level safety behavior.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"ChannelGuard addresses the inter-agent channel surface in multi-agent LLM pipelines by placing six (or seven, see below) information-bottleneck gates on the channels between planner, workers, tool, memory, verifier, and synthesizer. Each gate scores channel text by maximum sentence-embedding cosine similarity to a hand-curated adversarial phrase bank and deterministically passes, compresses, or blocks. On 2,100 traces across eight attack families, three backends, and five defenses, the paper reports that an undefended Azure GPT-5 pipeline achieves ASR=0.000 on tool/memory poisoning, but per-trace attribution shows 54/60 of those blocks come from Azure's server-side filter; on Anthropic Sonnet/Haiku the same zero ASR is attributed entirely to model alignment. ChannelGuard's IB-3 gate blocks Tool Poisoning 30/30 identically on all three backends, lowers Prompt Injection ASR from 0.333 to 0.167, preserves GSM8K accuracy, and runs faster on the traced workload. The appendix adds baselines, ablations, sweeps, benign-preservation analysis, a judge audit, and a detailed cost breakdown.","tokens_in":28151,"tokens_out":11442,"duration_ms":106184,"significance":"The paper's core methodological proposal — reporting not only outcome ASR but also which layer stopped each attack — is important and largely well executed. The deterministic gate rule, the released trace corpus and code, the backend-invariance corollary (Appendix B, Corollary B.5), the sentinel-randomization test, and the honest treatment of adaptive-paraphrase and narrative-memory failures are real strengths. If the attribution story survives scrutiny, the paper makes a valuable point: outcome-only ASR can hide dependence on opaque, non-portable provider filters. However, the 'complete re-sourcing' claim on Anthropic backends currently rests on an unmeasured negative assertion that conflicts with the paper's own Section 1, and several reported numbers are mutually inconsistent. The contribution justifies a major revision rather than rejection.","major_comments":[{"comment":"The premise that Anthropic Sonnet 4.5 and Haiku 4.5 have 'no comparable server-side content filter' is asserted without measurement and is in tension with §1, which lists 'Anthropic's acceptable-use classifier' as a provider-side, opaque, stochastic filter. The non-Azure rows of Table 4 have no 'prov' category, so any API-level blocking would be silently absorbed into 'model-hdl.' The same-vendor counterfactual cannot separate model alignment from a shared Anthropic-side filter. Since the attribution methodology is claimed to make the mechanism observable, this is load-bearing for the 'complete re-sourcing' claim. Please (a) report a direct probe of API-level refusals/error codes for the exact poisoned tool strings, (b) add a 'prov' class on non-Azure traces, or (c) reclassify the claim as 'non-attributed/no leak' rather than 'model-handled.'","section":"§6.3, Table 4; §1; §5"},{"comment":"Internal inconsistencies make the point estimates hard to trust as reported. Table 2 gives full ChannelGuard Prompt Injection ASR=0.167; Appendix F's full row gives 0.067; Appendix G's default row gives 0.100. Appendix A's stated '±1 per 30-run cell' drift envelope does not cover the 5-vs-2 leak difference. Separately, §6.5 and the abstract report GSM8K CTCA=0.867, while Appendix J Table 19 reports GSM8K BPR=0.833 (25/30). Also, §6.4 refers to '62 IB-block events,' but Table 5 lists 32 original IB events (30+2+0). Please reconcile these numbers, identify which runs produce which tables, and state whether the paired design permits multiple independent runs of the same nominal configuration.","section":"§6.5/Table 2 vs. Appendices F, G, J; §6.4/Table 5"},{"comment":"The Tool Poisoning 30/30 result is close to construction: the hand-curated phrase bank and the evaluation templates share the same instruction-override and marker-emission families, and Appendix I.5 shows the top four bank phrases account for 87% of BLOCK decisions. The sentinel-randomization test removes literal marker-grep but does not control for shared non-marker template structure. The paper is transparent about static-signature limits in §7, but the abstract and §6.3 present the 30/30 result as evidence of 'deterministic, application-owned' blocking without noting that the attack corpus was built from the same template families as the bank. Please qualify the claim as corpus-specific, or add a transfer/out-of-distribution attack test if broader generality is intended.","section":"§4.1/§5/§6.4; Appendix I.5"}],"minor_comments":[{"comment":"Table 1 is titled 'six gate placements' but lists seven rows (IB-0, IB-1, IB-2, IB-3, IB-3-mem, IB-4, IB-5). The figure caption refers to 'six information bottleneck gates,' treating IB-2/IB-4 as one gate. Please clarify whether IB-2 and IB-4 are separate placements of one gate or two distinct gates, and make the count consistent throughout.","section":"Table 1"},{"comment":"§4.1 says the compress-first-N heuristic is 'safe for inter-agent channels (100% stop-rate)' and §7 repeats that 'the inter-agent gates (IB-2–IB-5)' are 100%, but Table 22 shows IB-1 (planner→worker) has a 2/5 leak rate under COMPRESS. If IB-1 is intentionally excluded from 'inter-agent,' define the term; otherwise the claim is contradicted by the appendix data.","section":"Appendix L, Table 22; §4.1"},{"comment":"The sentinel-randomization denominator is unclear: '62 IB-block events' is not the sum of any column in Table 5 (original sum is 32; randomized sum is 28). Please explain what the 62 counts, and if the table is partial, include the full event set.","section":"§6.4, Table 5"},{"comment":"The judge audit is a strength, but the initial vague rubric producing κ=0.060 and the sharpened rubric recovering κ=0.900 should be described in the main text or at least in the appendix summary, since it shows that the reported agreement is conditional on the rubric naming the literal marker as the success signal.","section":"Appendix N"}],"recommendation":"major_revision","confidential_remarks":"The paper's central idea is timely and the authors are unusually transparent about limitations and reproduction details. However, the provider-invariance story depends on an unverified negative about Anthropic's server-side filtering, and the numerical inconsistencies in ASR, GSM8K, and sentinel counts need to be resolved before the quantitative claims can be relied upon. I would not accept in current form; a major revision with direct provider-filter measurement and a cleaned number audit is appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core finding here is genuinely new: outcome-only ASR hides which layer stopped an attack, and on Azure an undefended pipeline's zero-ASR on tool/memory poisoning is mostly the provider's server-side filter, not the application. The per-trace attribution methodology is the real contribution, and the sentinel-randomization check (94% of IB blocks preserved when the marker is changed) plus the deterministic backend-invariance of the tool-output gate give the mechanism-substitution claim a solid empirical core. I also give the authors credit for reporting their failures—adaptive paraphrase evades the gates, COMPRESS leaks on the input boundary, the verifier has a documented blind spot—instead of hiding them.\n\nThe soft spots are real, but most are fixable. The biggest one: Section 6.3 asserts Anthropic has 'no comparable server-side content filter,' which contradicts the paper's own Section 1 listing Anthropic's acceptable-use classifier as a provider-side filter. The stress-test note is right—Table 4's 'model-hdl.' attribution could silently absorb provider-side blocking, since the trace schema apparently has no 'prov' category for non-Azure runs. Same-vendor counterfactuals (Sonnet vs. Haiku) can't separate model alignment from shared provider behavior. That needs to be measured or qualified, or the 'complete re-sourcing' claim is overstated.\n\nThere are also internal inconsistencies that should not survive review: the main text says GSM8K CTCA is 0.867 for both systems, but Appendix J reports 0.833; the Prompt Injection default cell appears with different ASR/IB-block triples in different tables (e.g., 0.167 with 21 IB blocks in §6.1 vs. 0.067 with 17 IB blocks in the gate ablation), which the ±1 stochastic envelope cannot explain; and the default block threshold is stated as θ=0.60 while the β=1.0 sweep table lists θ_eff=0.70. The artifact URL is a placeholder and the full phrase bank isn't disclosed, which matters for reproducibility.\n\nSet against those fixable issues, the central mechanism-substitution finding holds up as an attribution-count result on a deterministic rule. The paper is honest about its small n and its directional, non-significant Prompt Injection win. This is the kind of work that should get referee time despite needing revision: the methodological point about layered attribution is important for how the field reports agent-safety evaluations. I'd send it to peer review, and I'd want the reviewers to push on the Anthropic premise and the number consistency before acceptance. For a reading group, it's a good springboard for a discussion about evaluation methodology even with the warts.","headline":"The mechanism-attribution result is real and worth taking seriously, but the Anthropic 'no filter' premise and several internal number inconsistencies need fixing before the paper's strongest claims can be trusted.","tokens_in":28852,"tokens_out":1741,"would_cite":true,"duration_ms":17735,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A pipeline with zero reported attack success can owe that safety to the cloud provider's filter, not to the application itself.","keywords":["LLM security","multi-agent systems","prompt injection","tool poisoning","attack success rate","defense-in-depth","information bottleneck","provider-invariance"],"falsifier":"Set up the same tool-poisoning pipeline on a self-hosted open-weight model with comparable instruction-following behavior but no provider API: if zero attack success does not persist, then the off-Azure attribution to 'model alignment' is wrong. Alternatively, send a tool output containing a pattern known to trigger Anthropic's acceptable-use classifier and check whether refusal occurs through the API even when a locally hosted copy of the same model would pass it—a provider-side filter would be confirmed.","tokens_in":27703,"feed_emoji":"🛡️","tokens_out":6560,"duration_ms":52715,"temperature":0.7,"pith_summary":"The paper claims that in multi-agent LLM pipelines the standard attack-success-rate metric is systematically misleading: an undefended pipeline can show zero attack success on tool- and memory-poisoning while 90% of the blocks are performed by the cloud provider's opaque server-side filter (54 of 60 on the Azure backend), not by the application. On a backend without that filter, the same zero attack success silently shifts to the agent model's own alignment. The paper introduces ChannelGuard, a set of six information-bottleneck gates placed on the channels between agents, each scoring text by embedding similarity to an adversarial phrase bank and deterministically passing, compressing, or blocking it. Its tool-output gate blocks tool poisoning 30/30 identically across three backends, converting borrowed, stochastic, provider-owned blocking into deterministic application-owned blocking. A careful reader should care because outcome-only reporting hides which layer actually stops an attack, and that hidden dependence determines whether a deployment's safety survives a backend change.","feed_headline":"A zero attack rate can hide the cloud's filter, not real safety","feed_subtitle":"Attribution shows the same zero-ASR can come from cloud filter or model alignment; ChannelGuard's gate makes it deterministic and app-owned.","key_machinery":"The central object is an information-bottleneck gate: it splits channel text into sentences, embeds each with a local sentence-embedding model, takes the maximum cosine similarity to a fixed bank of adversarial phrases, and maps that scalar to pass/compress/block. Six such gates sit on every inter-agent channel—input, planner-to-worker, tool output, memory read, worker output, synthesizer input—and because the gate never queries the agent model its decision is backend-invariant. A companion attribution method records which layer first stopped each trace (gate, provider filter, verifier, synthesizer refusal, or model-handled), making mechanism substitution measurable.","core_discovery":"Safety does not compose across agent hops: an undefended multi-agent pipeline can report zero attack success while its protection is an artifact of the backend. Per-trace attribution shows that on Azure GPT-5 the provider filter accounts for 54 of 60 zero-ASR blocks on tool- and memory-poisoning; on Anthropic backends the same zero ASR comes entirely from model alignment. ChannelGuard's tool-output gate instead blocks tool poisoning 30/30 at the application layer on all three backends, because its decision depends only on channel text, a fixed phrase bank, and a local embedding model. It also halves prompt-injection ASR (0.333 to 0.167), preserves GSM8K accuracy exactly (0.867), runs 1.19x f","pith_inferences":["If mechanism substitution generalizes, multi-agent safety benchmarks should report attribution classes alongside attack success rate; otherwise apparent gains from a defense could actually be shifts in provider filtering or model alignment.","The tool-output gate's backend invariance implies a composable building block: any channel carrying untrusted content (tool results, retrieved documents, memory reads) can be gated independently of the agent model, so a defense can be validated once and reused across providers.","The paper's adaptive-paraphrase failure suggests testing gates against paraphrase families generated from the phrase bank itself, and learning the bank from an adaptively generated attack corpus—a direction the authors name as the next step.","A third backend that is neither Azure nor Anthropic would settle whether the off-Azure zero ASR on memory-poisoning is genuinely model alignment or another hidden provider layer."],"forward_implications":["Deployers who report only attack success rate cannot tell whether a zero is produced by the cloud provider, the model's alignment, or their own defense; per-trace attribution is needed to reason about robustness.","Application-owned gates at inter-agent channels convert borrowed, stochastic provider blocking into deterministic, inspectable blocking that is invariant to backend choice, at least for tool-poisoning attacks.","ChannelGuard's phrase-bank gates halve prompt-injection attack success and preserve reasoning accuracy on benign tasks, while an early-exit block short-circuits the pipeline before expensive LLM calls, giving a 3.30x speedup on prompt injection.","Static signature-based gates have a bounded ceiling: white-box adaptive paraphrase evades them, and a perturb-and-vote baseline does better, so no single mechanism dominates.","The provider filter and the gates catch overlapping but non-identical attack subsets, so the right deployment is defense-in-depth rather than substitution.","The paper's own verifier audit exposes a blind spot: a safe-but-uncertain band leaks 19.5% of the time, suggesting a design fix that would need a benign-preservation trade-off measurement."],"fun_headline_variants":["Zero attack success can hide a cloud filter, not real safety","Multi-agent safety doesn't compose—zero ASR is backend luck","ChannelGuard gates each hop: deterministic zero-ASR on all backends","Why your multi-agent zero-ASR may be a provider artifact"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The paper assumes the Anthropic backends have no comparable server-side content filter, so it attributes the undefended zero attack-success rate on those backends entirely to model alignment; if Anthropic's API applies any hidden safety layer—and the paper itself lists Anthropic's acceptable-use classifier as a provider-side filter in Section 1—the mechanism-substitution story misidentifies the cause.","fun_headline_variants_meta":{"raw":{"variants":["Zero attack success can hide a cloud filter, not real safety","Multi-agent safety doesn't compose—zero ASR is backend luck","ChannelGuard gates each hop: deterministic zero-ASR on all backends","Why your multi-agent zero-ASR may be a provider artifact"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000336,"raw_usage":{"total_tokens":1793,"prompt_tokens":936,"completion_tokens":857,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":680,"completion_tokens_details":{"reasoning_tokens":781}},"tokens_in":680,"tokens_out":857,"duration_ms":15147,"temperature":1.0,"reasoning_tokens":781,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T15:22:03.204786+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Set up the same tool-poisoning pipeline on a self-hosted open-weight model with comparable instruction-following behavior but no provider API: if zero attack success does not persist, then the off-Azure attribution to 'model alignment' is wrong. Alternatively, send a tool output containing a pattern known to trigger Anthropic's acceptable-use classifier and check whether refusal occurs through the API even when a locally hosted copy of the same model would pass it—a provider-side filter would be confirmed.","supporting_citations":[],"review_version":1}