{"id":"61819658-02b0-4536-aa8d-1c98c447f7e7","arxiv_id":"2607.14754","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"FlowGuard detects MCP tool vulnerabilities by verifying runtime evidence rather than treating suspicious metadata or responses as proof.","lead":"FlowGuard is a new security scanner for MCP-based AI agents that confirms real runtime evidence — actual command output, leaked files, or exposed secrets — instead of trusting suspicious-looking text. It reports strong detection rates on a new 1,880-case benchmark and on real MCP servers, which could make agent tool ecosystems safer.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark labels and FlowGuard's adjudication rule share the same operational definition, so headline F1 may measure self-consistency rather than independent detection accuracy.","rationale":"The reader's weakest assumption identifies exactly the concern that is most load-bearing: if benchmark ground truth is defined using the same evidence categories and decision rule as FlowGuard's adjudicator, then the F1 numbers do not independently validate the central claim that FlowGuard 'verifies execution-related findings' in a way that outperforms existing scanners in the wild. The concern is not that the labels are fabricated — the benchmark includes executable servers and canary-style evidence, and the paper reports validation runs that confirm positive cases produce the expected markers. But independence of ground truth is a precondition for interpreting absolute F1 as evidence-grounded accuracy; without it, the numbers primarily demonstrate that FlowGuard is well-calibrated to the authors' own definition of a real finding. The MarkItDown case study and manual verification of real-world findings provide complementary support, which is why this is not a rejection-level objection. It does, however, justify maintaining the CONDITIONAL verdict: the benchmark must be independently labeled or the artifact released for third-party evaluation. My proposed test — independent human re-labeling of a stratified sample — directly settles whether the shared definition materially inflates the headline F1. If the test shows minimal change, the concern is resolved; if it shows a significant drop, the central performance claim would need to be scaled back. For these reasons, I agree with the reader and recommend no change to the verdict.","tokens_in":25838,"tokens_out":5796,"duration_ms":60651,"concrete_test":"Have two independent security researchers, who have not seen FlowGuard's prompts or adjudication code, re-label a stratified random sample of the 1,880 benchmark servers using only the server source code, tool metadata, and a predefined vulnerability rubric (e.g., CWE-78/CWE-22). Then recompute FlowGuard's F1 against these external labels, excluding categories where labels are genuinely semantic (Tool Poisoning, Prompt Injection). If the externally-labeled F1 on Command Injection and File System Access drops by more than 0.05, the shared evidence definition inflates the headline results; if it remains within noise, the circularity concern is refuted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central performance claim depends on the 1,880-case benchmark having ground-truth labels that are independent of FlowGuard's decision procedure. That independence is not established. Section IV-A states positive labels are assigned when a case has 'category-specific evidence, such as command markers, fixture file contents, synthetic canaries, or instruction patterns,' and negative cases are those whose 'safe implementation should not produce the corresponding evidence.' This is the same operational criterion encoded in FlowGuard's evidence adjudication rule in Section III-D2 and Prompt C: R = S ∧ O ∧ ¬E (signal identified, system-originated, not expected behavior). The benchmark's oracle and FlowGuard's adjudicator thus implement the same definition of 'real finding.' Consequently, the reported F1 of 0.879 for Command Injection and 0.942 for File System Access may reflect how faithfully FlowGuard reproduces the authors' evidence definition, not how well it identifies vulnerabilities according to an external, independently grounded standard. The baselines are compared on the same labels, so relative rankings may still be meaningful, but the absolute F1 values and the claim that FlowGuard 'verifies execution-related findings' are weaker than they appear if the ground truth is not truly independent. The paper does include one strong independent check — the MarkItDown file:///etc/passwd finding was externally confirmed via the project's issue tracker — but this is a single case and does not validate the benchmark-wide F1 numbers.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents FlowGuard, a black-box MCP security scanner that separates semantic risk detection from execution-evidence verification. It operates through five stages: triage of tool metadata, recon probes to extract backend fingerprints, schema-valid strike probes, response analysis with evidence adjudication using rule R = S ∧ O ∧ ¬E, and history-guided refinement. The authors introduce an executable benchmark of 1,880 MCP cases across five categories (Command Injection, Tool Poisoning, Prompt Injection, Credential Leakage, File System Access) and report F1 scores of 0.879, 0.995, 0.957, 0.864, and 0.942, respectively, outperforming MCPScan, MCP-Scanner, and A.I.G, with up to 2.23× lower latency than A.I.G. On 8,000 real-world MCP servers, FlowGuard reports 523 findings in 326 servers; manual verification of 100 servers confirms concrete evidence in 84, and one case (MarkItDown file:// disclosure) was externally acknowledged. The paper claims that evidence-grounded detection can assess both execution-related and semantic risks in MCP interactions.","tokens_in":26212,"tokens_out":9600,"duration_ms":90562,"significance":"If the results hold, FlowGuard would be a useful contribution: it directly addresses a real limitation of signal-based MCP scanners, and the structured verification loop with explicit adjudication is a principled design. The paper's strengths include extensive ablations (triage/recon/feedback, E1/E2 adjudication), an LLM-backbone sensitivity study, a novel scanner-level benchmark, and one externally confirmed real-world case. The main weakness is that the benchmark's ground truth and FlowGuard's decision rule share the same evidence definition, so the absolute F1 numbers may overstate performance against an independent standard. Reproducibility is also currently limited by the absence of a released artifact. The central idea is defensible; the evaluation needs strengthening before the headline claims can be accepted.","major_comments":[{"comment":"The benchmark's ground-truth labels and FlowGuard's adjudication rule are defined by the same operational evidence criterion. Section IV-A labels a case positive when it has 'category-specific evidence, such as command markers, fixture file contents, synthetic canaries, or instruction patterns' and negative when the safe implementation 'should not produce the corresponding evidence.' Prompt C defines a real finding as R = S ∧ O ∧ ¬E (signal identified, system-originated, not expected behavior). These are two formulations of the same test. The F1 values in Table II therefore measure, to a large degree, how faithfully FlowGuard reproduces the authors' evidence taxonomy rather than how well it detects vulnerabilities against an independent ground truth. Hard negatives mitigate this partially but do not remove the overlap. Please (i) release the benchmark and show label generation did not us","section":"§IV-A and §III-D2/Prompt C"},{"comment":"The benchmark is not neutral across detection modalities. Since positive labels are defined by runtime response evidence, static (MCPScan) and metadata-only (MCP-Scanner) baselines are evaluated on a target that their design cannot in principle satisfy; their very low F1 scores in Table II are partly an artifact of benchmark design rather than a measure of detection quality. The comparison against A.I.G, which also probes at runtime, is more meaningful. I recommend restricting the headline comparison to dynamic baselines, or providing modality-appropriate ground truth (e.g., code-level labels for static scanners) so that the reported 'best overall results' reflect detection accuracy rather than modality fit.","section":"§V-A, Table II"},{"comment":"The paper introduces a new 1,880-case benchmark and a full system but provides no release artifact, code, or benchmark URL. This is a stated contribution, and the central performance claims rest entirely on this self-built benchmark. Without the benchmark generator, label-assignment scripts, and scanner implementation, the claims cannot be independently verified or reused. A reproducibility appendix with these artifacts, or a clear availability statement, is essential before publication.","section":"§IV, §V"},{"comment":"Table II's footnote excludes 'Error' runs from denominators. A.I.G has 42 errors on Prompt Injection, 19 on Command Injection, and 2 on File System Access, while FlowGuard has none. If failed runs are non-random (e.g., timeouts on hard cases), this exclusion systematically favors FlowGuard. Please report error-inclusive precision/recall/F1 (treating errors as false negatives) or at least provide per-method error rates. This is needed for fair cross-scanner comparison.","section":"Table II, §V-B"}],"minor_comments":[{"comment":"For Credential Leakage, FlowGuard's Evidence Calls percentage (30.04%) is lower than A.I.G's (40.56%), yet FlowGuard obtains more true positives. Define what counts as a 'security-relevant signal' for each scanner; otherwise the metric is not comparable across systems.","section":"Table III"},{"comment":"'Fluffy's parameter to sink observations' is mentioned without a citation; please add the reference for this knowledge source.","section":"§III-C2"},{"comment":"Report whether probing budget B, timeout, and patience k were held constant across LLM backbones; temperature alone is not sufficient for a controlled comparison.","section":"Figure 5, §V-E4"},{"comment":"'OW ASP' should be 'OWASP' throughout (e.g., §IV-A, §IV-B, references [10], [17]).","section":"Global typo"},{"comment":"Specify the popularity criterion for selecting the top 8,000 MCPZoo servers and report connection/interaction success rates; this affects external validity of the real-world numbers.","section":"§V-F"},{"comment":"The row 'Total 82 24 –' is confusing; clarify the counts and column semantics.","section":"Appendix B, Table VI"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope and the topic is timely. I do not suspect any author misconduct. The main risk is that the benchmark may not provide an independent test of FlowGuard's central claim; this can be addressed with a released artifact and independent labels. I would support acceptance after major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is a competent systems paper that deserves a real referee, but the headline F1 numbers should not be taken at face value because the benchmark's ground-truth definition and FlowGuard's adjudication rule are the same rule.\n\nWhat's actually new: FlowGuard is not a breakthrough in any single technique. The components — semantic triage, recon-guided payload narrowing, schema-valid probes, evidence adjudication, history-guided refinement — all exist in web scanning and agent security. The contribution is the combination, tuned to MCP's structure, plus a new 1,880-case executable benchmark with paired positive/negative servers. The evaluation is thorough for a systems paper: ablations of triage, recon, feedback, and adjudication; probe-efficiency metrics; latency CDFs; LLM backbone sensitivity; and a real-world run over 8,000 servers. The MarkItDown case study, where a file:// probe in convert_to_markdown was confirmed via the project's issue tracker, is a genuinely nice external check.\n\nThe soft spots are real but not fatal. The central concern is circularity: Section IV-A labels positives by 'category-specific evidence, such as command markers, fixture file contents, synthetic canaries, or instruction patterns,' which is the same operational criterion encoded in the adjudication rule R = S ∧ O ∧ ¬E in Prompt C. So the F1 of 0.879 on Command Injection and 0.942 on File System Access may partly measure how faithfully FlowGuard reproduces the authors' own evidence definition, not how well it detects vulnerabilities by an independent standard. The baselines are scored on the same labels, so the relative ranking is still meaningful, but the absolute numbers are weaker than they look.\n\nAlso, no code or benchmark is released, all metrics are single-run with no variance, and the fast-path risk knowledge base is summarized but not fully specified. Minor but worth noting: the triage ablation on Command Injection shows recall drops when triage is removed — the paper interprets this correctly as a cost-control trade-off, but it does mean the current triage policy is conservative for execution-path risks.\n\nBottom line: this is a solid, honest systems paper with a genuine new benchmark and a plausible mechanism. It belongs in peer review, not a desk reject. The referee should push for released artifacts, repeated runs or error bars, and ideally an independent or externally validated benchmark before the absolute F1 claims are taken as established.","headline":"A well-built MCP scanner with a thorough evaluation, but the benchmark and the adjudication rule share the same evidence definition, so the absolute F1 numbers should be read with caution.","tokens_in":26642,"tokens_out":2167,"would_cite":true,"duration_ms":19870,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FlowGuard claims that MCP security detection should verify execution-related risks through runtime evidence and handle semantic risks in metadata and returned content separately, and that its central adjudication rule—a finding is real only","keywords":["Model Context Protocol security","evidence-grounded detection","runtime probing","command injection","file system access","tool poisoning","prompt injection","credential leakage"],"falsifier":"Re-label the 1,880 benchmark cases by inspecting backend source code or having independent human analysts decide vulnerability status, without reference to FlowGuard's adjudication rule, then rerun FlowGuard and measure F1; if the scores drop substantially below 0.879 and 0.942, the performance claim is partly an artifact of circular labeling. A simpler check: any server that returns the probe string verbatim should never be counted as a finding—if a scan reports it as credential leakage, that is a false positive by the paper's own definition.","tokens_in":25768,"feed_emoji":"🛡️","tokens_out":4269,"duration_ms":43063,"temperature":0.7,"pith_summary":"FlowGuard argues that suspicious signals in MCP interactions are not enough: a credential-like string can be a placeholder, a command-like output can be reflection, and a rejection can be correct defense. Instead, the paper proposes an evidence-grounded scanner that verifies execution-related risks through runtime probing and adjudicates returned content by source attribution and behavioral expectation. On a new 1,880-case executable benchmark, it reports F1 scores of 0.879 for command injection and 0.942 for file system access, and on a real-world corpus it reports 523 findings across 326 servers. If true, this shifts MCP security from flagging suspicious text to confirming actual exploitability.","feed_headline":"Catch MCP attacks with runtime proof","feed_subtitle":"FlowGuard verifies command injection and file access through execution evidence, not just suspicious text.","key_machinery":"The load-bearing mechanism is the adjudication rule R = S ∧ O ∧ ¬E, applied to every candidate response signal: signal identified, system-originated (not echo reflection), and not expected defensive behavior. Around this rule, the pipeline is a closed-loop verification cycle: Triage ranks parameters for risk; Recon sends schema-valid low-impact probes to extract backend fingerprints; Strike generates a single targeted payload per parameter and threat family; Analysis extracts candidate signals; Adjudication applies the evidence rule; Refinement updates a probe history and decides whether to continue, stop, or report. This loop is what converts raw responses into confirmed findings or rejecte","core_discovery":"The paper's central claim is that a security finding in an MCP tool should be defined by observable evidence, not by suspicious semantics. FlowGuard operationalizes this with the rule R = S ∧ O ∧ ¬E: a real finding requires a signal that is identified, is system-originated rather than reflected from the probe, and is not expected behavior for the tool. The system combines semantic triage of tool metadata with a two-phase probing strategy—recon to fingerprint the backend using low-impact probes, then strike with targeted payloads—and a history-guided refinement loop that repairs rejected probes. Evaluated on a benchmark that pairs positive and negative cases with explicit evidence, it reaches","pith_inferences":["The adjudication rule is a transferable recipe: any agent-tool protocol that exposes metadata and responses could use 'system-originated and unexpected' as a general finding criterion, not just MCP.","The benchmark's positive/negative pairing with explicit evidence is a reusable template for evaluating security scanners independently of the downstream agent model.","The reported over-conservative triage for command injection suggests a future variant that retains uncertain parameters in a lightweight probing path to recover missed recall.","If the ground-truth labeling concern is resolved, the same benchmark could serve as a community standard for comparing MCP scanners across labs."],"forward_implications":["Execution-related risks like command injection and file-system access can be confirmed from runtime evidence, closing the gap left by static and metadata-only scanners.","Credential leakage detection can stop treating placeholder-like strings as real leaks, reducing false positives while improving recall.","Semantic risks such as tool poisoning and prompt injection can be reported separately from execution findings, without asserting that a downstream agent will follow the injected instruction.","The recon-and-strike design narrows the probe space, cutting end-to-end latency by up to 2.23× compared with broader dynamic scanning while requiring fewer probes per true positive.","Bounded probing with termination rules makes the approach suitable for offline scanning and CI/CD integration."],"fun_headline_variants":["MCP security: prove it, don't pattern-match it","FlowGuard catches MCP attacks with runtime proof","Evidence-based MCP detection reduces false alarms","MCP risks: see the proof, not just the signal","Runtime evidence exposes real MCP threats"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The benchmark's ground-truth labels are defined using the same evidence categories that FlowGuard's adjudicator emits (R = S ∧ O ∧ ¬E), so the high F1 scores may partly reflect the scanner being evaluated against its own decision rule rather than an independent ground truth.","fun_headline_variants_meta":{"raw":{"variants":["MCP security: prove it, don't pattern-match it","FlowGuard catches MCP attacks with runtime proof","Evidence-based MCP detection reduces false alarms","MCP risks: see the proof, not just the signal","Runtime evidence exposes real MCP threats"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000223,"raw_usage":{"total_tokens":1298,"prompt_tokens":751,"completion_tokens":547,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":495,"completion_tokens_details":{"reasoning_tokens":473}},"tokens_in":495,"tokens_out":547,"duration_ms":5511,"temperature":1.0,"reasoning_tokens":473,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T01:07:40.915411+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-label the 1,880 benchmark cases by inspecting backend source code or having independent human analysts decide vulnerability status, without reference to FlowGuard's adjudication rule, then rerun FlowGuard and measure F1; if the scores drop substantially below 0.879 and 0.942, the performance claim is partly an artifact of circular labeling. A simpler check: any server that returns the probe string verbatim should never be counted as a finding—if a scan reports it as credential leakage, that is a false positive by the paper's own definition.","supporting_citations":[],"review_version":1}