{"id":"45beac0e-f020-4a09-abdc-3dc1fac06d5b","arxiv_id":"2506.05431","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A multi-agent RL black-box attack that localizes sensitive frames and patches in videos achieves lower L1 perturbation and query counts than prior video attacks on HMDB-51 and UCF-101.","lead":"This paper trains two reinforcement learning agents to find which frames and which spots within those frames to alter when attacking video recognition models. The attack uses fewer queries and smaller pixel changes than previous black-box methods, which could make robustness testing of video classifiers cheaper and more practical.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA comparison rests on an unverified protocol match: baseline subsets and query counts (including reverse-distortion queries) are unspecified, so the reported MAP/QN advantage may be an artifact.","rationale":"The reader's weakest assumption identifies the same core concern, and my stress test sharpens it: the reverse-distortion queries are a concrete, named stage whose query cost is not accounted for in the reported QN. This is not a mere methodological nit; the reported QN values are the central evidence for the abstract's claim. The paper gives no error bars, no seeds, and no code, so the numbers cannot be independently checked. However, there is no internal contradiction that would force rejection; the method is plausible, and the ablations are directionally consistent with the approach. Thus the appropriate verdict remains CONDITIONAL, with release of code and a matched-protocol rerun as the condition. I do not see a reason to move to REJECT or to accept the empirical claims as they stand.","tokens_in":11224,"tokens_out":3687,"duration_ms":45162,"concrete_test":"Release code and rerun AstFocus (and at least VBAD and Sparse) on the exact same correctly classified validation subset with the same 10,000-query cap; instrument every victim-model call, including those inside reverse distortion removal (Algorithm 1, line 17), and report paired per-video QN and MAP with standard errors. If the counted QN for the proposed method increases by more than 10% or any baseline's QN/MAP changes by more than 10% under the matched protocol, the abstract's superiority claim should be weakened or withdrawn.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the method outperforms SOTA on MAP and average query number (abstract; Table 1). For this to be true, the baselines and the proposed method must be evaluated under the same protocol. Two protocol elements are unstated. (1) Query accounting: Section 5.3 and Algorithm 1 line 17 describe a reverse distortion removal stage that queries the victim to check whether misclassification is retained. Section 6.3 defines QN as 'the number of query attempts needed to successfully fool' with a 10,000 cap, but never states whether reverse-removal queries are counted. If they are excluded, the reported QN undercounts the true query cost by an unknown amount, and no baseline has a comparable uncounted phase. (2) Dataset subset: Section 6.1 says only correctly classified samples are used, but the paper does not say that the cited baseline numbers were produced on the same split/subset or with the same cap. Since MAP and QN are averages over the test set, different subsets or caps change them. In addition, comparing GB/DP/GN distortion MAP against baselines that optimize different perturbation types is only meaningful under a matched budget and metric. Without code or rerun baselines, the headline outperformance over state-of-the-art is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a multi-agent reinforcement learning black-box attack for video action recognition. A temporal agent selects a sparse set of frames and a hierarchical spatial agent localizes a patch within each selected frame; both are trained with PPO using a shared victim-feedback reward plus auxiliary rewards, and the attack supports custom distortion types (Gaussian blur, dead pixels, Gaussian noise). A reverse-distortion stage then removes perturbations that are not needed for misclassification. The authors report experiments on HMDB-51 and UCF-101 against C3D, TSN, TSM, and SlowFast models, claiming state-of-the-art results in mean absolute perturbation (MAP) and average query number (QN).","tokens_in":11519,"tokens_out":5065,"duration_ms":62884,"significance":"If the reported comparison is trustworthy, the method would be a useful contribution to query-efficient robustness evaluation for video models, with strengths including evaluation on four victim models and two datasets, an ablation of patch size, and an ablation of the two agents. The paper also makes a falsifiable claim: lower MAP and lower QN than prior black-box video attacks under the same protocol. However, the central comparison is currently not established because the evaluation protocols for cited baselines are not shown to match, the reported numbers lack statistical grounding, and the novelty relative to prior multi-agent RL attacks (particularly AstFocus and the authors' own earlier works) is not clearly delineated. The significance is therefore conditional on a careful re-evaluation, not on the current evidence.","major_comments":[{"comment":"The headline claim that the method outperforms state-of-the-art on MAP and average QN is not supported by a matched protocol. Section 6.1 states that only correctly classified samples are used, but the paper does not state that the baseline numbers copied into Table 1 were produced on the same subset, under the same 10,000-query cap, or with the same query-accounting rule. This matters because Algorithm 1, line 17 performs a reverse distortion-removal stage that queries the victim model, and Section 6.3 defines QN as 'the number of query attempts needed to successfully fool' without stating whether reverse-removal queries are counted. If they are excluded, the reported QN undercounts the true cost, and no baseline has a comparable uncounted phase. The cap inconsistency in Table 1 reinforces the concern: several Heuristic-attack QN entries exceed 10,000 even though Section 6.3 states a 10,000 upper limit. The authors should rerun all baselines under the identical protocol, or explicitly report how each cited number was obtained, including the exact test subset, the cap, and the treatment of reverse-distortion queries.","section":"§6.3, §6.4, Table 1"},{"comment":"No statistical evidence is provided for any of the reported improvements. There are no error bars, no multiple seeds, and no significance tests, although the method is a stochastic RL procedure. Some differences in Table 1 are large, but others are not; the ablation tables report single numbers (e.g., Table 2 MAP values such as 0.39 vs 1.22, and Table 3 MAP 0.975 vs 1.021) without variance across training runs or test subsets. The authors should report mean and standard deviation over at least three independent training/evaluation runs for the proposed method and for any rerun baselines, and state the number of test videos used in each average.","section":"§6.4, Table 1, Table 2, Table 3"},{"comment":"The novelty of the proposed architecture relative to prior multi-agent RL attacks is not clearly established. Section 2.2 describes AstFocus [29] as a multi-agent RL setup with frame and patch selection and a shared reward, which is structurally similar to the proposed spatial/temporal agent design. The paper does not provide a controlled comparison with AstFocus under identical conditions, nor does it explain why replacing PGD/NES with PPO and adding a hierarchical localization is sufficient for the large reported gains. The issue is compounded by the paper's own statements that the level-1 localization is 'similar to the method proposed in [18]' (§5.2.1), the edgebox reward comes from [34] (§5.2.3), and the representativeness reward is 'inspired by [37]' (§5.1.2). The authors should either provide a direct ablation isolating each novel component or temper the contribution claim.","section":"§5.2.1, §5.2.3, §2.2"},{"comment":"The text in Section 6.4 overstates the results in its own table. It says that the query values of the proposed method are 'significantly lesser than the black-box attacks,' but Table 1 contains counterexamples: for C3D on HMDB-51, Ours (GB) has QN 8710 while VBAD has QN 4107; for C3D on UCF-101, Ours (DP) has MAP 6.351 while AstFocus has MAP 4.015. The abstract's universal-superiority claim should be restricted to the configurations and metric combinations where it actually holds, or the reported numbers should be corrected.","section":"§6.4, Table 1"},{"comment":"Several reward definitions are ambiguous or under-specified as printed. Equation (1) as typeset is not fully legible and the role of the threshold L is unclear (whether the reward encourages exactly L frames or at most L frames). Equation (3) uses a function L for the victim-confidence difference but L is already used as the frame threshold in Eq. (1), and the 'loss function' is not defined. These ambiguities make it difficult to reproduce the method exactly.","section":"§5.1.2, §5.2.3, Eq. (1)-(3), (7)-(8)"}],"minor_comments":[{"comment":"The abstract says 'Lp metric' but the paper evaluates only p = 1 (MAP); please state this explicitly in the abstract or use 'L1/MAP' throughout.","section":"Abstract and §6.3"},{"comment":"There are many typos and formatting artifacts, including 'hirararchical', 'actor-ciritc', 'the the', and garbled equation renderings. A thorough proofread is needed.","section":"Throughout"},{"comment":"The caption lists 'SVAL(2022) [28]' and 'Sparse (2022)[28]' as if they are different methods, but both point to reference [28]; also SVAL appears only for C3D in the table. Please clarify the identity and source of each baseline.","section":"Table 1 caption"},{"comment":"The dataset split is described as 70% training / 30% validation, but action recognition benchmarks normally use fixed official splits. Please specify which split is used and report the number of correctly classified test videos per model/dataset, since the averages depend on that subset.","section":"§6.1"},{"comment":"The difference images in Figure 6 have no scale or normalization information, so a reader cannot judge the visual imperceptibility claim quantitatively.","section":"Figure 6"},{"comment":"The limitations subsection is only a brief note about computational cost vs. performance; it should also acknowledge the protocol limitations discussed in the major comments, especially the lack of matched baselines and statistical error bars.","section":"§6.4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper has a large number of self-citations (refs [14]-[21]) to the authors' prior workshop and conference works, and the main novelty appears to be an incremental combination of those components. I would ask the editor to require the authors to (1) rerun or transparently re-derive all baseline numbers under a matched protocol, (2) provide multi-seed statistics, and (3) clearly state the delta over their prior work and over AstFocus. These are fixable within the scope of a revision, but without them the central claim is not verifiable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a competent engineering extension of your group's RL attack work to video, and the reverse-distortion step is a nice practical trick. The headline claim—'outperforms SOTA on Lp and queries'—is plausible but not yet established, because the comparison against literature baselines doesn't show a matched protocol, and the paper never says whether reverse-removal queries are counted in QN.\n\nWhat's genuinely useful: the framing of robustness evaluation as a cooperative two-agent problem (spatial + temporal) is clean, and the custom distortion types (Gaussian blur, dead pixels, Gaussian noise) make the attack portable to real-world distortions. The hierarchical two-level localization is a sensible way to cut search space, and the ablation in Table 3 showing the combined agents beat either alone is a legitimate check. Four victim models and two datasets is a decent spread for a systems paper.\n\nSoft spots, in order of severity. First, the SOTA comparison. Table 1 takes baseline numbers from other papers, but there's no statement that those baselines were run on the same correctly-classified subset, the same 10,000 query cap, or the same definition of a query. The queried numbers (e.g., QN under 1,000 vs. AstFocus at 1,500–4,000) are good enough that the difference could be real, but with no matched protocol it's unverifiable. This is the main barrier to taking the headline seriously. Second, query accounting for the reverse distortion removal (Algorithm 1, line 17). If the reversion loop queries the victim and those queries aren't counted, QN is understated in a way no baseline has an equivalent for. The paper should state this explicitly. Third, no error bars or multiple seeds. RL training is stochastic; a single run per condition doesn't support the 'significantly better' language in the text. Fourth, the novelty is incremental—most components are from your own prior image-attack papers—but the combination for video with a temporal agent is new enough to be a contribution.\n\nThe citation pattern is mostly self-citations to your own prior work. That's defensible because the method is built on it, but it also means an independent reader can't easily assess how much is genuinely new without digging.\n\nThis paper is for people building robustness benchmarks and black-box attack toolkits. They'd get value from the custom-distortion framing and the reversion trick. It's not a breakthrough, but it's a solid systems contribution if the protocol issues are fixed. I'd like to see it go to peer review, with the clear expectation that the authors provide code, run the baselines themselves, and clarify the query definition.","headline":"Incremental RL attack on video models with a plausible but unverified SOTA claim due to non-matched baselines and unclear query accounting; worth reviewing if protocols get fixed.","tokens_in":12054,"tokens_out":2502,"would_cite":false,"duration_ms":31465,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A multi-agent reinforcement-learning attack that picks sensitive frames and patches evaluates video-model robustness with smaller distortion and fewer queries than prior black-box attacks.","keywords":["adversarial attack","black-box attack","video action recognition","reinforcement learning","multi-agent reinforcement learning","robustness evaluation","query efficiency","spatiotemporal localization"],"falsifier":"Rerun the strongest baseline attack and the proposed attack on the same correctly classified test videos with the same query cap, and count the queries spent in the reverse distortion-removal stage. If the baseline's mean absolute perturbation approaches the proposed method's values (for example, AstFocus on TSM over UCF-101 moving from 3.411 toward 0.835) or its average query count drops once protocols are matched, the reported superiority is a protocol artifact rather than a property of the attack.","tokens_in":11061,"feed_emoji":"🎬","tokens_out":10009,"duration_ms":105965,"temperature":0.7,"pith_summary":"Video models are harder to audit than image models because their extra time dimension makes black-box adversarial attacks expensive in both queries and visible distortion. This paper tries to make such audits cheap by having two reinforcement-learning agents cooperate: one picks the frames that matter, one picks the patch within each frame, and both are rewarded by how much the victim model's confidence drops. The authors report that the resulting attack induces misclassification with smaller mean absolute perturbation and far fewer queries than existing black-box video attacks on four standard video recognizers and two datasets, while staying competitive in success rate. A final reverse step strips away perturbations that no longer contribute, trimming the distortion further. If the claim holds, robustness evaluation of video models becomes more practical and can use distortions tailored to a deployment scenario.","feed_headline":"Two agents find a video model's weak spots with fewer queries","feed_subtitle":"They pick sensitive frames and patches together, cutting distortion and query counts in robustness checks.","key_machinery":"The central mechanism is a multi-agent PPO formulation with a hierarchical spatial policy and a recurrent temporal policy. The temporal policy is an LSTM that consumes per-frame CNN features and global video features and binary-decides inclusion of each frame. The spatial policy picks one of $D$ non-overlapping patches at level 1, then subdivides that patch and picks a finer patch at level 2, concentrating perturbation on a small region. The shared reward ties both agents to the victim's confidence drop: the temporal agent receives $r_3 = \\exp(\\mathcal{L}(V(x_{t+1}) - V(x_t)))$ and the spatial agent receives $r_3 = \\exp(P(y|x') - P(y|x))$, with additional rewards for frame sparsity, representativeness, objectness, and motion saliency. A reverse distortion-removal step iteratively prunes low-impact perturbations after misclassification, which is what holds the $L_1$ norm down.","core_discovery":"The paper proposes a black-box adversarial attack for video action recognition in which two reinforcement-learning policies cooperate. A temporal policy, built from an LSTM over features extracted from frames, decides for each frame whether to include it for perturbation; a spatial policy first localizes a large patch in the frame (level-1 localization), then subdivides that patch to choose a finer region (level-2 localization). Both policies are trained with Proximal Policy Optimization under a shared reward derived from the victim model's confidence change, plus task-specific rewards: the temporal agent is encouraged to select few yet representative frames, and the spatial agent is encouraged to pick patches that overlap foreground objects and motion-salient regions. After a successful misclassification, the method reverses distortions that no longer matter, so the reported $L_1$ perturbation is lower than the perturbation that actually achieved the attack. The paper reports that this combination beats prior black-box video attacks on mean absolute perturbation and average query number for C3D, TSN, TSM, and SlowFast on HMDB-51 and UCF-101, with success rates that remain competitive.","pith_inferences":["A testable accounting extension is to report the query cost of the reverse distortion-removal step separately; the paper's query counts do not state whether reversion queries are included, and that accounting determines how much of the reported query reduction belongs to the attack itself.","Because the policies identify sparse frames and patches rather than memorizing a classifier, the same multi-agent setup could be pointed at video transformers or spatiotemporal action-localization models, which the paper does not evaluate.","The combined agents beat the best single agent by roughly 21% in MAP in the ablation, so a natural next test is whether a single policy with a factored action space can recover the same gain with less training overhead.","The custom distortion mechanism suggests a broader use: evaluating robustness against compression artifacts or weather corruptions as an alternative to pixel-space $\\ell_p$ perturbations."],"forward_implications":["Robustness audits of video models could require only hundreds of queries instead of thousands, with average $L_1$ distortion below 1 on several tested model-dataset pairs.","The same framework can probe robustness under custom distortion families, such as Gaussian blur, dead pixels, or Gaussian noise, chosen to match a deployment scenario.","Experiments across C3D, TSN, TSM, and SlowFast on HMDB-51 and UCF-101 report stable success across all three distortion types, suggesting the attack is not tied to one perturbation model.","The reverse distortion-removal step lowers the net $L_1$ distortion after a successful attack, so the final perturbation is smaller than the perturbation that first caused misclassification."],"supporting_citations":[{"why":"Supplies the Proximal Policy Optimization algorithm used to train both the spatial and temporal policies.","marker":"[22]"},{"why":"Provides the hierarchical patch-localization and RL-based black-box attack ideas that the spatial agent extends to video.","marker":"[18]"},{"why":"Provides the platform for RL-based black-box attacks with custom distortion filters that the method builds on.","marker":"[21]"},{"why":"AstFocus is the closest state-of-the-art competitor; its reported MAP and query numbers are the main comparison in Table 1.","marker":"[29]"},{"why":"VBAD is a black-box video attack baseline whose Table 1 results the method claims to improve on.","marker":"[8]"},{"why":"Sparse is a sparse black-box video attack baseline whose reported results are used in the comparison.","marker":"[28]"},{"why":"M-S (Motion-Excited Sampler) is a baseline that incorporates motion priors; the paper compares against its Table 1 numbers.","marker":"[36]"},{"why":"RLSB is an RL-based sparse video attack baseline whose results are compared in Table 1.","marker":"[26]"},{"why":"Heuristic is a black-box video attack baseline whose results are compared in Table 1.","marker":"[30]"},{"why":"G-T (geometric-transform attack) is a query-based baseline whose results are compared in Table 1.","marker":"[11]"}],"fun_headline_variants":["Multi-agent attack finds video model blind spots with fewer queries","Spatial-temporal agents cooperate to fool video models efficiently","RL agents team up to expose video model vulnerabilities cheaply","Two RL agents cooperate to break video models with fewer queries"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the comparison is fair: the baseline numbers in Table 1 came from the same subset of correctly classified videos, the same 10,000-query cap, and the same counting of queries spent during reverse distortion removal, since the paper does not say the baselines were rerun under its protocol.","fun_headline_variants_meta":{"raw":{"variants":["Multi-agent attack finds video model blind spots with fewer queries","Spatial-temporal agents cooperate to fool video models efficiently","RL agents team up to expose video model vulnerabilities cheaply","Two RL agents cooperate to break video models with fewer queries"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00078,"raw_usage":{"total_tokens":3426,"prompt_tokens":905,"completion_tokens":2521,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":521,"completion_tokens_details":{"reasoning_tokens":2454}},"tokens_in":521,"tokens_out":2521,"duration_ms":21283,"temperature":1.0,"reasoning_tokens":2454,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:33:46.130518+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the strongest baseline attack and the proposed attack on the same correctly classified test videos with the same query cap, and count the queries spent in the reverse distortion-removal stage. If the baseline's mean absolute perturbation approaches the proposed method's values (for example, AstFocus on TSM over UCF-101 moving from 3.411 toward 0.835) or its average query count drops once protocols are matched, the reported superiority is a protocol artifact rather than a property of the attack.","supporting_citations":[{"cited_title":"Robustness with black-box adversarial attack using reinforce- ment learning","cited_arxiv_id":null,"evidence_quote":"Provides the hierarchical patch-localization and RL-based black-box attack ideas that the spatial agent extends to video."},{"cited_title":"Reinforcement learning platform for adversarial black - box attacks with custom distortion filters","cited_arxiv_id":null,"evidence_quote":"Provides the platform for RL-based black-box attacks with custom distortion filters that the method builds on."},{"cited_title":"Effi- cient robustness assessment via adversarial spatial -temporal focus on videos","cited_arxiv_id":null,"evidence_quote":"AstFocus is the closest state-of-the-art competitor; its reported MAP and query numbers are the main comparison in Table 1."},{"cited_title":"Black-box adversarial attacks on video recog- nition models","cited_arxiv_id":null,"evidence_quote":"VBAD is a black-box video attack baseline whose Table 1 results the method claims to improve on."},{"cited_title":"Sparse black - box video attack with reinforcement learning","cited_arxiv_id":null,"evidence_quote":"Sparse is a sparse black-box video attack baseline whose reported results are used in the comparison."},{"cited_title":"Motion- excited sampler: Video adversarial attack with sparked prior","cited_arxiv_id":null,"evidence_quote":"M-S (Motion-Excited Sampler) is a baseline that incorporates motion priors; the paper compares against its Table 1 numbers."},{"cited_title":"Reinforcement Learning Based Sparse Black-box Adversarial Attack on Video Recognition Models","cited_arxiv_id":"2108.13872","evidence_quote":"RLSB is an RL-based sparse video attack baseline whose results are compared in Table 1."},{"cited_title":"Heuristic black-box adversarial attacks on video recognition models","cited_arxiv_id":null,"evidence_quote":"Heuristic is a black-box video attack baseline whose results are compared in Table 1."},{"cited_title":"Adversarial attacks on black box video classifiers: Leveraging the power of geometric transformations","cited_arxiv_id":null,"evidence_quote":"G-T (geometric-transform attack) is a query-based baseline whose results are compared in Table 1."}],"review_version":1}