{"id":"beb382f8-31f7-4e43-b21a-4224a3630f05","arxiv_id":"2607.11472","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Evolved attack programs cut a composite success–query–distortion score by 8–41% versus best optimized seeds on pHash, PDQ, PhotoDNA, and NeuralHash under a graded black-box oracle.","lead":"LLM-driven program evolution finds stronger black-box collision attacks on four deployed perceptual hashes than tuned classical attacks, on a 30-pair ImageNet benchmark. The work matters because those hashes underpin large-scale content moderation and copyright systems that currently lack strong adversarial guarantees.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.5","headline":"Score gains from LLM program evolution may be in-sample artifacts of searching and ranking on the same fixed 30 ImageNet pairs used for final reporting.","rationale":"The reader’s weakest_assumption correctly isolates the single load-bearing premise of the strongest claim: that in-sample S reductions on the shared 30-pair split constitute meaningful evidence of attack superiority. The paper is transparent about the limitation in later sections, which is why CONDITIONAL (not REJECT) remains appropriate; no internal contradiction, fabricated derivation, or oracle misuse appears. A held-out re-evaluation would settle whether the claim survives. No stronger technical concern displaces this one, so the verdict stays CONDITIONAL and agreement is full.","tokens_in":11681,"tokens_out":507,"duration_ms":23397,"concrete_test":"Hold out a fresh set of 30 ImageNet pairs never seen during optimization or evolution; re-execute the best evolved programs and best optimized seeds from Tables 6–8 under the same per-hash query budgets, recompute ASR/L2/Q/S with the external evaluator. If the relative S improvement falls below ~15% or ASR gaps reverse on the held-out set, the superiority claim is an in-sample artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of composite-score reductions (34.0% pHash, 38.3% PDQ, 8.1% PhotoDNA, 41.2% NeuralHash) rests on Tables 7–8, which report programs evolved and selected on the identical 30-pair ImageNet split used for hyperparameter optimization, evolution, and final metrics (§4, §5.3, §6.1). ASR (Eq. 2), L2, Q and therefore S (Eq. 3) are all computed on these pairs, so the selection objective can reward programs that overfit the specific images and hash-distance landscapes of this tiny fixed set rather than discovering generally superior algorithms. The paper itself scopes the result as “controlled benchmark evidence” (Limitations, Conclusion) and notes single-seed evolution plus non-compute-matched baselines, yet the abstract and introduction present the reductions as demonstrating attack superiority and previously unreported vulnerabilities. Without held-out pairs the reported algorithmic changes (Table 9) cannot be distinguished from in-sample selection artifacts.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper proposes an LLM-guided program-evolution framework (GigaEvo/OpenEvolve) for black-box targeted second-image attacks on perceptual hash algorithms. Attacks are ranked by a composite score S = (L2 + 0.02 Q) / max(ASR, 0.05) under a graded distance oracle and a strict external evaluator. On four deployed PHAs (pHash, PDQ, PhotoDNA, NeuralHash) and a fixed set of 30 ImageNet pairs, the best evolved programs reduce S relative to the best optimized seed by 34.0%, 38.3%, 8.1%, and 41.2% respectively (Tables 7–8), often with competitive ASR and lower L2 and/or query cost. The authors argue that program-level search is better suited than gradient methods to non-differentiable, quantized hash outputs and that the results motivate stronger robustness evaluation of content-moderation hashes.","tokens_in":11993,"tokens_out":961,"duration_ms":14952,"significance":"If the score reductions reflect genuine algorithmic improvements rather than in-sample selection, the work is a useful contribution to adversarial evaluation of PHAs: it is fully black-box, handles discrete hash outputs without gradients, and systematically compares hand-written seeds, Optuna-tuned configurations, and LLM-mutated attack programs under a shared external evaluator. The seed portfolio (SimBA, NES, ZO-SignSGD, Prokos, ATKScopes), the two-phase configuration protocol, and the program-level qualitative changes in Table 9 are concrete and reproducible in spirit. Novelty relative to prior hash attacks (Prokos, ATKScopes) and to LLM evolution (AlphaEvolve/OpenEvolve/GigaEvo) is real for this application. The significance is currently limited by the tiny fixed benchmark and single-run evolution design, which the paper itself scopes as controlled benchmark evidence rather than deployment-scale proof of vulnerability.","major_comments":[{"comment":"§4, §5.3, §6.1 and Tables 7–8: the same 30 ImageNet pairs are used for hyperparameter optimization, program evolution, and final reporting. ASR (Eq. 2), L2, Q and therefore S (Eq. 3) are all computed on this fixed set, so reported score reductions (34.0%/38.3%/8.1%/41.2%) cannot be distinguished from in-sample selection of programs that overfit these particular images and hash-distance landscapes. The Limitations and Conclusion correctly call this “controlled benchmark evidence,” but the Abstract and Introduction present the reductions as demonstrating attack superiority and “previously unreported vulnerabilities.” A held-out pair set (or at least a train/report split with frozen programs) is load-bearing for the central comparative claim.","section":null},{"comment":"§5.3: evolution results come from a single run with a single random seed per hash per backend (~1000 candidates), with no cross-seed variance, confidence intervals, or paired significance. Given stochastic LLM mutation and MAP-Elites/island search, a single champion trajectory (Fig. 2) is insufficient to support stable percentage improvements in Table 8. At minimum, multi-seed evolution (or bootstrap over pairs) and reporting of score dispersion are needed before treating the Table 8 deltas as reliable.","section":null},{"comment":"§5.3 and Table 8: the comparison of evolved programs to optimized seeds is explicitly not compute-matched (“seed-configuration stage and the evolution stage were not run under a matched total-evaluation budget”). Because evolution evaluates ~1000 full attack programs per hash while Stage 2 only tunes scalar hyperparameters of fixed algorithms, part of the S reduction may be extra search budget rather than the benefit of program-level mutation. A compute-matched seed search (or budget-normalized comparison) is required to isolate the claimed advantage of LLM program evolution.","section":null},{"comment":"§5.4 and Table 6 (NeuralHash row): only ATKScopes receives full configuration optimization for NeuralHash; the other four seeds remain at original configurations with ASR ≤ 0.13. The 41.2% improvement in Table 8 is therefore against a partially optimized portfolio, not a fully optimized five-seed baseline. This asymmetry should be reflected in the Abstract’s NeuralHash claim and, ideally, remedied by optimizing at least the next-best viable seeds or by clearly restricting the NeuralHash claim to “vs. optimized ATKScopes.”","section":null}],"minor_comments":[],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The punchline is simple: they take known black-box seeds (SimBA, NES, ZO-SignSGD, Prokos, ATKScopes), tune them per hash, then let GigaEvo/OpenEvolve mutate the actual Python attack programs under a strict external evaluator that only sees graded Hamming/L1 distance. On four deployed hashes the best evolved programs beat the best optimized seeds on their composite S by 34–41% for pHash/PDQ/NeuralHash and a more modest 8% for PhotoDNA, often with lower L2 and fewer queries. That is new; nobody had run quality-diversity LLM code evolution against PHAs before, and the external-evaluator contract is cleanly designed so generated code cannot game bookkeeping.\n\nWhat they do well is the engineering discipline. Thresholds τ0 are fixed operating points, not fitted. The oracle returns only distance and enforces budget. Success, L2, LPIPS and S are recomputed outside the candidate. Table 9 shows the evolved changes are algorithmic (DCT search bases, post-success refinement, plateau handling), not just hyperparameter noise. The conclusion and limitations are more careful than the abstract: they call the numbers “controlled benchmark evidence” and list the missing held-out split, multi-seed variance, and compute-matched seed search.\n\nThe soft spots are exactly the ones the stress-test flags, and they are real but already scoped. Everything—tuning, evolution, reporting—runs on the same 30 ImageNet pairs. Single evolution seed per hash/backend, no error bars. NeuralHash only fully tunes ATKScopes because the other seeds were dead on arrival. LPIPS stays high for PhotoDNA and NeuralHash, so these are not imperceptible collisions. None of that invents a circularity or breaks the math; S is an explicit ranking objective they defined. It just means the abstract’s “previously unreported vulnerabilities” claim is stronger than the evidence supports until someone re-runs on held-out pairs.\n\nThis is for people who audit content-moderation hashes or who care about LLM-driven attack search. It is not a theory paper and does not claim to be. I would send it to peer review: the method is reproducible in principle, the tables are clear, and the caveats are already written down. Engage if you work on PHA robustness or black-box evolution; otherwise skim the tables and move on.","headline":"Solid first application of LLM program evolution to black-box PHA second-image attacks; the score gains are real on their fixed evaluator but rest on a 30-pair in-sample design the authors themselves flag.","tokens_in":12650,"tokens_out":585,"would_cite":true,"duration_ms":6778,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"LLM-evolved attack programs beat hand-tuned black-box methods at forging perceptual-hash collisions with fewer queries and less distortion.","keywords":["perceptual hash algorithms","black-box attacks","targeted collision","program evolution","LLM-guided search","content moderation","adversarial robustness"],"falsifier":"Rerun the entire evolution pipeline on a held-out set of image pairs never seen during hyperparameter search or program evolution; if the evolved programs no longer beat the best optimized seeds on the same composite score, the central claim fails.","tokens_in":12556,"feed_emoji":"🔓","tokens_out":657,"duration_ms":6414,"temperature":0.7,"pith_summary":"Perceptual hashes are meant to stay stable under ordinary image edits while still distinguishing different content, so platforms use them to flag known harmful or copyrighted images. This paper asks whether an attacker who can only see the hash distance to a target can still force a source image to collide with that target. The authors show that evolving the attack programs themselves—with language models mutating Python code under a fixed external score that balances success rate, query count, and pixel distortion—finds better collisions than tuning the parameters of existing black-box attacks. On thirty ImageNet pairs and four widely deployed hashes, the best evolved programs cut the composite score by roughly 8–41 percent relative to the best optimized seeds, often with lower L2 distortion and fewer oracle calls. The work therefore treats program search as a practical audit tool for content-moderation hashes that lack formal robustness guarantees.","feed_headline":"Evolved code forges hash collisions cheaper than tuned attacks","feed_subtitle":"On four deployed perceptual hashes, program search cuts the attack score 8–41% versus optimized seeds.","key_machinery":"The composite score S together with a strict external evaluator: candidate programs may only call a black-box oracle that returns hash distance and enforces a query budget; all success, L2, LPIPS, and S values are recomputed outside the candidate so that bookkeeping cannot inflate results. LLM mutation (via two evolution backends) then searches over full attack programs rather than scalar hyperparameters alone.","core_discovery":"On a controlled 30-pair ImageNet benchmark under a graded black-box distance oracle, LLM-guided evolution of attack programs reduces the composite score S = (L2 + 0.02 Q) / max(ASR, 0.05) relative to the best optimized seed attack by 34.0 percent for pHash, 38.3 percent for PDQ, 8.1 percent for PhotoDNA, and 41.2 percent for NeuralHash, while matching or improving attack success rate and frequently lowering visual distortion or query cost.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["LLM-evolved programs cut perceptual-hash attack scores 8–41%","Program evolution forges PHA collisions with fewer queries","Evolved black-box code beats tuned seeds on four PHAs","GigaEvo programs lower hash-attack cost and distortion","LLM-guided evolution weakens pHash, PDQ, PhotoDNA, NeuralHash"],"cache_read_input_tokens":128,"weakest_assumption_plain":"Score gains found by searching and ranking on the same fixed thirty image pairs used for tuning, evolution, and final reporting are treated as meaningful evidence of stronger attacks rather than in-sample selection artifacts.","fun_headline_variants_meta":{"raw":{"variants":["LLM-evolved programs cut perceptual-hash attack scores 8–41%","Program evolution forges PHA collisions with fewer queries","Evolved black-box code beats tuned seeds on four PHAs","GigaEvo programs lower hash-attack cost and distortion","LLM-guided evolution weakens pHash, PDQ, PhotoDNA, NeuralHash"]},"model":"grok-4.5","effort":"low","cost_usd":0.00348,"raw_usage":{"total_tokens":1192,"prompt_tokens":870,"num_sources_used":0,"completion_tokens":95,"cost_in_usd_ticks":34800000,"prompt_tokens_details":{"text_tokens":870,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":227,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":870,"tokens_out":95,"duration_ms":3378,"temperature":1.0,"reasoning_tokens":227,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-14T05:21:26.519277+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Rerun the entire evolution pipeline on a held-out set of image pairs never seen during hyperparameter search or program evolution; if the evolved programs no longer beat the best optimized seeds on the same composite score, the central claim fails.","supporting_citations":[],"review_version":1}