REVIEW 5 major objections 5 minor 36 references
A new cross-platform evaluation system shows that continuous camera motion defeats every published adversarial vehicle camouflage.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-01 19:04 UTC pith:VTOHQR2A
load-bearing objection A genuinely useful adversarial-attack evaluation system that deserves referee time, but the headline empirical claims are softer than the abstract suggests once you look at the stats and the unquantified Mitsuba-to-UE5 renderer gap. the 5 major comments →
ALLUDE: A Unified Evaluation System for Configurable Attacks in Differentiable Environments
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
On the paper's own terms, the central discovery is that camera motion is a first-order threat to physical adversarial attacks. In the configuration-space analysis, camera trajectory is the second-strongest predictor of attack success, explaining 10% of model fit (p=0.020), behind only target object class (22%, p=0.002); weather (4%), optimizer (2%), and detector (<1%) are not statistically significant. In the stress test, the ALLUDE-optimized texture and the strongest published texture (RAUCA) both achieve total weather-invariant suppression at a static elevated view, yet no texture — published or ours — achieves evasion across the moving flyover, tight-orbit, or wide-orbit trajectories, wit
What carries the argument
The system's core mechanism is a two-process architecture that isolates Unreal Engine 5's scene capture (parent process) from PyTorch/Mitsuba gradient computation (child process), allowing engine-faithful rendering on headless Linux and Windows. Textures are optimized through Mitsuba 3's differentiable renderer and then transferred to UE5, after a UV-unification step re-bakes published attack textures onto a common vehicle mesh so they can be compared fairly. The evaluation uses adv-mAP@50 as the metric, chosen because it remains meaningful when weather or trajectory degrades benign detection, and Latin Hypercube Sampling to make the 100 sampled configurations representative of the 5,400-cel
Load-bearing premise
The evaluation assumes that adversarial textures optimized in Mitsuba 3's renderer transfer faithfully to Unreal Engine 5's renderer, but the paper measures no such sim-to-sim transfer gap, so the reported rankings and trajectory effects could be artifacts of renderer mismatch rather than stable properties of the attacks.
What would settle it
Render the six Table 4 textures through both Mitsuba and Unreal Engine under identical conditions and compare with a physical test of the same textures on a real vehicle filmed with the same five trajectories; if a texture that fails under moving cameras in ALLUDE succeeds in the physical test, the sim-to-sim gap is the true cause of the reported trajectory degradation.
If this is right
- Prior static-viewpoint, postprocessed benchmarks are likely to overstate attack success; future robustness claims for physical attacks should include continuous camera trajectories.
- Attack difficulty is governed primarily by object class and viewpoint, not weather or detector choice, which suggests defensive research should focus on view-robust detection and silhouette-based features for hard objects like pedestrians.
- The open-source, cross-platform release with CARLA assets decoupled from legacy Unreal Engine versions makes this style of large-scale evaluation reproducible on common GPU clusters.
- The mAP@50 metric with a benign-detection baseline should become standard for simulation-based attack evaluation, since 14 of 100 sampled configurations were indeterminate due to environmental detection failure.
Where Pith is reading between the lines
- We infer that the same trajectory-sensitivity will appear for patch-based attacks on pedestrians and traffic signs, because those attacks rely on 2D texture insertion rather than full-UV 3D rendering; the per-object LHS results (pedestrian 10% suppression, fire hydrant 0%) already hint that silhouette-driven classes resist texture attacks regardless of trajectory.
- The weather-invariance of ALLUDE and RAUCA is measured within their training weather distribution; we infer that cross-weather transfer (e.g., a texture optimized at noon tested at night) is untested and likely weaker, since weather still contributed 4% of variance.
- We infer that the Mitsuba-to-Unreal sim-to-sim gap is the key unknown: a texture's trajectory failure in ALLUDE might vanish if the same texture were physically deployed and filmed, so a physical-world replication of the Table 4 experiment would settle whether the trajectory effect is a renderer artifact or a true property of the attacks.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces ALLUDE, a system that integrates Unreal Engine 5 (UE5) with Mitsuba 3, enabling end-to-end differentiable optimization of adversarial textures and subsequent evaluation in a photorealistic simulator with configurable scenes, weather, lighting, camera trajectories, detectors, and optimizers. It claims the first cross-platform (Linux/Windows) evaluation framework of its kind, and decouples CARLA assets from legacy UE versions. The authors demonstrate the system via (1) a Latin Hypercube Sampling study over 100 configurations drawn from 5,400, analyzed with logistic regression, reporting that object class and camera trajectory are the strongest factors in attack success, while weather, detector, and optimizer are negligible; and (2) a stress test of published attacks (CAMOU, RAUCA, FCA) plus an ALLUDE-optimized texture across 9 weather and 5 trajectory conditions, reporting weather-invariant suppression for ALLUDE/RAUCA and universal failure under moving trajectories. The paper emphasizes that existing post-hoc 'green-screen' evaluation pipelines overestimate attack robustness.
Significance. If the system works as described, it is a valuable resource: open-source, cross-platform, engine-faithful evaluation for adversarial attacks, with a modernized CARLA asset library and unusually detailed engineering documentation. The two-pronged evaluation attempts to characterize attack success over a large configuration space, which is a novel contribution and could serve as a benchmark for future work. The shipped code, data, and decoupled assets are concrete community assets. However, the empirical findings are currently under-supported due to methodological issues (unquantified renderer transfer gap, confounded scene–object factors, arbitrary exclusion of indeterminate cells, lack of repeated runs, and detector-specific transfer evaluation). The system contribution is strong; the empirical claims need substantial reinforcement before they can be accepted as general conclusions about attack robustness.
major comments (5)
- [Sec. S1.2 / Sec. 4.2 / Table 4] The central empirical claims — weather-invariant suppression and universal failure under moving trajectories — rest on an unquantified sim-to-sim transfer. The ALLUDE texture is optimized in Mitsuba 3 (Sec. S1.2) and evaluated in UE5 (Sec. 4.2, Table 4). No matched-frame comparison, no calibration against the green-screen pipeline (Fig. 2), and no physical-world sanity check are provided. The paper itself shows (Sec. 3.2) that Linux/Windows gamma handling changes detector input. Without quantifying the Mitsuba→UE5 distribution gap, the reported invariance/degradation could be artifacts of UE5's tone mapping, fog/rain rendering, or material response, rather than intrinsic attack properties. I request a paired-frame comparison under identical camera/weather and, ideally, a control texture optimized directly in UE5 via a differentiable surrogate; at minimum, reword the claims to be explicit
- [Sec. 4.1 / Table S10 / Table 3] The LHS treats 'scene–object pair' as a single 10-level factor; each object is tied to exactly one town/setting (Table S10). The 22% variance explained by 'target object' (Table 3) therefore conflates object class with scene layout, framing, and object placement. The discussion of texture surface area and shape priors (Sec. 4.1) is not supported by the design. Please disentangle by either varying objects inside shared scenes or adding scene as a separate factor; otherwise, rename the factor 'scene–object pair' and avoid causal attribution to object geometry/UV.
- [Sec. 4.1] Fourteen of 100 LHS cells are discarded as indeterminate (benign mAP@50=0), without reporting how these cells distribute across trajectory/weather/object. If, as seems likely, they concentrate in flyover/fog/small-object conditions, their removal biases the main-effect estimates and the 'trajectory is second-strongest' conclusion. Please report the indeterminate-cell distribution and re-run the factor analysis with different coding (e.g., indeterminate as attack failure, as success, or as a third outcome) to demonstrate robustness of the results to this exclusion rule.
- [Sec. 4.2 / Table 4] All reported mAP@50 values are single-run; no seeds, error bars, or confidence intervals are given. Similarly, each LHS cell is one attack run. With binary outcomes and N=86, the likelihood-ratio p-values (object p=0.002, trajectory p=0.020) and the qualitative statements in Sec. 4.2 ('always suppresses', 'weakens substantially') do not have quantified sampling uncertainty. Please provide multiple independent runs (at least 3–5) for representative cells and/or bootstrap confidence intervals, or explicitly frame the numbers as single-run demonstrations of system capability rather than precise attack-performance estimates.
- [Sec. 4.2 / Table 4] The published attacks (CAMOU, RAUCA, FCA) are evaluated only against DETR, after UV re-unification onto a common mesh (Sec. S2.2). These attacks were originally optimized for Mask R-CNN or YOLOv3, so the finding 'No attack, published or ours, achieves evasion across the moving-camera views' is specific to transferred textures under DETR. The abstract and conclusion generalize it to 'revealing degradation of attack success across every attack' and 'exposing evaluation gaps in prior work.' Please qualify the claim to state the detector-specific transfer setting, or evaluate the attacks against at least one native detector within ALLUDE. Otherwise, the conclusion that prior static-viewpoint benchmarks overstate robustness is broader than the evidence.
minor comments (5)
- [Sec. 4.2 vs Table S8] The perturbation budget is given as ϵ=255 in Sec. 4.2 but as ϵ=256 in Table S8. Please reconcile the exact value.
- [Sec. S2.3 / Sec. S1.6] Typos: 'psuedo-R2' should be 'pseudo-R2'; 'instructiuon' should be 'instruction'.
- [Sec. 3.2] The paper notes a Linux/Windows gamma-encoding difference but does not state which platform the reported experiments (LHS and stress tests) were run on or whether the sRGB-forcing docker configuration was used. Please include this information for reproducibility.
- [Sec. S2.1 / S2.2] The text says published textures are deployed 'unmodified,' but then describes a UV re-unification step that changes how the texture maps onto the vehicle. Clarify that the texture bitmap is unmodified while the UV mapping is adapted to a common topology; this distinction matters because UV distortion can itself affect attack transferability.
- [Table 1] ALLUDE receives full marks in all columns, but the comparison set is narrow (8 prior methods). Consider discussing whether other recent sim-to-real evaluation frameworks exist and how they compare, to justify the 'first-of-its-kind' claim more carefully.
Circularity Check
No significant circularity: ALLUDE is an empirical systems evaluation whose findings are not built into the system by construction.
full rationale
ALLUDE is a systems/evaluation paper, not a derivation, so the classic circularity modes do not apply. The central empirical claims — published attacks degrade under weather and continuous camera trajectories, camera trajectory is the second-strongest factor, and target object dominates LHS outcomes — are measured outputs of running the system, not consequences of how ALLUDE is defined or fitted. The self-citation of the authors' prior UNDREAM [22] is used only to credit earlier feasibility of in-simulation attack optimization; ALLUDE independently re-implements and demonstrates that capability (Sec. 3.2, Sec. 4), so the argument does not reduce to the citation. The ALLUDE texture's reported weather-invariance in Table 4 is a transfer result, not a fit: the texture is optimized once at a static viewpoint and clear noon, then applied unchanged across conditions, and the degradation of published attacks is measured against external baselines with a documented external anchor (RAUCA > CAMOU consistent with [36]). The unquantified Mitsuba-to-UE5 renderer gap is a validity/correctness threat, not circularity, because the paper's conclusions are not defined in terms of that gap.
Axiom & Free-Parameter Ledger
free parameters (7)
- ε (L∞ perturbation budget) =
128/255 ≈ 0.502
- α (step size) =
16/255 ≈ 0.0627
- optimizer iteration counts =
FGSM=1, PGD=10, AutoPGD=10, AutoAttack=20
- success threshold =
adv-mAP@50 < 0.5
- L2 regularization strength =
not reported
- indeterminate exclusion rule =
14/100 cells excluded
- ALLUDE texture resolution =
2048×2048
axioms (6)
- domain assumption Mitsuba 3 and Unreal Engine 5 render the same texture and weather similarly enough that optimizations transfer; simulated attacks predict physical deployment.
- ad hoc to paper Balanced categorical LHS with N=100 (86 after exclusions) yields representative main-effect estimates over the 5,400-cell configuration space.
- domain assumption The implemented detector losses and pretrained weights faithfully represent YOLOv11, Faster R-CNN, and DETR.
- domain assumption Re-unifying published attack UV maps onto the CARLA e-tron mesh preserves the published attacks' deployed appearance.
- standard math Logistic regression and likelihood-ratio tests on 86 binary outcomes with 26 predictors is a valid basis for factor attribution.
- domain assumption adv-mAP@50 with per-frame benign detection isolates attack effect even when benign detection is degraded.
read the original abstract
Adversarial attacks against vision models like object detectors are often evaluated under limited conditions, leaving their performance under-characterized. Bridging simulation and differentiable rendering enables more robust, end-to-end evaluation of these adversarial attacks, yet there is no easy-to-use, unified system that offers a rich set of customizable configurations for adversarial attacks across multiple scenes, objects, environmental and lighting conditions, and camera trajectories. We present ALLUDE, which addresses these gaps, offering first-of-its-kind evaluation capabilities across Linux and Windows. We comprehensively demonstrate ALLUDE's evaluation breadth through a two-pronged strategy: (1) using Latin Hypercube Sampling, we draw a representative subset from 5,400 configurations spanning 10 scene-object pairs, 9 weather conditions, 4 optimizers, 5 camera trajectories, and 3 detection models; (2) we stress-test existing attacks (CAMOU, RAUCA, FCA) under diverse weather conditions and continuous camera trajectories, revealing degradation of attack success across every attack, exposing evaluation gaps in prior work. Through ALLUDE's end-to-end differentiable rendering, adversarial attacks can be optimized against shifting real-world deployment conditions. Our cross-platform code is open source.
Figures
Reference graph
Works this paper leans on
-
[1]
Synthesizing robust adversarial exam- ples, 2018
Anish Athalye et al. Synthesizing robust adversarial exam- ples, 2018. 3
2018
-
[2]
Springer International Publishing, 2019
Shang-Tse Chen, Cory Cornelius, Jason Martin, and Duen Horng Chau.ShapeShifter: Robust Physical Adver- sarial Attack on Faster R-CNN Object Detector, page 52–68. Springer International Publishing, 2019. 2, 3
2019
-
[3]
Reliable evalua- tion of adversarial robustness with an ensemble of diverse parameter-free attacks, 2020
Francesco Croce and Matthias Hein. Reliable evalua- tion of adversarial robustness with an ensemble of diverse parameter-free attacks, 2020. 4, 11, 12
2020
-
[4]
Carla: An open urban driving sim- ulator, 2017
Alexey Dosovitskiy et al. Carla: An open urban driving sim- ulator, 2017. 2, 3
2017
-
[5]
Mid-air: A multi-modal dataset for ex- tremely low altitude drone flights
Micha ¨el Fonder. Mid-air: A multi-modal dataset for ex- tremely low altitude drone flights. In2019 IEEE/CVF Con- ference on Computer Vision and Pattern Recognition Work- shops (CVPRW), pages 553–562, 2019. 3
2019
-
[6]
Wichmann, and Wieland Brendel
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-trained CNNs are biased towards texture; increas- ing shape bias improves accuracy and robustness. InInter- national Conference on Learning Representations, 2019. 6 Table 4. adv-mAP@50 of the target vehicle,DETR, on the Audi E-Tron litera...
2019
-
[7]
Goodfellow, Jonathon Shlens, and Christian Szegedy
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples, 2015. 4, 11
2015
-
[8]
Matthew Hull, Haoyang Yang, Pratham Mehta, Mansi Phute, Aeree Cho, Haorang Wang, Matthew Lau, Wenke Lee, Wil- ian Lunardi, Martin Andreoni, et al. Complicitsplat: Down- stream models are vulnerable to blackbox attacks by 3d gaus- sian splat camouflages.arXiv preprint arXiv:2508.11854,
-
[9]
Wenzel Jakob, S ´ebastien Speierer, Nicolas Roussel, and De- lio Vicini. Dr. jit: A just-in-time compiler for differentiable rendering.ACM Transactions on Graphics (TOG), 41(4):1– 19, 2022. 3
2022
-
[10]
Cca: A novel camouflage coating attack on object detectors in remote sensing images.IEEE Trans- actions on Geoscience and Remote Sensing, 63:1–16, 2025
Ruiyang Jia et al. Cca: A novel camouflage coating attack on object detectors in remote sensing images.IEEE Trans- actions on Geoscience and Remote Sensing, 63:1–16, 2025. 3
2025
-
[11]
Neural 3d mesh renderer, 2017
Hiroharu Kato et al. Neural 3d mesh renderer, 2017. 3
2017
-
[12]
Skyscenes: A synthetic dataset for aerial scene understanding
Sahil Khose et al. Skyscenes: A synthetic dataset for aerial scene understanding. 2024. 3, 7
2024
-
[13]
Modular primitives for high-performance differentiable rendering, 2020
Samuli Laine et al. Modular primitives for high-performance differentiable rendering, 2020. 3
2020
-
[14]
Yang Li et al. Flexible physical camouflage generation based on a differential approach.arXiv preprint arXiv:2402.13575,
-
[15]
Padetbench: Towards benchmark- ing texture- and patch-based physical attacks against object detection.Knowledge-Based Systems, 329:114395, 2025
Jiawei Lian, Jianhong Pan, Lefan Wang, Yi Wang, Shaohui Mei, and Lap-Pui Chau. Padetbench: Towards benchmark- ing texture- and patch-based physical attacks against object detection.Knowledge-Based Systems, 329:114395, 2025. 4, 7
2025
-
[16]
Loper and Michael J
Matthew M. Loper and Michael J. Black. Opendr: An ap- proximate differentiable renderer. InComputer Vision – ECCV 2014, 2014. 3
2014
-
[17]
Towards deep learn- ing models resistant to adversarial attacks
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learn- ing models resistant to adversarial attacks. InInternational Conference on Learning Representations, 2018. 4, 11
2018
-
[18]
Conditional logit analysis of qualitative choice behavior
Daniel McFadden. Conditional logit analysis of qualitative choice behavior. 1972. 6
1972
-
[19]
Carla drone: Monocular 3d object detection from a different per- spective, 2024
Johannes Meier, Luca Scalerandi, Oussema Dhaouadi, Jacques Kaiser, Nikita Araslanov, and Daniel Cremers. Carla drone: Monocular 3d object detection from a different per- spective, 2024. 3
2024
-
[20]
Evaluating the robustness of seman- tic segmentation for autonomous driving against real-world adversarial patch attacks
Federico Nesti et al. Evaluating the robustness of seman- tic segmentation for autonomous driving against real-world adversarial patch attacks. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2280–2289, 2022. 3
2022
-
[21]
Capturing, reconstructing, and simu- lating: the urbanscene3d dataset, 2022
Liqiang Linand others. Capturing, reconstructing, and simu- lating: the urbanscene3d dataset, 2022. 3
2022
-
[22]
Mansi Phute, Matthew Hull, Haoran Wang, Alec Helbling, ShengYun Peng, Willian Lunardi, Martin Andreoni, Wenke Lee, and Duen Horng Chau. Undream: Bridging differen- tiable rendering and photorealistic simulation for end-to-end adversarial attacks.arXiv preprint arXiv:2510.16923, 2025. 3
Pith/arXiv arXiv 2025
-
[23]
Syndrone–multi-modal uav dataset for urban scenarios.arXiv preprint arXiv:2308.10491, 2023
Giulia Rizzoli et al. Syndrone–multi-modal uav dataset for urban scenarios.arXiv preprint arXiv:2308.10491, 2023. 3
Pith/arXiv arXiv 2023
-
[24]
Airsim: High-fidelity visual and physical simulation for autonomous vehicles, 2017
Shital Shah et al. Airsim: High-fidelity visual and physical simulation for autonomous vehicles, 2017. 3
2017
-
[25]
Active: Towards highly transferable 3d physical camouflage for universal and robust vehicle evasion
Naufal Suryanto, Yongsu Kim, Harashta Tatimma Larasati, Hyoeun Kang, Thi-Thu-Huong Le, Yoonyoung Hong, Hun- min Yang, Se-Yoon Oh, and Howon Kim. Active: Towards highly transferable 3d physical camouflage for universal and robust vehicle evasion. InProceedings of the IEEE/CVF In- ternational Conference on Computer Vision (ICCV), pages 4305–4314, 2023. 2, 3
2023
-
[26]
Donghua Wang, Tingsong Jiang, Jialiang Sun, Weien Zhou, Zhiqiang Gong, Xiaoya Zhang, Wen Yao, and Xiaoqian Chen. Fca: Learning a 3d full-coverage vehicle camou- flage for multi-view physical adversarial attack.Proceed- ings of the AAAI Conference on Artificial Intelligence, 36 (2):2414–2422, 2022. 3, 8, 15
2022
-
[27]
Adversarial examples in the physical world: A survey,
Jiakai Wang, Xianglong Liu, Jin Hu, Donghua Wang, Siyang Wu, Tingsong Jiang, Yuanfang Guo, Aishan Liu, and Jiantao Zhou. Adversarial examples in the physical world: A survey,
-
[28]
Tartanair: A dataset to push the limits of visual slam
Wenshan Wang et al. Tartanair: A dataset to push the limits of visual slam. In2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 4909–4916,
-
[29]
Physical adversarial attack meets computer vi- sion: A decade survey.IEEE Transactions on Pattern Anal- ysis and Machine Intelligence, 46(12):9797–9817, 2024
Hui Wei et al. Physical adversarial attack meets computer vi- sion: A decade survey.IEEE Transactions on Pattern Anal- ysis and Machine Intelligence, 46(12):9797–9817, 2024. 2, 3
2024
-
[30]
Tong Wu, Xuefei Ning, Wenshuo Li, Ranran Huang, Huazhong Yang, and Yu Wang. Physical adversarial attack on vehicle detector in the carla simulator.arXiv preprint arXiv:2007.16118, 2020. 3
Pith/arXiv arXiv 2007
-
[31]
Davis, and Tom Gold- stein
Zuxuan Wu, Ser-Nam Lim, Larry S. Davis, and Tom Gold- stein. Making an invisibility cloak: Real world adversarial attacks on object detectors. InComputer Vision – ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IV, page 1–17, Berlin, Hei- delberg, 2020. Springer-Verlag. 3
2020
-
[32]
Adversarial t-shirt! evading person detectors in a phys- ical world, 2020
Kaidi Xu, Gaoyuan Zhang, Sijia Liu, Quanfu Fan, Mengshu Sun, Hongge Chen, Pin-Yu Chen, Yanzhi Wang, and Xue Lin. Adversarial t-shirt! evading person detectors in a phys- ical world, 2020. 2, 3, 6
2020
-
[33]
Imperceptible adversarial examples in the physical world.arXiv preprint arXiv:2411.16622, 2024
Weilin Xu et al. Imperceptible adversarial examples in the physical world.arXiv preprint arXiv:2411.16622, 2024. 3
Pith/arXiv arXiv 2024
-
[34]
Tianyuan Zhang et al. Visual adversarial attack on vision- language models for autonomous driving.arXiv preprint arXiv:2411.18275, 2024. 3
Pith/arXiv arXiv 2024
-
[35]
CAMOU: Learning physical vehicle camouflages to adversarially attack detectors in the wild
Yang Zhang, Hassan Foroosh, Philip David, and Boqing Gong. CAMOU: Learning physical vehicle camouflages to adversarially attack detectors in the wild. InInternational Conference on Learning Representations, 2019. 2, 3, 15
2019
-
[36]
Jiawei Zhou et al. Rauca: A novel physical adversarial at- tack on vehicle detectors via robust and accurate camouflage generation.arXiv preprint arXiv:2402.15853, 2024. 2, 3, 8, 15 Supplementary Material S1. System Implementation Details S1.1. Weather and Lighting Configurations ALLUDEenables systematic evaluation of the attacks across nine distinct weat...
Pith/arXiv arXiv 2024
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.