REVIEW 3 major objections 5 minor 1 cited by
UnDREAM claims a framework that optimizes adversarial textures end-to-end inside photorealistic 3D simulation by routing gradients through a differentiable twin scene.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-04 09:07 UTC pith:YPAXGDNG
load-bearing objection Useful new simulation-to-differentiable-rendering bridge, but the 'end-to-end' label outruns the evidence: gradients come from a Mitsuba twin whose agreement with Unreal is never quantitatively checked. the 3 major comments →
UNDREAM: Bridging Differentiable Rendering and Photorealistic Simulation for End-to-end Adversarial Attacks
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The paper's central claim is that the non-differentiability of photorealistic simulation need not block gradient-based attacks. UnDREAM automatically converts a simulator scene—including camera and object poses, rotation conventions, and units—into a differentiable renderer scene for every frame. Each attack iteration renders images in the simulator, computes the victim model's loss, backpropagates through the differentiable twin to update the adversarial texture, and writes the updated texture back into the simulator. The authors demonstrate this on classification and object detection with PGD and Auto-PGD, reducing accuracy and mAP dramatically, and show that the optimized texture changes
What carries the argument
The load-bearing component is the twin-scene bridge: an automatic transformation of each simulator frame into a differentiable scene with the adversarial texture attached as a differentiable texture. The transformation reconciles the coordinate systems, rotation conventions, and units between the two renderers, so that gradients computed in the differentiable scene can be attributed to texture pixels that appear in the simulator's image. This is what lets the optimizer update the texture as it actually appears under the simulator's lighting, materials, and motion, without requiring the simulator itself to be differentiable.
Load-bearing premise
The loss is computed from the photorealistic simulator's images, but gradients are backpropagated through a separate differentiable re-rendering of the same scene, so the two renderings must match closely enough that texture updates truly improve the simulator's output.
What would settle it
Take a fixed texture and a fixed frame, render it in the simulator and in the differentiable twin, and compare the two images pixel by pixel under several lighting and weather conditions; any substantial mismatch means the gradient path is optimizing a different rendering than the one the victim model sees. A tighter test would measure whether a texture gradient computed through the differentiable renderer points in the same direction as a finite-difference perturbation of the texture in the simulator.
If this is right
- Adversarial attacks can be optimized directly inside scenes with weather, lighting, moving cameras, and animated objects, instead of superimposing patches onto pre-rendered frames.
- Any 3D object with a UV mapping—sphere, bin, car, umbrella—can serve as an attack surface, removing the restriction to 2D rectangular patches.
- Switching between attack algorithms reduces to changing a single line that updates the texture, making the pipeline easy to extend to PGD, Auto-PGD, or other gradient-based attacks.
- Because the texture is embedded in the scene rather than pasted per frame, attacks are temporally consistent across consecutive frames, which matters for tracking and video-based evaluation.
- The same loop can be turned around to evaluate defenses under realistic, high-fidelity conditions, giving a unified testbed for attacks and countermeasures.
Where Pith is reading between the lines
- If the twin-scene gradient flow is accurate enough, the same bridge could be used for non-adversarial inverse problems such as designing textures or materials to match a target appearance under simulated weather and lighting.
- Since the differentiable renderer supports differentiation with respect to scene parameters beyond textures, the framework could extend to optimizing object pose, camera path, or lighting rather than only the adversarial texture.
- A decisive but unperformed check is comparing a texture gradient computed through the twin renderer against a finite-difference perturbation in the simulator itself; the paper's white-texture verification alone does not rule out divergence under weather or material effects.
- The framework could plausibly generate physically varying adversarial training data, where perturbations are optimized in simulation before being used to robustify detectors.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents UnDREAM, a software framework that couples Unreal Engine 5 (UE5) photorealistic simulation with Mitsuba differentiable rendering. The framework converts UE5 LevelSequence scenes into Mitsuba XML scenes through coordinate/scale/rotation transforms, then runs an adversarial attack loop: render frames in UE5, compute a victim model's loss, backpropagate the loss through the Mitsuba twin scene, and update the 3D object texture before re-inserting it into UE5. The paper claims this is the first framework to enable end-to-end optimization of adversarial perturbations on arbitrary 3D objects inside photorealistic simulation. Experiments demonstrate PGD and Auto-PGD attacks on DETR-ResNet-50 for classification and detection tasks in park/city environments under different lighting and weather conditions, with a small set of objects (sphere, bin, car, umbrella).
Significance. If the central claim holds, UnDREAM is a useful open-source systems contribution: it lowers the barrier to optimizing 3D adversarial textures in a high-fidelity simulator, and the single-line attack integration is a practical advantage. The paper ships code and assets, which is commendable. However, the load-bearing 'end-to-end' claim depends on an unvalidated proxy-gradient assumption: the loss is scored on UE5 renderings while gradients are backpropagated through a separately constructed Mitsuba scene. The manuscript provides only a white-texture visual check for the twin scene and no quantitative alignment evidence. Until that is supplied, the framework is better described as a surrogate-gradient pipeline than an end-to-end differentiable simulation pipeline.
major comments (3)
- [§3.2, Algorithm 1; §3.1.2 step 4] The central claim of end-to-end optimization is undercut by an unvalidated proxy gradient. In Algorithm 1, lines 3–5 compute the loss on Unreal-rendered frames, but lines 7–9 backpropagate through the Mitsuba twin scene. For this to be a true end-to-end optimization, the Mitsuba rendering must be a faithful differentiable analogue of the Unreal rendering under the same lighting, weather, material, and post-processing conditions. The only validation described is a plain-white-texture visual check (§3.1.2 step 4), which can confirm geometry/coordinate conversion but not appearance or gradient fidelity. Appendix A documents only coordinate/rotation/scale conversion, not conversion of lights, weather, or materials. No image-level or gradient-level comparison between UE5 and Mitsuba is reported. Please add quantitative alignment experiments (e.g., pixel-wise error or gradient cosine similarit
- [§4.3, Table 3] The attack evaluation is presented as a single demonstration: one person walking in a park, one victim model (DETR-ResNet-50), and no indication of the number of frames, scenes, or random seeds. The 100% benign accuracy/mAP in Table 3 suggests a small or possibly single-frame evaluation set. For a framework whose contribution is to enable systematic evaluation of adversarial attacks, please report results across multiple scenes, sequences, and seeds, with mean and standard deviation. Also clarify the attack budget: what norm and bound do the values 0.78 and 1.00 refer to, and how is the budget applied to a detection mAP metric? Without this information, the quantitative claim in §4.3 is not reproducible.
- [§1, §4] The paper argues that existing pipelines optimize textures outside simulation and therefore fail under environmental interactions, but no experiment compares UnDREAM against such baselines. To support the claim that in-simulation optimization 'preserves lighting, perspective, and physical material interactions' and leads to better threat modeling, the manuscript should include a quantitative comparison with at least one prior approach (e.g., a 2D patch superimposed on UE5 frames, or a texture optimized in Mitsuba alone and then transferred to UE5). Such baselines are essential to show that the UnDREAM pipeline provides an actual improvement in attack effectiveness or transferability rather than merely a convenient implementation.
minor comments (5)
- [Contribution 2 (§1)] The sentence 'UnDREAM eliminates the need for calculating the bounds of the adversarial object across This leads to better alignment' is incomplete. Please revise.
- [Algorithm 1] The notation and control flow are unclear. Line 3 uses x_{i-n} without defining the index range, and the 'RENDERING JOB FINISH CALLBACK' procedure appears nested inside the for loop. Clarify the asynchronous rendering semantics and the relationship between attack iterations and rendering jobs.
- [Table 1] The 'only framework' claim in the caption and text is a strong novelty claim. Please define the comparison scope more precisely, and consider citing or discussing prior differentiable-simulation bridges if any exist, to avoid overclaiming.
- [§2.2] The statement that 'Mitsuba, due to its differentiable ray tracing method, has increased photorealism compared to alternatives such as PyTorch3D' is not substantiated. Rephrase as an architectural difference or provide a citation.
- [§7] Typo: 'hte free assets' should be 'the free assets'. The reproducibility statement is otherwise a strength.
Circularity Check
No significant circularity found; the framework is evaluated against external victim models, and the proxy-gradient concern is a validation issue, not circularity.
full rationale
I walked the paper's claimed derivation chain. UnDREAM's core loop (Algorithm 1) computes a loss from an external victim model's predictions (M(x)) on Unreal-rendered images, then updates textures via gradients propagated through a separately constructed Mitsuba scene ($3.2, Algorithm 1 lines 3–9). This is not circular: no parameter is fitted to a target quantity and then renamed as a prediction. The reported attack results in Table 3 are measured against an external model (DETR-ResNet-50) using standard attacks (PGD, Auto-PGD), so the headline results do not reduce to the framework's own assumptions. The 'first/only framework' claims in the abstract and Table 1 are comparative novelty claims, not derived predictions; no uniqueness theorem or load-bearing result is imported from prior self-citations. The self-citations (e.g., Hull et al. 2024) appear only in related-work context and are not load-bearing. The main weakness—that gradients are computed through a Mitsuba twin validated only by a white-texture visual check ($3.1.2 step 4)—is a real correctness/validation gap about whether the surrogate gradients match Unreal's rendering, but it is not a circularity: the derivation does not assume what it claims to establish. Accordingly, no circular step is identified.
Axiom & Free-Parameter Ledger
axioms (4)
- domain assumption Unreal Engine rendering is non-differentiable and Mitsuba can serve as a differentiable surrogate of the same scene.
- domain assumption Coordinate, scale, handedness, and rotation conversions (§3.1.2, Table 4) preserve object-camera geometry and UV texture mapping between Unreal and Mitsuba.
- ad hoc to paper Backpropagating the loss gradient through the Mitsuba twin scene yields texture updates that reduce the loss on Unreal-rendered images.
- domain assumption Victim models (DETR-ResNet-50, YOLO variants) are treated as fixed white-box models with known gradients.
read the original abstract
Deep learning models deployed in safety critical applications like autonomous driving use simulations to test their robustness against adversarial attacks in realistic conditions. However, these simulations are non-differentiable, forcing researchers to create attacks that do not integrate simulation environmental factors, reducing attack success. To address this limitation, we introduce UNDREAM, the first software framework that bridges the gap between photorealistic simulators and differentiable renderers to enable end-to-end optimization of adversarial perturbations on any 3D objects. UNDREAM enables manipulation of the environment by offering complete control over weather, lighting, backgrounds, camera angles, trajectories, and realistic human and object movements, thereby allowing the creation of diverse scenes. We showcase a wide array of distinct physically plausible adversarial objects that UNDREAM enables researchers to swiftly explore in different configurable environments. This combination of photorealistic simulation and differentiable optimization opens new avenues for advancing research of physical adversarial attacks.
Figures
Forward citations
Cited by 1 Pith paper
-
ALLUDE: A Unified Evaluation System for Configurable Attacks in Differentiable Environments
A configurable, cross-platform simulator-based evaluation platform shows that object class and camera trajectory, not weather or detector choice, dominate whether 3D adversarial patch attacks succeed, and that all tes...
Reference graph
Works this paper leans on
-
[2]
Samuli Laine, Janne Hellsten, Tero Karras, Yeongho Seol, Jaakko Lehtinen, and Timo Aila
URLhttps://doi.org/ 10.1007/978-3-031-72986-7_2. Samuli Laine, Janne Hellsten, Tero Karras, Yeongho Seol, Jaakko Lehtinen, and Timo Aila. Modular primitives for high-performance differentiable rendering,
-
[6]
Sahil Khose, Anisha Pal, Aayushi Agarwal, Deepanshi, Judy Hoffman, and Prithvijit Chattopad- hyay
URL https://arxiv.org/abs/1711.07566. Sahil Khose, Anisha Pal, Aayushi Agarwal, Deepanshi, Judy Hoffman, and Prithvijit Chattopad- hyay. Skyscenes: A synthetic dataset for aerial scene understanding. InComputer Vision – ECCV 2024: 18th European Conference, pp. 19–35, Berlin, Heidelberg,
Pith/arXiv arXiv 2024
-
[11]
URLhttps://arxiv.org/abs/1810. 10438. 10 Johannes Meier, Luca Scalerandi, Oussema Dhaouadi, Jacques Kaiser, Nikita Araslanov, and Daniel Cremers. Carla drone: monocular 3d object detection from a different perspective. InDAGM German Conference on Pattern Recognition, pp. 137–152. Springer, 2024a. Johannes Meier, Luca Scalerandi, Oussema Dhaouadi, Jacques ...
-
[12]
Adversarial robustness toolbox v1
Maria-Irina Nicolae, Mathieu Sinn, Minh Ngoc Tran, Beat Buesser, Ambrish Rawat, Martin Wis- tuba, Valentina Zantedeschi, Nathalie Baracaldo, Bryant Chen, Heiko Ludwig, et al. Adversarial robustness toolbox v1. 0.0.arXiv preprint arXiv:1807.01069,
-
[13]
doi: 10.1109/W ACV .2018.00168. Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan-Yen Lo, Justin John- son, and Georgia Gkioxari. Accelerating 3d deep learning with pytorch3d.arXiv preprint arXiv:2007.08501,
arXiv 2018
-
[14]
Syndrone–multi-modal uav dataset for urban scenarios.arXiv preprint arXiv:2308.10491,
Giulia Rizzoli, Francesco Barbato, Matteo Caligiuri, and Pietro Zanuttigh. Syndrone–multi-modal uav dataset for urban scenarios.arXiv preprint arXiv:2308.10491,
-
[15]
URLhttps://arxiv.org/abs/ 1705.05065. Wenshan Wang, Delong Zhu, Xiangwei Wang, Yaoyu Hu, Yuheng Qiu, Chen Wang, Yafei Hu, Ashish Kapoor, and Sebastian Scherer. Tartanair: A dataset to push the limits of visual slam. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4909– 4916,
Pith/arXiv arXiv 2020
-
[16]
doi: 10.1109/IROS45743.2020.9341801. Hui Wei, Hao Tang, Xuemei Jia, Zhixiang Wang, Hanxun Yu, Zhubo Li, Shin’ichi Satoh, Luc Van Gool, and Zheng Wang. Physical adversarial attack meets computer vision: A decade survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(12):9797–9817,
arXiv 2020
-
[17]
Tong Wu, Xuefei Ning, Wenshuo Li, Ranran Huang, Huazhong Yang, and Yu Wang. Physical adversarial attack on vehicle detector in the carla simulator.arXiv preprint arXiv:2007.16118,
Pith/arXiv arXiv 2007
-
[18]
Imperceptible adversarial examples in the physical world.arXiv preprint arXiv:2411.16622,
Weilin Xu, Sebastian Szyller, Cory Cornelius, Luis Murillo Rojas, Marius Arvinte, Alvaro Ve- lasquez, Jason Martin, and Nageen Himayat. Imperceptible adversarial examples in the physical world.arXiv preprint arXiv:2411.16622,
-
[19]
Tianyuan Zhang, Lu Wang, Xinwei Zhang, Yitong Zhang, Boyi Jia, Siyuan Liang, Shengshan Hu, Qiang Fu, Aishan Liu, and Xianglong Liu. Visual adversarial attack on vision-language models for autonomous driving.arXiv preprint arXiv:2411.18275,
-
[20]
Jiawei Zhou, Linye Lyu, Daojing He, and Yu Li. Rauca: A novel physical adversarial attack on ve- hicle detectors via robust and accurate camouflage generation.arXiv preprint arXiv:2402.15853,
-
[2017]
URLhttps://arxiv.org/abs/1711.03938. Unreal Engine. Lumen Global Illumination and Reflections in Unreal Engine — Un- real Engine 5.6 Documentation — Epic Developer Community — dev.epicgames.com. https://dev.epicgames.com/documentation/en-us/unreal-engine/ lumen-global-illumination-and-reflections-in-unreal-engine. [Ac- cessed 20-09-2025]. Unreal Engine. U...
Pith/arXiv arXiv 2025
-
[2018]
Wenxiao Cai, Ke Jin, Jinyan Hou, Cong Guo, Letian Wu, and Wankou Yang
URLhttps://arxiv.org/abs/1707.07397. Wenxiao Cai, Ke Jin, Jinyan Hou, Cong Guo, Letian Wu, and Wankou Yang. Vdd: Varied drone dataset for semantic segmentation,
-
[2019]
Zhanhao Hu, Wenda Chu, Xiaopei Zhu, Hui Zhang, Bo Zhang, and Xiaolin Hu
doi: 10.1109/CVPRW.2019.00081. Zhanhao Hu, Wenda Chu, Xiaopei Zhu, Hui Zhang, Bo Zhang, and Xiaolin Hu. Physically realizable natural-looking clothing textures evade person detectors via 3d modeling. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16975–16984,
arXiv 2019
-
[2020]
Yang Li, Wenyi Tan, Tingrui Wang, Xinkai Liang, and Quan Pan
URLhttps://arxiv.org/ abs/2011.03277. Yang Li, Wenyi Tan, Tingrui Wang, Xinkai Liang, and Quan Pan. Flexible physical camouflage generation based on a differential approach.arXiv preprint arXiv:2402.13575,
Pith/arXiv arXiv 2011
-
[2022]
URLhttps://arxiv.org/abs/2107. 04286. Matthew M. Loper and Michael J. Black. Opendr: An approximate differentiable renderer. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.),Computer Vision – ECCV 2014, pp. 154–169, Cham,
2014
-
[2023]
Matthew Hull, Haoran Wang, Matthew Lau, Alec Helbling, Mansi Phute, Chao Zhang, Zsolt Kira, Willian Lunardi, Martin Andreoni, Wenke Lee, et al. Renderbender: A survey on adversarial attacks using differentiable rendering.arXiv preprint arXiv:2411.09749,
-
[2024]
Springer-Verlag. ISBN 978-3-031-72985-0. doi: 10.1007/978-3-031-72986-7
-
[2025]
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun
URLhttps://arxiv.org/abs/2305.13608. Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator,
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.