REVIEW 4 major objections 6 minor 39 references
RGB-Event Fusion with Self-Attention for Collision Prediction
T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read Event cameras win the collision-prediction cost trade-off
desk verdict A competent, incremental empirical study whose qualitative takeaway (event-only wins on accuracy-per-cost) is credible, but whose headline percentages do not match its own tables and figures. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing design is a dual-encoder network: two six-layer convolutional branches take an $80\times80\times3$ RGB frame and an $80\times80\times1$ event representation, their outputs are concatenated, and a self-attention layer re-weights the fused features before a fully connected layer produces the two collision coordinates and the time to impact. Self-attention is the fusion mechanism that lets the network decide, per feature, how much each sensor modality matters. The event representation is a polarity-difference accumulation over a $20\,\mathrm{ms}$ window, kept deliberately single-channel for efficient FPGA deployment, and the training target is the minimum distance between the ball and drone trajectories recorded by a motion-capture system. The compression story runs through the same encoder trained with LSQ+ quantization for 2-8 bit weights and with three binarization schemes (DoReFa-Net, IRNet, ReActNet), which together trace the accuracy-versus-operations frontier for the event-only model.
What would settle it
Run the trained models on a moving quadrotor that must dodge a thrown object and compare the predicted time-to-collision with the time the maneuver actually needs; if the event-only model's error exceeds the available reaction time, or if fusion's long-range advantage disappears outside the arena, the paper's accuracy-per-cost conclusion is overturned.
Extended reading notes
Core claim
The paper's central claim is that fusing RGB and event streams through self-attention improves collision prediction, but only at the margin, because the event stream already carries most of the task-relevant information. At a common 50 Hz prediction throughput, the fusion model is reported to be about 1% more accurate than single-modality models on average and 10% more accurate for objects beyond 0.5 m, while needing 71% more memory and 105% more FLOPs; the event-only model is reported to beat the RGB-only model by 4% in position error and 26% in time error at similar cost. Quantized 8-bit and 4-bit event models keep most of this accuracy while shrinking memory, whereas 1-bit binarized variants degrade substantially. Read on the authors' terms, the result is a quantified trade-off: multi-modal fusion is a real but expensive refinement, and event-only perception is the efficient default for fast on-board collision avoidance.
Load-bearing premise
The results assume that predictions learned from foam-ball throws at a static, plexiglass-protected drone in a motion-capture arena, with collisions defined by minimum trajectory distance, transfer to a real moving drone that must evade an obstacle; the paper itself notes that flying-drone validation and thrust-to-weight analysis are still missing.
Editorial extensions
If this is right
- Single-modality event-based perception can serve as the default sensor for this task, since it is more accurate than RGB at similar compute and nearly as accurate as fusion.
- Multi-modal fusion pays off mainly for objects farther than roughly 0.5 m, where the paper measures a 10% accuracy gain, while near objects show no fusion benefit.
- Quantized 4-bit and 8-bit event models preserve most of the accuracy while cutting memory, making them suitable for memory-limited edge hardware such as an FPGA.
- At reported error levels near 200 mm in position and 100 ms in time, a drone may have time to execute an evasive maneuver, contingent on the real-world flight validation the paper says is still needed.
Reading between the lines
- One extension the authors do not explore is a gating strategy: run the event branch continuously and enable the RGB branch only for objects beyond 0.5 m, which could capture most of the fusion gain without paying the full +105% FLOPs cost.
- The 1% average fusion gain was measured in a well-lit indoor arena; under low light or motion blur, where RGB is known to degrade, fusion might plausibly offer larger gains, a hypothesis that would require new data to test.
- Because the collision label is defined by the minimum distance between two 3D trajectories, it treats the drone as a point; using the drone's physical footprint as the collision surface could change the error distribution and the apparent advantage of each modality.
- The fixed 20 ms event window bounds the temporal resolution of the event branch, so increasing the event rate is a natural next test for reducing time-to-collision error, as the conclusion already hints.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a neural network for predicting collision position and time-to-collision for a UAV, fusing RGB frames and event-based camera streams. The architecture uses separate convolutional encoders per modality, concatenates the features, and applies self-attention before a fully connected output layer. The authors evaluate fusion against single-modality EVS and RGB models on the ABCD dataset, report computational costs in FLOPs and memory, and additionally study low-bit quantization and binarization of the EVS model. The headline claims are that fusion improves accuracy by about 1% on average and 10% for distances beyond 0.5 m at a cost of +71% memory and +105% FLOPs, and that the EVS-only model is a competitive, more efficient alternative.
Significance. If the quantitative claims are correct, the paper provides a useful benchmark and a practical comparison of RGB-only, event-only, and fused collision prediction on an edge-oriented platform. Strengths include the public code and dataset link, the inclusion of multiple distance bins, and the systematic evaluation of quantized and binarized variants. However, the manuscript's headline numbers are internally inconsistent across the abstract, conclusion, Table I, and Fig. 2, and no statistical tests or multiple-seed variance are reported. The qualitative conclusion that EVS offers the best accuracy-per-cost trade-off is plausible, but the specific accuracy-improvement percentages need to be corrected and supported before the paper can be accepted.
major comments (4)
- [Abstract and Section IV.A.2, Table I] The abstract's claim that the fusion model offers a '10% for distances beyond 0.5 m' improvement is not supported by Table I. In the 0.5-1.0 m bin the fusion mean error (228.92 mm) is 1.5% worse than the RGB mean (225.45 mm), and in the 2.0+ m bin it is 4.6% worse than the EVS mean (193.55 mm). The 10% figure appears only when comparing fusion to RGB in the 1.0-1.5 m and 1.5-2.0 m bins (10.4% and 11.3%). The '1% on average' figure is also not derivable from Fig. 2, where fusion is 0.87% better than EVS and 4.5% better than RGB. Please state the exact aggregation rule used and correct the abstract and conclusion accordingly.
- [Conclusion vs. Fig. 2] The conclusion states that the EVS model reduces position error by approximately 11% over RGB, but Fig. 2 reports 216.42 mm for EVS versus 224.60 mm for RGB, a reduction of 3.6%. The abstract's value of 4% matches Fig. 2. These two claims cannot both be correct. Either the conclusion should be restated to match the reported figure, or the authors should identify the specific subset of data or metric that produces the 11% value.
- [Fig. 2 and Table I] No significance tests, confidence intervals, or multiple-seed variance are reported. The fusion advantage over EVS in Fig. 2 is only 1.9 mm (0.87%), while the per-bin standard deviations in Table I are often larger than 100 mm. Without uncertainty quantification, the claim that 'the fusion model consistently achieves higher precision' is not established. Please add statistical comparisons, error bars, or per-seed variance for the main results.
- [Section IV.A.1] The paper motivates the work by UAV obstacle avoidance, but all experiments use a static drone protected by a plexiglass barrier. The text acknowledges that 'further real world evaluations, with a flying drone, and analysis of thrust to weight ratio' are needed. This limitation should be reflected not only as a caveat but also in the abstract and conclusion, which currently present the fusion and EVS results without this external-validity qualification.
minor comments (6)
- [Section II.A.2] There is a double period in 'monocular event camera..' that should be corrected.
- [Section IV.A.1] The word 'thurst' should be 'thrust'.
- [Section III.A] The dataset description reports 'elevation angles from 0.017 m to 2.009 m', which uses meters for an angular quantity; please correct the units or the quantity described.
- [Section III.D and Section IV.B.2] The notation 'IRNet*' is used in the figures and experiments but is only informally defined in the text; please define it explicitly at first use.
- [Fig. 4] The x-axis labels appear as '102' and '103', which should be rendered as 10^2 and 10^3; also, please add a note in the caption that the axis is logarithmic.
- [Section IV.A.2, Table I] The loss is described as MSE, but Table I reports mean absolute error in millimeters; please clarify whether the means are of the Euclidean norm of the prediction error or derived from an RMSE-style metric.
Circularity Check
No circular derivation found; self-built dataset and platform are empirical infrastructure, not circular inputs.
full rationale
The derivation chain is empirical end-to-end: RGB and event inputs are preprocessed into fixed-size tensors, passed through separate encoders, fused by self-attention, and mapped by a fully connected layer to collision position and time. Training minimizes MSE against ground-truth labels computed from an external Vicon motion-capture system, and all reported accuracy numbers come from held-out test splits. No parameter is fitted to the test set and then renamed a prediction; no equation is defined in terms of the quantity it is said to predict. The dataset ABCD [8] and the SwiftEagle platform [31] are self-cited, but they serve as data-collection infrastructure rather than as mathematical premises, and the ground truth is externally measured rather than produced by the model. The abstract's specific numeric claims, such as '10% for distances beyond 0.5m,' are not uniformly supported by Table I (e.g., fusion is worse than RGB in the 0.5-1.0 m bin and worse than EVS in the 2.0+ m bin), but that is a consistency and correctness concern, not circularity. Similarly, the conclusion's 'approximately 11%' EVS-over-RGB position improvement differs from Fig. 2's 3.6%; again, this is an internal numerical inconsistency, not a reduction of the result to its inputs. The paper's central contributions remain independently testable empirical claims, so no circular step can be exhibited.
Assumptions & free parameters
free parameters (2)
- Event accumulation window T =
20 ms (varied in Section IV.B)
- Input resolution =
80 x 80 pixels
assumptions (4)
- domain assumption Vicon motion capture provides accurate ground truth for ball and drone positions.
- domain assumption The FPGA pipeline synchronizes RGB frames and event streams within the 20 ms input window.
- domain assumption The static-drone ball-throw scenario is representative of UAV collision avoidance.
- domain assumption A single thrown ball is the only obstacle and is visible through its IR LEDs.
Cite this review
Pith. "Pith review of RGB-Event Fusion with Self-Attention for Collision Prediction." pith.science (2026). https://pith.science/paper/NOQ35WPP
@misc{pith2026250504258,
author = {Pith},
title = {Pith review of: RGB-Event Fusion with Self-Attention for Collision Prediction},
year = {2026},
howpublished = {\url{https://pith.science/paper/NOQ35WPP}},
note = {Machine review of arXiv:2505.04258}
}
read the original abstract
Ensuring robust and real-time obstacle avoidance is critical for the safe operation of autonomous robots in dynamic, real-world environments. This paper proposes a neural network framework for predicting the time and collision position of an unmanned aerial vehicle with a dynamic object, using RGB and event-based vision sensors. The proposed architecture consists of two separate encoder branches, one for each modality, followed by fusion by self-attention to improve prediction accuracy. To facilitate benchmarking, we leverage the ABCD [8] dataset collected that enables detailed comparisons of single-modality and fusion-based approaches. At the same prediction throughput of 50Hz, the experimental results show that the fusion-based model offers an improvement in prediction accuracy over single-modality approaches of 1% on average and 10% for distances beyond 0.5m, but comes at the cost of +71% in memory and + 105% in FLOPs. Notably, the event-based model outperforms the RGB model by 4% for position and 26% for time error at a similar computational cost, making it a competitive alternative. Additionally, we evaluate quantized versions of the event-based models, applying 1- to 8-bit quantization to assess the trade-offs between predictive performance and computational efficiency. These findings highlight the trade-offs of multi-modal perception using RGB and event-based cameras in robotic applications.
Figures
Reference graph
Works this paper leans on
-
[1]
Deep learning using rectified linear units (relu)
Abien Fred Agarap. Deep learning using rectified linear units (relu). arXiv, 2019
work page 2019
-
[2]
Xcel-ram: Accelerating binary neural networks in high-throughput sram compute arrays
Amogh Agrawal, Akhilesh Jaiswal, Deboleena Roy, Bing Han, Gopalakrishnan Srinivasan, Aayush Ankit, and Kaushik Roy. Xcel-ram: Accelerating binary neural networks in high-throughput sram compute arrays. IEEE Transactions on Circuits and Systems I: Regular Papers , 2019
work page 2019
-
[3]
Lsq+: Improving low-bit quantization through learnable offsets and better initialization
Yash Bhalgat, Jinwon Lee, Markus Nagel, Tijmen Blankevoort, and Nojun Kwak. Lsq+: Improving low-bit quantization through learnable offsets and better initialization. arXiv:2004.09576, 2020
arXiv 2004
-
[4]
Monocular event-based vision for obstacle avoidance with a quadrotor
Anish Bhattacharya, Marco Cannici, Nishanth Rao, Yuezhan Tao, Vijay Kumar, Nikolai Matni, and Davide Scaramuzza. Monocular event-based vision for obstacle avoidance with a quadrotor. 8th Annual Conference on Robot Learning , 2024
work page 2024
-
[5]
Sizhen Bian, Lukas Schulthess, Georg Rutishauser, Alfio Di Mauro, Luca Benini, and Michele Magno. Colibriuav: An ultra-fast, energy- efficient neuromorphic edge processing uav-platform with event-based and frame-based cameras. 9th International Workshop on Advances in Sensors and Interfaces (IWASI) , 2023
work page 2023
-
[6]
Fast motion understanding with spatiotemporal neural networks and dynamic vision sensors
Anthony Bisulco, Fernando Cladera, V olkan Isler, and Daniel Lee. Fast motion understanding with spatiotemporal neural networks and dynamic vision sensors. IEEE International Conference on Robotics and Automation (ICRA) , 2021
work page 2021
-
[7]
Retina : Low-power eye tracking with event camera and spiking hardware
Pietro Bonazzi, Sizhen Bian, Giovanni Lippolis, Yawei Li, Sadique Sheik, and Michele Magno. Retina : Low-power eye tracking with event camera and spiking hardware. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops , 2024
work page 2024
-
[8]
Towards low-latency event-based obstacle avoidance on a fpga-drone, 2025
Pietro Bonazzi, Christian V ogt, Michael Jost, Lyes Khacef, Federico Paredes-Vall´es, and Michele Magno. Towards low-latency event-based obstacle avoidance on a fpga-drone, 2025
work page 2025
Show all 39 references
-
[9]
Benosman
Xavier Clady, Charles Clercq, Sio-Hoi Ieng, Fouzhan Houseini, Marco Randazzo, Lorenzo Natale, Chiara Bartolozzi, and Ryad B. Benosman. Asynchronous visual event-based time-to-contact. Frontiers in Neuro- science, 2014
2014
-
[10]
Christian Creß, Walter Zimmer, Nils Purschke, Bach Ngoc Doan, Sven Kirchner, Venkatnarayanan Lakshminarasimhan, Leah Strand, and Alois C. Knoll. Tumtraf event: Calibration and fusion resulting in a dataset for roadside event-based and rgb cameras. IEEE Transactions on Intellig...
2024
-
[11]
Multi-modal fusion of event and rgb for monocular depth estimation using a unified transformer-based architec- ture
Anusha Devulapally, Md Fahim Faysal Khan, Siddharth Advani, and Vijaykrishnan Narayanan. Multi-modal fusion of event and rgb for monocular depth estimation using a unified transformer-based architec- ture. IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion (CVPR)...
2024
-
[12]
Dynamic obstacle avoidance for quadrotors with event cameras
Davide Falanga, Kevin Kleber, and Davide Scaramuzza. Dynamic obstacle avoidance for quadrotors with event cameras. Science Robotics, 2020
2020
-
[13]
Event-based agile object catching with a quadrupedal robot
Benedek Forrai, Takahiro Miki, Daniel Gehrig, Marco Hutter, and Da- vide Scaramuzza. Event-based agile object catching with a quadrupedal robot. IEEE International Conference on Robotics and Automation (ICRA), 2023
2023
-
[14]
Advancing object detection for autonomous vehicles via general purpose event-rgb fusion
Hajer Fradi and Panagiotis Papadakis. Advancing object detection for autonomous vehicles via general purpose event-rgb fusion. 2024 Eighth IEEE International Conference on Robotic Computing (IRC) , 2024
2024
-
[15]
Davison, Jorg Conradt, Kostas Daniilidis, and Davide Scaramuzza
Guillermo Gallego, Tobi Delbruck, Garrick Orchard, Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger, Andrew J. Davison, Jorg Conradt, Kostas Daniilidis, and Davide Scaramuzza. Event-based vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intell...
2022
-
[16]
Combining events and frames using recurrent asynchronous multimodal networks for monocular depth prediction
Daniel Gehrig, Michelle R ¨uegg, Mathias Gehrig, Javier Hidalgo-Carri ´o, and Davide Scaramuzza. Combining events and frames using recurrent asynchronous multimodal networks for monocular depth prediction. IEEE Robotics and Automation Letters , 2021
2021
-
[17]
Letizia Gionfrida, Daekyum Kim, Davide Scaramuzza, Dario Farina, and Robert D. Howe. Wearable robots for the real world need vision. Science Robotics , 2024
2024
-
[18]
Simultaneous neuromorphic selection of multiple salient objects for event vision
Am ´elie Gruel, Jean Martinet, and Michele Magno. Simultaneous neuromorphic selection of multiple salient objects for event vision. 2023 international joint conference on neural networks (IJCNN) , 2023
2023
-
[19]
Fast-dynamic-vision: Detection and tracking dynamic objects with event and depth sensing
Botao He, Haojia Li, Siyuan Wu, Dong Wang, Zhiwei Zhang, Qianli Dong, Chao Xu, and Fei Gao. Fast-dynamic-vision: Detection and tracking dynamic objects with event and depth sensing. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , 2021
2021
-
[20]
Kingma and Jimmy Ba
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv, 2017
2017
-
[21]
Reactnet: Towards precise binary neural network with generalized acti- vation functions
Zechun Liu, Zhiqiang Shen, Marios Savvides, and Kwang-Ting Cheng. Reactnet: Towards precise binary neural network with generalized acti- vation functions. arXiv, 2020
2020
-
[22]
Learning high-speed flight in the wild
Antonio Loquercio, Elia Kaufmann, Ren ´e Ranftl, Matthias M ¨uller, Vladlen Koltun, and Davide Scaramuzza. Learning high-speed flight in the wild. Science Robotics , 2021
2021
-
[23]
Milde, Olivier J.N
Moritz B. Milde, Olivier J.N. Bertrand, Ryad Benosmanz, Martin Egel- haaf, and Elisabetta Chicca. Bioinspired event-driven collision avoidance algorithm based on optic flow. 2015 International Conference on Event- based Control, Communication, and Signal Processing (EBCCSP), 2015
2015
-
[24]
Fast trajectory end-point prediction with event cameras for reactive robot control
Marco Monforte, Luna Gava, Massimiliano Iacono, Arren Glover, and Chiara Bartolozzi. Fast trajectory end-point prediction with event cameras for reactive robot control. 2023
2023
-
[25]
Paredes-Vall ´es, J
F. Paredes-Vall ´es, J. J. Hagenaars, J. Dupeyroux, S. Stroobants, Y . Xu, and G. C. H. E. de Croon. Fully neuromorphic vision and control for autonomous drone flight. Science Robotics , 2024
2024
-
[26]
Deep visual odometry with events and frames
Roberto Pellerito, Marco Cannici, Daniel Gehrig, Joris Belhadj, Olivier Dubois-Matra, Massimo Casasco, and Davide Scaramuzza. Deep visual odometry with events and frames. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , 2024
2024
-
[27]
Forward and backward information retention for accurate binary neural networks
Haotong Qin, Ruihao Gong, Xianglong Liu, Mingzhu Shen, Ziran Wei, Fengwei Yu, and Jingkuan Song. Forward and backward information retention for accurate binary neural networks. arXiv, 2020
2020
-
[28]
Free as a bird: Event-based dynamic sense-and-avoid for ornithopter robot flight
Juan Pablo Rodr ´ıguez-G´omez, Raul Tapia, Maria del Mar Guzm ´an Garcia, Jose Ramiro Mart ´ınez-de Dios, and Anibal Ollero. Free as a bird: Event-based dynamic sense-and-avoid for ornithopter robot flight. IEEE Robotics and Automation Letters , 2022
2022
-
[29]
Evdodgenet: Deep dynamic obstacle dodging with event cameras
Nitin J Sanket, Chethan M Parameshwara, Chahat Deep Singh, Ash- win V Kuruttukulam, Cornelia Ferm ¨uller, Davide Scaramuzza, and Yiannis Aloimonos. Evdodgenet: Deep dynamic obstacle dodging with event cameras. IEEE International Conference on Robotics and Automation (ICRA) , 2020
2020
-
[30]
Fusing event-based and rgb camera for robust object detection in adverse conditions
Abhishek Tomy, Anshul Paigwar, Khushdeep S Mann, Alessandro Renzaglia, and Christian Laugier. Fusing event-based and rgb camera for robust object detection in adverse conditions. 2022 International Conference on Robotics and Automation (ICRA) , 2022
2022
-
[31]
Swifteagle: An ad- vanced open-source, miniaturized fpga uas platform with dual dvs/frame camera for cutting-edge low-latency autonomous algorithms
Christian V ogt, Michael Jost, and Michele Magno. Swifteagle: An ad- vanced open-source, miniaturized fpga uas platform with dual dvs/frame camera for cutting-edge low-latency autonomous algorithms. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , 2024
2024
-
[32]
Evreflex: Dense time-to-impact prediction for event-based obstacle avoidance
Celyn Walters and Simon Hadfield. Evreflex: Dense time-to-impact prediction for event-based obstacle avoidance. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , 2021
2021
-
[33]
Dynamic environment-driven au- tonomous drone path planning via deep reinforcement learning
Qiuhong Wang and Jingjing Gu. Dynamic environment-driven au- tonomous drone path planning via deep reinforcement learning. 2024
2024
-
[34]
A real-time dynamic obstacle tracking and mapping system for uav navigation and collision avoidance with an rgb-d camera
Zhefan Xu, Xiaoyang Zhan, Baihan Chen, Yumeng Xiu, Chenhao Yang, and Kenji Shimada. A real-time dynamic obstacle tracking and mapping system for uav navigation and collision avoidance with an rgb-d camera. 2023
2023
-
[35]
Yasin, Sherif A
Jawad N. Yasin, Sherif A. S. Mohamed, Mohammad-hashem Hagh- bayan, Jukka Heikkonen, Hannu Tenhunen, Muhammad Mehboob Yasin, and Juha Plosila. Night vision obstacle detection and avoidance based on bio-inspired vision sensors. IEEE SENSORS , 2020
2020
-
[36]
Neurobench: A framework for benchmarking neuromorphic computing algorithms and systems
Jason Yik, Korneel Van den Berghe, and Douwe den Blanken et al. Neurobench: A framework for benchmarking neuromorphic computing algorithms and systems. arXiv, 2025
2025
-
[37]
Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv:1606.06160, 2016
2016 arXiv
-
[38]
Rgb-event fusion for moving object detection in autonomous driving
Zhuyun Zhou, Zongwei Wu, R ´emi Boutteau, Fan Yang, C ´edric Demon- ceaux, and Dominique Ginhac. Rgb-event fusion for moving object detection in autonomous driving. IEEE International Conference on Robotics and Automation (ICRA) , 2023
2023
-
[39]
From chaos comes order: Ordering event representations for object recognition and detection
Nikola Zubi ´c, Daniel Gehrig, Mathias Gehrig, and Davide Scaramuzza. From chaos comes order: Ordering event representations for object recognition and detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 12846–12856, October 2023
2023
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.