pith. sign in

arxiv: 2605.26328 · v1 · pith:FP5DB343new · submitted 2026-05-25 · 💻 cs.CV

RadarSim: Simulating Single-Chip Radar via Multimodal Neural Fields

Chuhan Chen , Tianshu Huang , Akarsh Prabhakara , Chaithanya Kumar Mummadi , Zhongxiao Cong , Anthony Rowe , Matthew O'Toole , Deva Ramanan This is my paper

Pith reviewed 2026-06-29 22:15 UTC · model grok-4.3

classification 💻 cs.CV
keywords radar simulationneural fieldsmultimodal sensingdifferentiable renderingdoppler radarcamera-radar fusionnovel view synthesis
0
0 comments X

The pith

RadarSim generates sharper Doppler radar range images by initializing a neural field from camera data instead of radar alone.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents RadarSim, a unified differentiable renderer that builds a neural field representation starting from RGB camera views. This field then produces simulated Doppler radar range images, using the camera's high angular resolution to compensate for the sparse sampling typical of radar. Demonstrations on a new dataset of synchronized radar and camera recordings from a handheld rig show clearer geometry and motion details than methods relying only on radar inputs. The approach treats radar reconstruction as a novel view synthesis task but augments it with cross-modal initialization. If the claim holds, simulation of single-chip radar becomes more practical for sensor prototyping and algorithm development.

Core claim

RadarSim is a unified differentiable renderer which leverages the high angular resolution of RGB cameras to generate Doppler radar range images from a camera-initialized neural field. On a custom dataset of calibrated radar-camera recordings, this produces sharper geometry and Doppler range frames than radar-only reconstructions.

What carries the argument

A camera-initialized neural field that acts as the shared scene representation inside a unified differentiable renderer for Doppler radar outputs.

If this is right

  • Radar simulation fidelity increases when camera angular resolution supplies the missing spatial detail.
  • Cross-modal neural fields allow metric depth and weather robustness from radar to be rendered at visual resolutions.
  • New paired radar-camera datasets enable direct quantitative comparison of multimodal versus single-modality reconstruction.
  • Processing pipelines for single-chip radar can be developed and tested using higher-quality simulated inputs.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same initialization strategy could be tested on other sensor pairs where one modality offers dense angular sampling.
  • If the domain gap remains small, camera-heavy training sets might substitute for scarce labeled radar data in certain tasks.
  • Shared neural representations could support joint optimization of radar and camera processing in embedded systems.

Load-bearing premise

Camera data can initialize a neural field that accurately encodes the geometry and motion details that radar would observe, without a large mismatch between the two sensor domains.

What would settle it

A side-by-side evaluation on the collected dataset where the radar-only neural field baseline matches or exceeds the sharpness of geometry and Doppler frames produced by the camera-initialized version.

Figures

Figures reproduced from arXiv: 2605.26328 by Akarsh Prabhakara, Anthony Rowe, Chaithanya Kumar Mummadi, Chuhan Chen, Deva Ramanan, Matthew O'Toole, Tianshu Huang, Zhongxiao Cong.

Figure 1
Figure 1. Figure 1: Given synchronized measurements from a mmWave radar and RGB camera, we learn a spectral field model for rendering [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: (a) RadarSim uses volumetric radar reflectance and occupancy models to render a high resolution radar reflectance image. Importantly, it initializes (and regularizes) the radar occupancy model to be similar to a pre-trained RGB-NeRF occupancy model. (b) left: To better model radar reflectance, we repurpose classic specular reflectance models (e.g. Phong shading [16]), where the strength of the viewed specu… view at source ↗
Figure 3
Figure 3. Figure 3: Novel view synthesis for DART [23] and Radarfields [12] (middle) versus RadarSim (bottom). Since DART and Radarfields are based solely on radar data, it is limited in spatial, azimuth and elevation resolution and fine-detail. Specifically, they lack the ability to resolve reflectors at different heights because both doppler-range integration (DART) and range integration (Radarfields) still suffer from heig… view at source ↗
Figure 4
Figure 4. Figure 4: Simulating unseen antennae. RadarSim can generate novel-views with modified “intrinsics" that capture novel config￾urations of antennae. Here, we train RadarSim on the first 4 of 8 available antennae and generate renderings of the last 4, comparing them to held-out ground-truth antennae observations. We visualize 2 of the 4 unseen antennae in this figure. RadarSim’s renderings are sharper and closer to the… view at source ↗
Figure 5
Figure 5. Figure 5: We show range-azimuth reconstructions of [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Scale optimization process. Recall that our pipeline uses COLMAP to infer up-to-scale camera poses. We use radar to metrically-upgrade our scene reconstruction by optimizing for the scale that produces the best (metric) range-doppler reconstruction. The (left) plot shows the optimization curve, where the scale factor adjusts from a randomly initialized value of 1 to a final optimized value 0.483. The (righ… view at source ↗
Figure 7
Figure 7. Figure 7: RGB birds-eye view of the scene (a), radar occupancy αr slice (at 0.5m in height) (b) reconstructed by RadarSim. Reference RGB images (c) and corresponding depth map rendering using radar occupancy (d) and camera occupancy (e). Because radar transmits through materials such as plastic cardboard, foam, etc., such geometries (annotated in red) do not appear in the radar occupancy slices or depth renderings. … view at source ↗
Figure 8
Figure 8. Figure 8: A tent with (top) a person sitting inside, shown as an RGB image (left), radar reflectance rendered from radar occupancy (left-center) and camera occupancy (center), depth map rendered from radar occupancy (right-center), and camera occupancy (right). As radar can transmit through cloth, radar density reveals the presence of a person, while camera density is unable to do so. Comparison to Prior Art SSIM PS… view at source ↗
Figure 9
Figure 9. Figure 9: Ablation on our proposed BRDF bases encoding to model [PITH_FULL_IMAGE:figures/full_fig_p008_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Image of our hand-held data collection rig with three [PITH_FULL_IMAGE:figures/full_fig_p012_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Visualization of the 8 indoor and outdoor scenes we collect and evaluate on in our experiments, in lidar map [PITH_FULL_IMAGE:figures/full_fig_p013_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Architecture diagram of our framework. 3.2. Training details Following DART [23], we process radar raw data into Range￾Doppler-Azmith frames with dimension size 128, 128, 8. At each training iteration, a batch of radar Doppler columns Yr are sampled to form radar rays. Radar rays are sampled on a cone with directions determined by velocity of the sensor at the current frame, and apex angle determined by t… view at source ↗
Figure 13
Figure 13. Figure 13: Normal-dependent BRDF encoding allows RadarSim to improve reconstruction of textureless regions in a joint training setting, without using monocular-predicted normals [PITH_FULL_IMAGE:figures/full_fig_p015_13.png] view at source ↗
read the original abstract

Radars are an ideal complement to cameras: both are inexpensive, solid-state sensors, with cameras offering fine angular resolution, while radars provide metric depth and robustness under adverse weather. However, radar data is more difficult to interpret than camera images and varies significantly between sensors, necessitating increased reliance on simulation for prototyping sensors and processing pipelines. Recent work treating radar reconstruction as a novel view synthesis problem has shown great promise in reconstructing radar-relevant geometry and simulating low-level radar data. However, such methods are constrained by the low spatial resolution of the underlying radar. To address this, we propose a unified differentiable renderer, RadarSim, which leverages the high angular resolution of RGB cameras to generate Doppler radar range images from a camera-initialized neural field. Using a novel data set of calibrated radar camera recordings from a custom hand-held rig, we demonstrate that RadarSim produces sharper geometry and Doppler range frames than radar-only reconstructions.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper proposes RadarSim, a unified differentiable renderer that initializes a neural field from high-angular-resolution RGB camera data to generate simulated Doppler radar range images. It introduces a custom hand-held rig for calibrated radar-camera recordings and claims that this multimodal approach yields sharper geometry and Doppler frames than radar-only neural reconstructions.

Significance. If the central claim holds with supporting evidence, the work could improve radar simulation pipelines by exploiting camera resolution to compensate for radar's low spatial resolution, potentially aiding sensor prototyping and data augmentation in adverse conditions. The use of a new calibrated multimodal dataset is a positive contribution, but the absence of quantitative evaluation limits assessment of practical impact.

major comments (2)
  1. [Abstract] Abstract: The central demonstration is described only qualitatively ('sharper geometry and Doppler range frames') with no reported quantitative metrics, error analysis, ablation studies, or baseline comparisons. This prevents verification of the claim that the camera-initialized field improves upon radar-only methods.
  2. [Abstract] Abstract and method description: No details are provided on how the neural field encodes radial velocity for Doppler simulation, nor whether any radar measurements are used to supervise or fine-tune the field after camera initialization. This leaves the domain-gap concern unaddressed: camera data supplies passive appearance and lacks direct metric depth or velocity, while radar measures active time-of-flight and Doppler shifts.
minor comments (1)
  1. [Abstract] The abstract refers to 'a novel data set' but provides no information on its size, diversity, or calibration procedure; this should be expanded in the main text for reproducibility.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive feedback. We address each major comment below and will revise the manuscript accordingly to strengthen the presentation and address the noted gaps.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The central demonstration is described only qualitatively ('sharper geometry and Doppler range frames') with no reported quantitative metrics, error analysis, ablation studies, or baseline comparisons. This prevents verification of the claim that the camera-initialized field improves upon radar-only methods.

    Authors: We agree that the current abstract and presentation emphasize qualitative results. The manuscript body contains some visual comparisons, but we acknowledge the absence of quantitative metrics, error analysis, ablations, and explicit baselines limits verifiability. In revision we will add these elements, including PSNR/SSIM-style metrics on simulated vs. measured radar frames where ground truth is available, ablation on camera vs. radar initialization, and direct comparison to radar-only neural reconstruction baselines. The abstract will be updated to summarize the key quantitative findings. revision: yes

  2. Referee: [Abstract] Abstract and method description: No details are provided on how the neural field encodes radial velocity for Doppler simulation, nor whether any radar measurements are used to supervise or fine-tune the field after camera initialization. This leaves the domain-gap concern unaddressed: camera data supplies passive appearance and lacks direct metric depth or velocity, while radar measures active time-of-flight and Doppler shifts.

    Authors: We appreciate this observation on the domain gap. The neural field is camera-initialized for geometry and appearance but is subsequently supervised and fine-tuned using the radar measurements for both range and Doppler. Radial velocity is encoded as an additional output head of the MLP and queried along radar rays during differentiable rendering to produce the simulated Doppler spectrum. Radar time-of-flight and Doppler data provide the metric supervision that bridges the gap. We will expand the method section with explicit equations and diagrams describing the velocity encoding and the two-stage (camera init + radar supervision) training procedure. revision: yes

Circularity Check

0 steps flagged

No circularity; derivation relies on new calibrated dataset and external calibration

full rationale

The paper's core contribution is a camera-initialized neural field rendered via a differentiable renderer (RadarSim) to produce radar outputs, evaluated on a newly collected multi-modal dataset from a custom rig with stated external calibration. No equations or claims reduce by construction to fitted inputs, self-citations, or renamed known results; the demonstration compares against radar-only baselines on held-out recordings. The approach is self-contained against external benchmarks (new data + calibration) and does not invoke load-bearing self-citations or uniqueness theorems from prior author work.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review provides no details on free parameters, axioms, or invented entities; ledger is empty pending full text.

pith-pipeline@v0.9.1-grok · 5714 in / 978 out tokens · 26359 ms · 2026-06-29T22:15:28.070292+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

73 extracted references · 6 canonical work pages

  1. [1]

    Törf: Time-of-flight radiance fields for dynamic scene view synthesis.Advances in Neural Information Processing Sys- tems, 34, 2021

    Benjamin Attal, Eliot Laidlaw, Aaron Gokaslan, Changil Kim, Christian Richardt, James Tompkin, and Matthew O’Toole. Törf: Time-of-flight radiance fields for dynamic scene view synthesis.Advances in Neural Information Processing Sys- tems, 34, 2021. 5

    2021

  2. [2]

    Raysar - 3d sar simulator: Now open source

    Stefan Auer, Richard Bamler, and Peter Reinartz. Raysar - 3d sar simulator: Now open source. 2016. 2

    2016

  3. [3]

    Connect- ing nerfs images and text

    Francesco Ballerini, Pierluigi Zama Ramirez, Roberto Mirabella, Samuele Salti, and Luigi Di Stefano. Connect- ing nerfs images and text. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 866–876, 2024. 3

    2024

  4. [4]

    Barron, Ben Mildenhall, Dor Verbin, Pratul P

    Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields.CVPR, 2022. 5, 14

    2022

  5. [5]

    Radsimreal: Bridging the gap between synthetic and real data in radar object detection with simulation

    Oded Bialer and Yuval Haitman. Radsimreal: Bridging the gap between synthetic and real data in radar object detection with simulation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15407– 15416, 2024. 2

    2024

  6. [6]

    Radar fields: Frequency-space neural scene representations for fmcw radar

    David Borts, Erich Liang, Tim Broedermann, Andrea Ra- mazzina, Stefanie Walz, Edoardo Palladin, Jipeng Sun, David Brueggemann, Christos Sakaridis, Luc Van Gool, et al. Radar fields: Frequency-space neural scene representations for fmcw radar. InACM SIGGRAPH 2024 Conference Papers, pages 1–10, 2024. 2, 3, 6, 8, 12, 15, 16

    2024

  7. [7]

    Rf genesis: Zero-shot gener- alization of mmwave sensing through simulation-based data synthesis and generative diffusion models

    Xingyu Chen and Xinyu Zhang. Rf genesis: Zero-shot gener- alization of mmwave sensing through simulation-based data synthesis and generative diffusion models. InACM Confer- ence on Embedded Networked Sensor Systems (SenSys ’23), pages 1–14, Istanbul, Turkiye, 2023. ACM, New York, NY , USA. 3

    2023

  8. [8]

    C. J. Coleman. A ray tracing formulation and its application to some problems in over-the-horizon radar.Radio Science, 33(4):1187–1197, 1998. 2

    1998

  9. [9]

    Radargrammetry and sar interferometry for dem generation: validation and data fusion

    Michele Crosetto and F Pérez Aragues. Radargrammetry and sar interferometry for dem generation: validation and data fusion. InSAR workshop: CEOS committee on earth observation satellites, page 367, 2000. 2

    2000

  10. [10]

    Migration of ground- penetrating radar data method with a finite-element and dis- persion.Geophysics, 69, 2004

    Qingyun di and Miaoyue Wang. Migration of ground- penetrating radar data method with a finite-element and dis- persion.Geophysics, 69, 2004. 2

    2004

  11. [11]

    Christopher Doer and Gert F. Trommer. Yaw aided radar iner- tial odometry uisng manhattan world assumptions. In2021 28th Saint Petersburg International Conference on Integrated Navigation Systems (ICINS), pages 1–10, 2021. 2

    2021

  12. [12]

    Radar fields: An extension of radiance fields to sar

    Thibaud Ehret, Roger Marí, Dawa Derksen, Nicolas Gasnier, and Gabriele Facciolo. Radar fields: An extension of radiance fields to sar. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 564–574,

  13. [13]

    Panoptic nuscenes: A large-scale benchmark for lidar panoptic segmentation and tracking.arXiv, 2021

    Fong et al. Panoptic nuscenes: A large-scale benchmark for lidar panoptic segmentation and tracking.arXiv, 2021. 12

    2021

  14. [14]

    Raw high-definition radar for multi-task learning

    Rebut et al. Raw high-definition radar for multi-task learning. InCVPR, pages 17021–17030, 2022. 12

    2022

  15. [15]

    Fidelis, Fabio Reway, Herick Y

    Eduardo C. Fidelis, Fabio Reway, Herick Y . S. Ribeiro, Pietro L. Campos, Werner Huber, Christian Icking, Lester A. Faria, and Torsten Schön. Generation of realistic synthetic raw radar data for automated driving applications using generative adversarial networks, 2023. 3

    2023

  16. [16]

    A modern approach.Com- puter vision: a modern approach, 17:21–48, 2003

    David A Forsyth and Jean Ponce. A modern approach.Com- puter vision: a modern approach, 17:21–48, 2003. 3

    2003

  17. [17]

    Ge- owizard: Unleashing the diffusion priors for 3d geometry estimation from a single image

    Xiao Fu, Wei Yin, Mu Hu, Kaixuan Wang, Yuexin Ma, Ping Tan, Shaojie Shen, Dahua Lin, and Xiaoxiao Long. Ge- owizard: Unleashing the diffusion priors for 3d geometry estimation from a single image. InECCV, 2024. 6, 15

    2024

  18. [18]

    Furse, S.P

    C.M. Furse, S.P. Mathur, and O.P. Gandhi. Improvements to the finite-difference time-domain method for calculating the radar cross section of a perfectly conducting target.IEEE Transactions on Microwave Theory and Techniques, 38(7): 919–927, 1990. 2

    1990

  19. [19]

    ThermoNeRF: Joint RGB and thermal novel view synthesis for building facades using multimodal neural radiance fields.arXiv preprint arXiv:2403.12154,

    Mariam Hassan, Florent Forest, Olga Fink, and Mal- colm Mielle. Thermonerf: Multimodal neural radiance fields for thermal novel view synthesis.arXiv preprint arXiv:2403.12154, 2024. 3

    work page arXiv 2024

  20. [20]

    Soac: Spatio-temporal overlap- aware multi-sensor calibration using neural radiance fields

    Quentin Herau, Nathan Piasco, Moussab Bennehar, Luis Roldao, Dzmitry Tsishkou, Cyrille Migniot, Pascal Vasseur, and Cédric Demonceaux. Soac: Spatio-temporal overlap- aware multi-sensor calibration using neural radiance fields. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15131–15140, 2024. 3

    2024

  21. [21]

    A ray launching approach for modeling an fmcw radar system

    Nils Hirsenkorn, Paul Subkowski, Timo Hanke, Alexander Schaermann, Andreas Rauch, Ralph Rasshofer, and Erwin Biebl. A ray launching approach for modeling an fmcw radar system. In2017 18th International Radar Symposium (IRS), pages 1–10, 2017. 2

    2017

  22. [22]

    Neural lidar fields for novel view synthesis.arXiv preprint arXiv:2305.01643, 2023

    Shengyu Huang, Zan Gojcic, Zian Wang, Francis Williams, Yoni Kasten, Sanja Fidler, Konrad Schindler, and Or Litany. Neural lidar fields for novel view synthesis.arXiv preprint arXiv:2305.01643, 2023. 2, 3

    work page arXiv 2023

  23. [23]

    Dart: Im- plicit doppler tomography for radar novel view synthesis

    Tianshu Huang, John Miller, Akarsh Prabhakara, Tao Jin, Tarana Laroia, Zico Kolter, and Anthony Rowe. Dart: Im- plicit doppler tomography for radar novel view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 24118–24129, 2024. 2, 3, 4, 5, 6, 7, 8, 12, 14, 15, 16

    2024

  24. [24]

    PhD thesis, Linköping University Electronic Press, 2005

    Jonas Jansson.Collision Avoidance Theory: With application to automotive collision mitigation. PhD thesis, Linköping University Electronic Press, 2005. 1

    2005

  25. [25]

    Lerf: Language embedded radiance fields

    Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. Lerf: Language embedded radiance fields. InProceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 19729–19739,

  26. [26]

    Coloradar: The direct 3d millimeter wave radar dataset.The International Journal of Robotics Research, 41(4):351–360, 2022

    Andrew Kramer, Kyle Harlow, Christopher Williams, and Christoffer Heckman. Coloradar: The direct 3d millimeter wave radar dataset.The International Journal of Robotics Research, 41(4):351–360, 2022. 2, 12, 13

    2022

  27. [27]

    Multiview three- dimensional reconstruction by millimetre-wave portable cam- era.Scientific reports, 7(1):6479, 2017

    Jaime Laviada, Ana Arboleya-Arboleya, Yuri Álvarez, Borja González-Valdés, and Fernando Las-Heras. Multiview three- dimensional reconstruction by millimetre-wave portable cam- era.Scientific reports, 7(1):6479, 2017. 2, 3

    2017

  28. [28]

    Multistatic millimeter-wave imaging by multiview portable camera.IEEE Access, 5:19259–19268, 2017

    Jaime Laviada, Ana Arboleya-Arboleya, and Fernando Las- Heras. Multistatic millimeter-wave imaging by multiview portable camera.IEEE Access, 5:19259–19268, 2017

    2017

  29. [29]

    Multiview mm-wave imaging with augmented depth camera information.IEEE Access, 6:16869–16877, 2018

    Jaime Laviada, Miguel Lopez-Portugues, Ana Arboleya- Arboleya, and Fernando Las-Heras. Multiview mm-wave imaging with augmented depth camera information.IEEE Access, 6:16869–16877, 2018. 2, 3

    2018

  30. [30]

    Signal processing for tdm mimo fmcw millimeter-wave radar sensors

    Xinrong Li, Xiaodong Wang, Qing Yang, and Song Fu. Signal processing for tdm mimo fmcw millimeter-wave radar sensors. IEEE Access, 9:167959–167971, 2021. 2

    2021

  31. [31]

    Markowitz, and Minh N

    Teck-Yian Lim, Spencer A. Markowitz, and Minh N. Do. Radical: A synchronized fmcw radar, depth, imu and rgb camera data dataset with low-level fmcw radar signals.IEEE Journal of Selected Topics in Signal Processing, 15(4):941– 953, 2021. 12

    2021

  32. [32]

    Teck-Yian Lim, Spencer A Markowitz, and Minh N Do. Rad- ical: A synchronized fmcw radar, depth, imu and rgb camera data dataset with low-level fmcw radar signals.IEEE Journal of Selected Topics in Signal Processing, 15(4):941–953, 2021. 2, 13

    2021

  33. [33]

    Ranerf: Neural 3d reconstruction of space targets from isar image sequences.IEEE Transactions on Geoscience and Remote Sensing, 2023

    Afei Liu, Shuanghui Zhang, Chi Zhang, Shuaifeng Zhi, and Xiang Li. Ranerf: Neural 3d reconstruction of space targets from isar image sequences.IEEE Transactions on Geoscience and Remote Sensing, 2023. 3

    2023

  34. [34]

    Malinen and P

    M. Malinen and P. Råback.Elmer finite element solver for multiphysics and multiscale problems. Forschungszentrum Juelich, 2013. 2

    2013

  35. [35]

    60 ghz synthetic aper- ture radar for short-range imaging: Theory and experiments

    Babak Mamandipoor, Greg Malysa, Amin Arbabian, Upa- manyu Madhow, and Karam Noujeim. 60 ghz synthetic aper- ture radar for short-range imaging: Theory and experiments. In2014 48th Asilomar Conference on Signals, Systems and Computers, pages 553–558. IEEE, 2014. 3

    2014

  36. [36]

    Nerf: Representing scenes as neural radiance fields for view synthe- sis.Communications of the ACM, 65(1):99–106, 2021

    Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthe- sis.Communications of the ACM, 65(1):99–106, 2021. 2, 3, 4

    2021

  37. [37]

    Minkler and J

    G. Minkler and J. Minkler. CFAR: The principles of auto- matic radar detection in clutter.NASA STI/Recon Technical Report A, 90:23371, 1990. 2

    1990

  38. [38]

    High-resolution radar dataset for semi- supervised learning of dynamic objects

    Mohammadreza Mostajabi, Ching Ming Wang, Darsh Ranjan, and Gilbert Hsyu. High-resolution radar dataset for semi- supervised learning of dynamic objects. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2020. 3

    2020

  39. [39]

    Instant neural graphics primitives with a multireso- lution hash encoding.ACM Transactions on Graphics (ToG), 41(4):1–15, 2022

    Thomas Müller, Alex Evans, Christoph Schied, and Alexan- der Keller. Instant neural graphics primitives with a multireso- lution hash encoding.ACM Transactions on Graphics (ToG), 41(4):1–15, 2022. 5

    2022

  40. [40]

    Exploring multi-modal neural scene rep- resentations with applications on thermal imaging.arXiv preprint arXiv:2403.11865, 2024

    Mert Ozer, Maximilian Weiherer, Martin Hundhausen, and Bernhard Egger. Exploring multi-modal neural scene rep- resentations with applications on thermal imaging.arXiv preprint arXiv:2403.11865, 2024. 3

    work page arXiv 2024

  41. [41]

    K-radar: 4d radar object detection for autonomous driving in various weather conditions.Advances in Neural Information Processing Systems, 35:3819–3829, 2022

    Dong-Hee Paek, Seung-Hyun Kong, and Kevin Tirta Wijaya. K-radar: 4d radar object detection for autonomous driving in various weather conditions.Advances in Neural Information Processing Systems, 35:3819–3829, 2022. 2, 13

    2022

  42. [42]

    Osprey: a mmwave approach to tire wear sens- ing

    Akarsh Prabhakara, Vaibhav Singh, Swarun Kumar, and An- thony Rowe. Osprey: a mmwave approach to tire wear sens- ing. InProceedings of the 18th International Conference on Mobile Systems, Applications, and Services, pages 28–41,

  43. [43]

    Neural implicit surface reconstruction using imaging sonar,

    Mohamad Qadri, Michael Kaess, and Ioannis Gkioulekas. Neural implicit surface reconstruction using imaging sonar. arXiv preprint arXiv:2209.08221, 2022. 3

    work page arXiv 2022

  44. [44]

    3d point cloud generation with millimeter-wave radar.Proc

    Kun Qian, Zhaoyuan He, and Xinyu Zhang. 3d point cloud generation with millimeter-wave radar.Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., 4(4), 2020. 3

    2020

  45. [45]

    Automotive radar and lidar systems for next generation driver assistance functions

    Ralph H Rasshofer and Klaus Gresser. Automotive radar and lidar systems for next generation driver assistance functions. Advances in Radio Science, 3:205–209, 2005. 1

    2005

  46. [46]

    Raw high-definition radar for multi-task learning

    Julien Rebut, Arthur Ouaknine, Waqas Malik, and Patrick Pérez. Raw high-definition radar for multi-task learning. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 17021–17030, 2022. 2, 13

    2022

  47. [47]

    Neural vol- umetric reconstruction for coherent synthetic aperture sonar

    Albert W Reed, Juhyeon Kim, Thomas Blanford, Adithya Pediredla, Daniel C Brown, and Suren Jayasuriya. Neural vol- umetric reconstruction for coherent synthetic aperture sonar. arXiv preprint arXiv:2306.09909, 2023. 3

    work page arXiv 2023

  48. [48]

    Radar cfar thresholding in clutter and multiple target situations.IEEE Transactions on Aerospace and Electronic Systems, AES-19(4):608–621, 1983

    Hermann Rohling. Radar cfar thresholding in clutter and multiple target situations.IEEE Transactions on Aerospace and Electronic Systems, AES-19(4):608–621, 1983. 2

    1983

  49. [49]

    Virtual radar: Real-time millimeter-wave radar sensor simulation for perception-driven robotics.IEEE Robotics and Automation Letters, 6(3):4704–4711, 2021

    Christian Schöffmann, Barnaba Ubezio, Christoph Böhm, Stephan Mühlbacher-Karrer, and Hubert Zangl. Virtual radar: Real-time millimeter-wave radar sensor simulation for perception-driven robotics.IEEE Robotics and Automation Letters, 6(3):4704–4711, 2021. 2

    2021

  50. [50]

    A realistic radar ray tracing simulator for large mimo-arrays in auto- motive environments.IEEE Journal of Microwaves, 1(4): 962–974, 2021

    Christian Schüßler, Marcel Hoffmann, Johanna Bräunig, In- grid Ullmann, Randolf Ebelt, and Martin V ossiek. A realistic radar ray tracing simulator for large mimo-arrays in auto- motive environments.IEEE Journal of Microwaves, 1(4): 962–974, 2021. 2

    2021

  51. [51]

    Radar remote sensing of agricul- tural canopies: A review.IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 10(5): 2249–2273, 2017

    Susan C Steele-Dunne, Heather McNairn, Alejandro Monsivais-Huertero, Jasmeet Judge, Pang-Wei Liu, and Kostas Papathanassiou. Radar remote sensing of agricul- tural canopies: A review.IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 10(5): 2249–2273, 2017. 1

    2017

  52. [52]

    Nerfstudio: A modular framework for neural radiance field development

    Matthew Tancik, Ethan Weber, Evonne Ng, Ruilong Li, Brent Yi, Justin Kerr, Terrance Wang, Alexander Kristoffersen, Jake Austin, Kamyar Salahi, Abhik Ahuja, David McAllister, and Angjoo Kanazawa. Nerfstudio: A modular framework for neural radiance field development. InACM SIGGRAPH 2023 Conference Proceedings, 2023. 5, 15

    2023

  53. [53]

    Nerfstudio: A modular frame- work for neural radiance field development.arXiv preprint arXiv:2302.04264, 2023

    Matthew Tancik, Ethan Weber, Evonne Ng, Ruilong Li, Brent Yi, Justin Kerr, Terrance Wang, Alexander Kristoffersen, Jake Austin, Kamyar Salahi, et al. Nerfstudio: A modular frame- work for neural radiance field development.arXiv preprint arXiv:2302.04264, 2023. 7

    work page arXiv 2023

  54. [54]

    Alignmif: Geometry-aligned multimodal implicit field for lidar-camera joint synthesis

    Tang Tao, Guangrun Wang, Yixing Lao, Peng Chen, Jie Liu, Liang Lin, Kaicheng Yu, and Xiaodan Liang. Alignmif: Geometry-aligned multimodal implicit field for lidar-camera joint synthesis. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21230– 21240, 2024. 3

    2024

  55. [55]

    Teixeira, Weng Cho Chew, M

    F.L. Teixeira, Weng Cho Chew, M. Straka, M.L. Oristaglio, and T. Wang. Finite-difference time-domain simulation of ground penetrating radar on dispersive, inhomogeneous, and conductive soils.IEEE Transactions on Geoscience and Re- mote Sensing, 36(6):1928–1937, 1998. 2

    1928

  56. [56]

    Ref-nerf: Struc- tured view-dependent appearance for neural radiance fields

    Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan T Barron, and Pratul P Srinivasan. Ref-nerf: Struc- tured view-dependent appearance for neural radiance fields. In2022 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 5481–5490. IEEE, 2022. 4, 15

    2022

  57. [57]

    Automotive radar—from first efforts to future systems

    Christian Waldschmidt, Juergen Hasch, and Wolfgang Men- zel. Automotive radar—from first efforts to future systems. IEEE Journal of Microwaves, 1(1):135–148, 2021. 1

    2021

  58. [58]

    Gesture recognition for smart home applications using portable radar sensors

    Qian Wan, Yiran Li, Changzhi Li, and Ranadip Pal. Gesture recognition for smart home applications using portable radar sensors. In2014 36th annual international conference of the IEEE engineering in medicine and biology society, pages 6414–6417. IEEE, 2014. 1

    2014

  59. [59]

    High-resolution 2d sar imaging by the millimeter-wave automobile radar

    Hiroyoshi Yamada, Takumi Kobayashi, Yoshio Yamaguchi, and Yuuichi Sugiyama. High-resolution 2d sar imaging by the millimeter-wave automobile radar. In2017 IEEE Conference on Antenna Measurements & Applications (CAMA), pages 149–150. IEEE, 2017. 3

    2017

  60. [60]

    Near-field mimo- sar millimeter-wave imaging with sparsely sampled aperture data.Ieee Access, 7:31801–31819, 2019

    Muhammet Emin Yanik and Murat Torlak. Near-field mimo- sar millimeter-wave imaging with sparsely sampled aperture data.Ieee Access, 7:31801–31819, 2019. 3

    2019

  61. [61]

    Rad- det: Range-azimuth-doppler based radar object detection for dynamic road users

    Ao Zhang, Farzan Erlik Nowruzi, and Robert Laganiere. Rad- det: Range-azimuth-doppler based radar object detection for dynamic road users. In2021 18th Conference on Robots and Vision (CRV), pages 95–102. IEEE, 2021. 2, 13

    2021

  62. [62]

    Nerf2: Neural radio-frequency radiance fields

    Xiaopeng Zhao, Zhenlin An, Qingrui Pan, and Lei Yang. Nerf2: Neural radio-frequency radiance fields. InProceed- ings of the 29th Annual International Conference on Mobile Computing and Networking, pages 1–15, 2023. 3

    2023

  63. [63]

    Multimodal neural radiance field

    Haidong Zhu, Yuyin Sun, Chi Liu, Lu Xia, Jiajia Luo, Nan Qiao, Ram Nevatia, and Cheng-Hao Kuo. Multimodal neural radiance field. In2023 IEEE International Conference on Robotics and Automation (ICRA), pages 9393–9399. IEEE,

  64. [64]

    Supplementary Videos Please refer to the attached supplementary videos for re- sults. We show comparison between RadarSim- Radar reflectance rendered with radar occupancy (left column) and radar-only baselines DART [23](middle column) and Radarfields[6](right column). Observe that compared to the baseline, our reconstruction quality shown in depth map at ...

  65. [65]

    Image of our hand-held data collection rig with three key components labeled

    Dataset Figure 10. Image of our hand-held data collection rig with three key components labeled. 2.1. Data capture rig and pose processing We build a hand-held rig for data collection with time-synced mmWave radar, fisheye camera, and lidar collecting data at 30 fps, 30 fps and 10 fps respectively. For each sequence, we run COLMAP to get camera poses Ac (...

  66. [66]

    L2 regularization on the offsets: Lregp =||∆x c||2 2 + ||∆Rc||2 2 +||∆v c||2 2

  67. [67]

    L2 loss that enforces velocity to be close to derivative of positionL regv =||d(x c)/dt−(v c + ∆vc)||2 2

  68. [68]

    L2 regularization on acceleration Lrega =||d(∆v c + vc)/dt||2 2

  69. [69]

    Such pose optimization scheme is shared across camera and radar as radar pose and velocity can be interpolated from camera poses and velocities

    kinematic loss: Lregk =||d window(xc + ∆x c)− dwindow( R ∆vc +v cdt)||2 2 Note that it is possible to derive optimized velocity from optimized positions (x+ ∆x ), but we empirically found it to be more stable to keep a different set of velocity offset and position offset and have them loosely connected through regularization. Such pose optimization scheme...

    2000

  70. [70]

    Coloradar [26] does capture indoor senes that can be used for reconstruction, but uses high-resolution radar which is also beyond our sin-radar fo- cus

    also uses Cascaded Radar with a complex modulation function, which makes modelling radar rendering equation difficult for inverse rendering. Coloradar [26] does capture indoor senes that can be used for reconstruction, but uses high-resolution radar which is also beyond our sin-radar fo- cus. Most public RGB-Radar datasets, such as NuScenes [13], provide ...

  71. [71]

    See figure 12 for detailed architecture diagram

    Method details We elaborate here on technical details omitted in the main paper. See figure 12 for detailed architecture diagram. 3.1. Modelling radar view dependence with BRDF bases View dependence of Radar reflectance can be broken into 2 scenarios, surface normal dependent reflectance and sur- face normal independent reflectance. The latter includes re...

  72. [72]

    We use λr = 1e −3 for outdoor scenes and λr = 1e−4 for indoor scenes where abundance of multi-path reflections result in overall high reflectance in the scenes

    to guide the predicted normal with gradient direction of the camera density field, and encourage normals to point outward from a surface. We use λr = 1e −3 for outdoor scenes and λr = 1e−4 for indoor scenes where abundance of multi-path reflections result in overall high reflectance in the scenes. We choose λbce = 0.01 and λssim = 0.01. We choose λnorm = ...

  73. [73]

    Evaluation metric and denoising procedure We calculate PSNR and SSIM values between RadarSim and ground truth for evaluation and com- parison against baselines

    Experiment details and additional result 4.1. Evaluation metric and denoising procedure We calculate PSNR and SSIM values between RadarSim and ground truth for evaluation and com- parison against baselines. Because most of the radar frame consists of noise, we design a denoising procedure by finding the noise threshold for each dataset. The noise threshol...

    2048