REVIEW 3 major objections 5 minor 35 references
Comparing Normalizing Flows with Kernel Density Estimation in Estimating Risk of Automated Driving Systems
T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Normalizing flows fit driving-scenario densities better than kernel density estimation, but the two fitted densities disagree by roughly 40-fold on collision risk.
desk verdict A careful density-modeling comparison whose risk-estimation conclusion is not supported by the paper's own tail evidence. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The risk pipeline is carried by importance sampling: a first Monte Carlo stage samples from the estimated density $p_x$, the $N_C$ most critical runs, selected by time-to-collision, seed a KDE-based importance density $p_x^*$, and a second stage estimates the collision probability $E[R(x)]$ with weighted samples $R(x)p_x(x)/p_x^*(x)$. The density estimates are the other half of the machinery: normalizing flows use the change-of-variables formula $$p_x(x)=p_z(f_\$theta^{{-1}}$(x))\,|\det \partial f_\$theta^{{-1}}$(x)/\partial x|$$ with four coupling layers, each a Masked Autoregressive Flow with batch normalization and a fixed random permutation, over a standard Gaussian base; KDE uses Gaussian kernels with leave-one-out cross-validated bandwidth. The case study evaluates both on four cut-in scenario parameters extracted from naturalistic highway trajectories.
What would settle it
Run the risk pipeline with an importance density built from each estimator's own density rather than from KDE alone, and report the effective sample size and the distribution of the importance weights in (3); if the KDE-versus-flow gap shrinks when the importance density is matched to each estimator, the discrepancy is an importance-sampling artifact, and if it persists, it reflects genuine differences in tail mass between the two density models.
Extended reading notes
Core claim
The paper's central claim is that normalizing flows, a generative model built from invertible differentiable maps, estimate the scenario-parameter density more accurately than KDE when judged by mean log-likelihood on held-out cut-in scenarios, and that this advantage grows with dataset size. At the extremes, however, KDE is as good or better once enough data are used, and with small samples KDE is better. When the estimated densities are used to estimate collision probability via importance sampling, KDE yields a median estimate roughly 40 times higher than normalizing flows; the paper explains this gap by KDE's tendency to overestimate density in the tails, where the simulated collisions lie. The paper does not claim to know the true collision probability, only that the two estimators disagree substantially and that the difference traces to tail behavior.
Load-bearing premise
The load-bearing premise is that the KDE-based importance density has positive mass everywhere the integrand $R(x)p_x(x)$ is non-negligible, for both the flow and the KDE estimate of $p_x$; if the flow places almost no mass in the collision tail where the importance density has mass, the 40-fold gap could be an artifact of the estimator weights instead of a real difference in collision probability.
Editorial extensions
If this is right
- Scenario exposure models can avoid parametric independence assumptions, improving the fidelity of scenario-based safety validation for automated driving systems.
- Risk estimates for rare collision events depend strongly on the tail behavior of the density estimator, so choosing KDE versus normalizing flows can change safety conclusions by an order of magnitude.
- Training a normalizing flow is roughly 60 times slower than fitting the KDE in the tested case, but flows are expected to scale better to higher-dimensional scenario parameterizations.
- The KDE-based importance density remains usable even when the target density is a flow, because KDE's conservative tails reduce the risk of importance weights near zero, at the cost of slower convergence.
- With small datasets, KDE may be preferable for extreme-value regions, whereas flows gain the advantage as more scenario data become available.
Reading between the lines
- A synthetic benchmark with a known heavy-tailed ground-truth density would separate estimator error from estimator design; the paper's real data has no ground truth, so the 40-fold gap remains a claim about the two models rather than about the true risk.
- The paper does not report effective sample size or weight diagnostics for the importance sampling; without these, part of the gap could be an estimator artifact, and a matched importance density for each estimator would test this directly.
- Because KDE's tail inflation is expected to worsen with dimension, the flow's relative advantage in higher-dimensional scenario parameterizations may be larger than the four-parameter case shows, but its tail behavior in those settings is also less predictable.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript compares normalizing flows (NF) and kernel density estimation (KDE) for estimating the distribution of scenario parameters and the downstream collision risk of an automated driving system in cut-in scenarios. Using 2916 cut-in scenarios extracted from the HighD dataset, the authors fit KDE and NF on varying fractions of the data with 50 repetitions, evaluate held-out mean log-likelihood both on the full held-out set and on Pareto-front extremes, and estimate the collision probability via importance sampling with a KDE-based importance density. The results show that NF mostly outperform KDE in overall held-out log-likelihood, that KDE is equal to or better than NF at the Pareto-front extremes for smaller data fractions, and that KDE-based risk estimates are on average roughly 40 times higher than NF-based estimates. The authors conclude that NF outperform KDE in estimating the scenario-parameter PDF and that KDE tends to overestimate probability density in the tails, which leads to higher risk estimates.
Significance. The paper provides a clean repeated held-out comparison of NF and KDE for scenario-parameter exposure estimation, with 50 repetitions and IQR reporting, and applies both to a concrete cut-in scenario case study. The bulk-density comparison in Fig. 2 is a solid contribution and supports the claim that NF fit the typical region of the data better than KDE. However, the risk-quantification comparison lacks a ground truth, and the paper's own tail-likelihood evidence in Fig. 3 does not support the explanation that KDE overestimates the tails. The risk comparison is therefore best read as an observation that the two estimators diverge substantially, not as a demonstrated accuracy advantage of NF. The paper is still a useful initial study of NF in scenario-based safety assessment, provided the conclusions are tempered.
major comments (3)
- [Section III-C, Eq. (3), Fig. 4] The roughly 40-fold difference in estimated collision probability is not validated because no diagnostic for the importance-sampling estimator is reported. In Eq. (3), the estimator is unbiased only if the importance density p* has positive mass wherever R(x)p_x(x) is non-negligible, and its variance is controlled by the weights p_x/p*. If the NF density places much less mass in the collision-relevant tail than the KDE-based importance density, the weights can be very small in exactly the region being sampled, and the finite sum with N_NIS = 10,000 could be dominated by a few draws. Please report effective sample sizes (or equivalent diagnostics) for both estimators, for each data fraction and repetition, and show that the NF result is stable with respect to the chosen N_NIS.
- [Section III-B, Fig. 3 vs Section V] The conclusion that 'KDE tends to overestimate probability density in the tails' is not supported by the paper's own tail-likelihood comparison. Fig. 3 shows that at the Pareto front KDE is equal to or better than NF for most data fractions, and significantly better when 40% or less data are used; this is the opposite of an overestimation of tail density. The held-out tail-likelihood evidence therefore cannot explain the 40x gap in Fig. 4, and the alternative that NF underweights the far tail [31] remains open. Please either add a direct comparison of the estimated densities in collision-relevant tail regions or revise the conclusion to present KDE tail overestimation as one unverified hypothesis among several.
- [Section IV and Section V] The paper correctly acknowledges in Section III-C that no ground truth is available for the collision probability, but this caveat does not carry through to the conclusions. The abstract and Section V present NF as improving risk uncertainty estimation and KDE as overestimating tails, whereas the experiments only demonstrate that the two density estimators yield substantially different risk estimates. Please either add a validation target with known tail behavior and known collision probability, or revise the conclusions to clearly separate the supported PDF-fitting claim from the unsupported risk-accuracy claim.
minor comments (5)
- [Abstract] The phrase 'a traditional methods for non-parametric PDF estimation' should read 'traditional methods'.
- [Section II-B] The text 'We opted to use this type of transformer' appears to intend 'transformation'; the same wording is used later and should be made consistent.
- [Fig. 4 caption] The caption says 'Estimated collision risk' while the axis label and text refer to 'estimated collision probability'; please make the terminology consistent.
- [Section III-C] The sentence 'Both results are not unexpected' is awkward and should be rephrased.
- [Section IV] The claim that using KDE for the importance density exhibits a 'lower risk of introducing bias' is asserted without supporting analysis; add a reference or a quantitative justification.
Circularity Check
No significant circularity: the held-out likelihood comparison and the shared importance-sampling estimator keep the NF-versus-KDE result independent of any fitted input.
full rationale
The derivation chain is not circular. Research question 1 is answered by fitting NF and KDE on a training fraction and evaluating mean log-likelihood on the held-out test set (Eq. 6; Section III-B), so the claim that NF outperform KDE is an out-of-sample comparison rather than an identity. Research question 2 applies the same importance-sampling estimator (Eq. 3) with a common KDE-based proposal density p* constructed from the most critical scenarios; neither NF nor KDE is defined in terms of the risk estimate, and the weights are evaluated at the estimated densities, so the comparison is not forced by construction. The paper explicitly flags in Section III-C that 'no ground truth is available, we cannot argue that one approach is better,' which limits the risk conclusion but is a validity limitation, not a circular step. Self-citations [7], [8], [29], [30] supply the risk-quantification framework, a bandwidth heuristic, and scenario-extraction procedures; they are used as tools and do not define the reported NF-versus-KDE outcome. The absence of effective-sample-size diagnostics is an estimator-variance concern, not evidence of circularity. No equation reduces to its own inputs and no fitted parameter is relabeled as a prediction.
Assumptions & free parameters
free parameters (4)
- KDE bandwidth h =
selected by leave-one-out cross-validation per experiment
- NF training hyperparameters =
4 coupling layers, 2d hidden features, 20% dropout, Adam, max 5000 iterations with patience 100 (learning rate not…
- Number of critical scenarios NC for importance density =
not specified in text
- Simulation counts NMC and NNIS =
10,000 each
assumptions (7)
- standard math Change of variables formula (Eq. 4) for normalizing flows
- standard math Importance sampling unbiasedness (Eq. 3)
- domain assumption The Fuzzy Safety Model (FSM) is a valid benchmark driver model for ADS rear-end collision risk
- domain assumption The four chosen parameters fully describe cut-in scenario exposure
- domain assumption HighD dataset is representative of real-world German highway traffic
- domain assumption Time-to-collision is an appropriate criticality metric for selecting important scenarios
- domain assumption Collisions are rare and occur at the tails of the parameter distribution
Cite this review
Pith. "Pith review of Comparing Normalizing Flows with Kernel Density Estimation in Estimating Risk of Automated Driving Systems." pith.science (2026). https://pith.science/paper/WP7LCJLR
@misc{pith2026250722429,
author = {Pith},
title = {Pith review of: Comparing Normalizing Flows with Kernel Density Estimation in Estimating Risk of Automated Driving Systems},
year = {2026},
howpublished = {\url{https://pith.science/paper/WP7LCJLR}},
note = {Machine review of arXiv:2507.22429}
}
read the original abstract
The development of safety validation methods is essential for the safe deployment and operation of Automated Driving Systems (ADSs). One of the goals of safety validation is to prospectively evaluate the risk of an ADS dealing with real-world traffic. Scenario-based assessment is a widely-used approach, where test cases are derived from real-world driving data. To allow for a quantitative analysis of the system performance, the exposure of the scenarios must be accurately estimated. The exposure of scenarios at parameter level is expressed using a Probability Density Function (PDF). However, assumptions about the PDF, such as parameter independence, can introduce errors, while avoiding assumptions often leads to oversimplified models with limited parameters to mitigate the curse of dimensionality. This paper considers the use of Normalizing Flows (NF) for estimating the PDF of the parameters. NF are a class of generative models that transform a simple base distribution into a complex one using a sequence of invertible and differentiable mappings, enabling flexible, high-dimensional density estimation without restrictive assumptions on the PDF's shape. We demonstrate the effectiveness of NF in quantifying risk and risk uncertainty of an ADS, comparing its performance with Kernel Density Estimation (KDE), a traditional method for non-parametric PDF estimation. While NF require more computational resources compared to KDE, NF is less sensitive to the curse of dimensionality. As a result, NF can improve risk uncertainty estimation, offering a more precise assessment of an ADS's safety. This work illustrates the potential of NF in scenario-based safety. Future work involves experimenting more with using NF for scenario generation and optimizing the NF architecture, transformation types, and training hyperparameters to further enhance their applicability.
Figures
Reference graph
Works this paper leans on
-
[31]
Flexible tails for normalizing flows,
T. Hickling and D. Prangle, “Flexible tails for normalizing flows,” arXiv preprint arXiv:2406.16971, 2024
arXiv 2024
-
[1]
Advancements, prospects, and impacts of automated driving systems,
C.-Y . Chan, “Advancements, prospects, and impacts of automated driving systems,” International Journal of Transportation Science and Technology, vol. 6, no. 3, pp. 208–216, 2017
work page 2017
-
[2]
Driving to safety: How many miles of driving would it take to demonstrate autonomous vehicle reliability?
N. Kalra and S. M. Paddock, “Driving to safety: How many miles of driving would it take to demonstrate autonomous vehicle reliability?” Transportation Research Part A: Policy and Practice , vol. 94, pp. 182– 193, 2016
2016
-
[3]
Survey on scenario-based safety assessment of automated vehicles,
S. Riedmaier, T. Ponn, D. Ludwig, B. Schick, and F. Diermeyer, “Survey on scenario-based safety assessment of automated vehicles,” IEEE Access, vol. 8, pp. 87 456–87 477, 2020
work page 2020
-
[4]
Scenario-based safety assessment of automated driving systems,
E. de Gelder, O. Op den Camp, J. Broos, J.-P. Paardekooper, S. van Montfort, S. Kalisvaart, and H. Goossens, “Scenario-based safety assessment of automated driving systems,” TNO, Tech. Rep., 2024. [Online]. Available: https://www.tno.nl/en/newsroom/papers/scenario- based-safety-assessment/
work page 2024
-
[5]
Remarks on some nonparametric estimates of a density function,
M. Rosenblatt, “Remarks on some nonparametric estimates of a density function,” The Annals of Mathematical Statistics, vol. 27, no. 3, pp. 832– 837, 1956
work page 1956
-
[6]
On estimation of a probability density function and mode,
E. Parzen, “On estimation of a probability density function and mode,” The Annals of Mathematical Statistics , vol. 33, no. 3, pp. 1065–1076, 1962
work page 1962
-
[7]
Risk quantification for automated driving systems in real-world driving scenarios,
E. de Gelder, H. Elrofai, A. Khabbaz Saberi, O. Op den Camp, J.-P. Paardekooper, and B. De Schutter, “Risk quantification for automated driving systems in real-world driving scenarios,” IEEE Access , vol. 9, pp. 168 953–168 970, 2021
work page 2021
Show all 35 references
-
[8]
How certain are we that our automated driving system is safe?
E. de Gelder and O. Op den Camp, “How certain are we that our automated driving system is safe?” Traffic Injury Prevention , vol. 24, no. sup1, S131–S140, 2023
2023
-
[9]
Normalizing flows for probabilistic modeling and inference,
G. Papamakarios, E. Nalisnick, D. J. Rezende, S. Mohamed, and B. Lakshminarayanan, “Normalizing flows for probabilistic modeling and inference,” Journal of Machine Learning Research , vol. 22, no. 57, pp. 1–64, 2021
2021
-
[10]
A taxonomy of validation strategies to ensure the safe operation of highly automated vehicles,
F. Batsch, S. Kanarachos, M. Cheah, R. Ponticelli, and M. Blundell, “A taxonomy of validation strategies to ensure the safe operation of highly automated vehicles,” Journal of Intelligent Transportation Systems, pp. 1–20, 2020
2020
-
[11]
A survey on data-driven scenario generation for automated vehicle testing,
J. Cai, W. Deng, H. Guang, Y . Wang, J. Li, and J. Ding, “A survey on data-driven scenario generation for automated vehicle testing,” Ma- chines, vol. 10, no. 11, p. 1101, 2022
2022
-
[12]
1001 ways of scenario generation for testing of self-driving cars: A survey,
B. Schütt, J. Ransiek, T. Braun, and E. Sax, “1001 ways of scenario generation for testing of self-driving cars: A survey,” in IEEE Intelligent Vehicles Symposium (IV), 2023, pp. 1–8
2023
-
[13]
A survey on safety-critical driving scenario generation — a methodological perspec- tive,
W. Ding, C. Xu, M. Arief, H. Lin, B. Li, and D. Zhao, “A survey on safety-critical driving scenario generation — a methodological perspec- tive,” IEEE Transactions on Intelligent Transportation Systems , 2023
2023
-
[14]
A review on scenario generation for testing autonomous vehicles,
J. Cai, S. Yang, and H. Guang, “A review on scenario generation for testing autonomous vehicles,” in IEEE Intelligent Vehicles Symposium (IV), 2024, pp. 3371–3376
2024
-
[15]
A comprehensive literature review on artificial dataset generation for repositioning challenges in shared electric automated and connected mobility,
A. K. Kayisu, W. V . Kambale, T. Benarbia, P. N. Bokoro, and K. Kyamakya, “A comprehensive literature review on artificial dataset generation for repositioning challenges in shared electric automated and connected mobility,” Symmetry, vol. 16, no. 1, p. 128, 2024
2024
-
[16]
Road Vehicles – Functional Safety,
ISO 26262, “Road Vehicles – Functional Safety,” International Organi- zation for Standardization, Standard, 2018. [Online]. Available: https: //www.iso.org/standard/68383.html
2018
-
[17]
Comparative assessment of safety indicators for vehicle trajectories on highways,
F. A. Mullakkal-Babu, M. Wang, H. Farah, B. van Arem, and R. Happee, “Comparative assessment of safety indicators for vehicle trajectories on highways,” Transportation Research Record, vol. 2659, no. 1, pp. 127– 136, 2017
2017
-
[18]
Variational inference with normaliz- ing flows,
D. Rezende and S. Mohamed, “Variational inference with normaliz- ing flows,” in International Conference on Machine Learning , 2015, pp. 1530–1538. [Online]. Available: https://proceedings.mlr.press/v37/ rezende15.pdf
2015
-
[19]
Masked autoregressive flow for density estimation,
G. Papamakarios, T. Pavlakou, and I. Murray, “Masked autoregressive flow for density estimation,” Advances in Neural Information Processing Systems, vol. 30, pp. 1–10, 2017
2017
-
[20]
Identity mappings in deep residual networks,
K. He, X. Zhang, S. Ren, and J. Sun, “Identity mappings in deep residual networks,” in Computer Vision – ECCV , 2016, pp. 630–645
2016
-
[21]
How does batch normalization help optimization?
S. Santurkar, D. Tsipras, A. Ilyas, and A. Madry, “How does batch normalization help optimization?” In 32nd International Conference on Neural Information Processing Systems , 2018, pp. 2488–2498
2018
-
[22]
NICE: Non-linear independent components estimation,
L. Dinh, D. Krueger, and Y . Bengio, “NICE: Non-linear independent components estimation,” arXiv preprint arXiv:1410.8516 , 2014
2014 arXiv
-
[23]
Adam: A method for stochastic optimization,
D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in 3rd International Conference on Learning Representations (ICLR) , 2015
2015
-
[24]
Bandwidth selection in kernel density estimation: A review,
B. A. Turlach, “Bandwidth selection in kernel density estimation: A review,” Institut für Statistik und Ökonometrie, Humboldt-Universität zu Berlin, Tech. Rep., 1993. [Online]. Available: https://www.researchgate. net/publication/2316108_Bandwidth_Selection_in_Kernel_Density_ ...
1993
-
[25]
B. W. Silverman, Density Estimation for Statistics and Data Analysis . CRC press, 1986
1986
-
[26]
Driver models for the definition of safety requirements of automated vehicles in international regulations. applica- tion to motorway driving conditions,
K. Mattas, G. Albano, R. Donà, M. C. Galassi, R. Suarez-Bertoa, S. Vass, and B. Ciuffo, “Driver models for the definition of safety requirements of automated vehicles in international regulations. applica- tion to motorway driving conditions,” Accident Analysis & Prevention , ...
2022
-
[27]
Fuzzy surrogate safety metrics for real-time assessment of rear-end collision risk. a study based on empirical observations,
K. Mattas, M. Makridis, G. Botzoris, A. Kriston, F. Minarini, B. adopoulos, F. Re, G. Rognelund, and B. Ciuffo, “Fuzzy surrogate safety metrics for real-time assessment of rear-end collision risk. a study based on empirical observations,” Accident Analysis & Prevention , vol. ...
2020
-
[28]
The highD dataset: A drone dataset of naturalistic vehicle trajectories on German highways for validation of highly automated driving systems,
R. Krajewski, J. Bock, L. Kloeker, and L. Eckstein, “The highD dataset: A drone dataset of naturalistic vehicle trajectories on German highways for validation of highly automated driving systems,” in IEEE 21st International Conference on Intelligent Transportations Systems (IT...
2018
-
[29]
Real-world scenario mining for the assessment of automated vehicles,
E. de Gelder, J. Manders, C. Grappiolo, J.-P. Paardekooper, O. Op den Camp, and B. De Schutter, “Real-world scenario mining for the assessment of automated vehicles,” inIEEE International Transportation Systems Conference (ITSC) , 2020, pp. 1073–1080
2020
-
[30]
Scenario-based assessment of automated driving systems: How (not) to parameterize scenarios?
E. de Gelder and O. Op den Camp, “Scenario-based assessment of automated driving systems: How (not) to parameterize scenarios?” In IEEE International Automated Vehicle Validation Conference , 2024
2024
-
[32]
Tail density estimation for exploratory data analysis using kernel methods,
B. Béranger, T. Duong, S. Perkins-Kirkpatrick, and S. Sisson, “Tail density estimation for exploratory data analysis using kernel methods,” Journal of Nonparametric Statistics , vol. 31, no. 1, pp. 144–174, 2019
2019
-
[33]
D. W. Scott, Multivariate Density Estimation: Theory, Practice, and Visualization. John Wiley & Sons, 2015
2015
-
[34]
Tutorial on variational autoencoders,
C. Doersch, “Tutorial on variational autoencoders,” arXiv preprint arXiv:1606.05908, 2016
2016 arXiv
-
[35]
Diffusion density estimators,
A. Premkumar, “Diffusion density estimators,” arXiv preprint arXiv:2410.06986, 2024
2024 arXiv
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.