Pith. sign in

REVIEW 3 major objections 4 minor 58 references

OpenCat: Improving Interoperability of ADS Testing

T0 review · 3 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read OpenCat converts all 32,580 SensoDat roads from OpenDRIVE to Catmull-Rom splines and reports that re-running them with an independent lane-keeping model lifts the pass rate from 61% to 98%, evidence that the original benchmark's failures…

desk verdict OpenCat is a useful converter and the cross-simulator pass-rate gap is real, but the 100% accuracy claim is tautological and the central attribution to benchmark coupling is not yet supported. read the letter →

arxiv 2502.07719 v2 pith:UFQJHVLH submitted 2025-02-11 cs.SE

Qurban Ali , Andrea Stocco , Leonardo Mariani , Oliviero Riganelli This is my paper · ORCID
classification cs.SE
keywords OpenDRIVECatmull-RomsplineADAStestingSensoDatbenchmarklane-keepingregressioninteroperabilityautonomousdrivingsimulation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that ADAS regression-testing benchmarks are too tightly coupled to the specific simulator and driving model used to build them. To test that claim, the authors introduce OpenCat, a converter that translates OpenDRIVE road geometry into Catmull-Rom splines, the format used by the Udacity simulator. Applying OpenCat to all 32,580 roads in the SensoDat benchmark and re-running them with a different lane-keeping model (Dave-2) yields pass rates 25 to 50 percentage points higher than SensoDat's original runs. The paper takes this gap as evidence that SensoDat's original pass/fail labels reflect its original ADAS and simulator rather than road difficulty, and concludes that reusable benchmarks should be architecture-agnostic.

What carries the argument

The central mechanism is the OpenCat conversion pipeline, built on the Catmull-Rom spline, a cubic Hermite interpolation curve that passes exactly through its control points. Algorithm 1 extracts road geometry (positions, elevations, lane widths) from OpenDRIVE's XML; Algorithm 2 computes the road centerline as the average of right and left lane points and uses those averaged points as control points for the spline. Because the spline interpolates those control points, comparing the generated spline to the same points yields the reported 100% accuracy and R² = 1; the spline then provides a smooth, local-control road representation that Udacity's test generator can consume.

What would settle it

Compute the maximum distance between the converted spline and the original OpenDRIVE reference line at finely spaced parameter values that were not used as control points; if the maximum deviation is non-negligible, the 'identical geometry' premise used to interpret the pass-rate gap is false.

Watch

Extended reading notes

Core claim

The authors' central discovery is that converting SensoDat's OpenDRIVE roads into Catmull-Rom splines and simulating them with an independent ADAS (Dave-2) in the Udacity simulator reverses most of the benchmark's failures: 32,035 of 32,580 converted roads pass (98%), against 19,926 of 32,580 (61%) in the original dataset. They report that OpenCat achieves 100% conversion accuracy (R² = 1) across all roads. From this they conclude that the SensoDat labels are an artifact of the specific ADAS model and simulator it was built with, and that benchmark scenarios defined without reference to a specific model would better support regression testing.

Load-bearing premise

The load-bearing premise is that a spline that hits the sampled control points exactly is geometrically identical to the original road between those points, and that the pass-rate gap can be attributed to ADAS/simulator coupling rather than to the different driving model or simulator.

Editorial extensions

If this is right

  • If SensoDat's failure labels do not transfer, then any regression-testing conclusion drawn from its original pass/fail distribution must be re-examined before being used to compare ADAS models.
  • The 25-50 point pass-rate improvement across all campaign types implies that the original benchmark's difficult roads are not inherently difficult for an independent lane-keeper; difficulty is relative to the ADAS/simulator pair.
  • OpenCat itself is a reusable artifact: any OpenDRIVE road set can be converted to Catmull-Rom splines, broadening the set of simulators and test generators that can consume it.
  • Benchmarks should be defined in an architecture-agnostic road representation, with pass/fail criteria stated independently of a particular model.
  • The overall pass-rate shift from 61% to 98% means the original 12,654 failures are mostly not road-geometry failures under a different, learning-based controller.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • OpenCat's reported 100% accuracy and R²=1 follow from the interpolating property of Catmull-Rom splines evaluated at their own control points; a stricter test would compare against independent samples of the reference line between control points, which the paper does not report.
  • The pass-rate gap could reflect differences between BeamNG's physics and Udacity's, plus the road-validity filter (bounding box, no intersections) and the specific OOB definition, not only ADAS-model coupling; decomposing these requires running crossed experiments the paper leaves to future work.
  • A testable extension would be to convert the same roads into another format (e.g., nuScenes or a second spline library) and check whether the pass-rate shift persists, which would separate format effects from simulator effects.
  • If the claim holds generally, then existing benchmarks built for one ADAS may systematically under-report the capabilities of other models, which would change how the community evaluates regression suites.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces OpenCat, a tool that converts OpenDRIVE road definitions (used by SensoDat and BeamNG) into Catmull-Rom splines (used by Udacity). Applying OpenCat to all 32,580 SensoDat roads, the authors report a perfect conversion accuracy (100% and R²=1). They then re-simulate the converted roads in the Udacity simulator with the Dave-2 lane-keeping model and report pass rates of 98% overall versus 61% for the original SensoDat labels, with per-campaign differences of 25–50 percentage points. From this, the paper concludes that SensoDat, being coupled to a specific ADAS model and simulator, has limited value as a regression-testing benchmark, and it argues for architecture-agnostic benchmarks.

Significance. If the conversion fidelity and the comparative pass-rate analysis were properly established, the paper would address a relevant interoperability problem in ADAS testing: enabling scenarios developed in one format/simulator to be reused in another. The open-source converter, the large-scale application to a publicly available benchmark, and the dual-OS execution are concrete practical contributions. However, the current evidence is not yet convincing: the accuracy metric is self-referential, and the comparison of pass rates mixes changes in road format, simulator, driving model, and validity filters. These issues are fixable, and the work has potential, but the central conclusions are currently under-supported.

major comments (3)
  1. [Section V-A (Eq. 3 and Eq. 4), with Algorithm 2] The accuracy evaluation is circular by construction. Algorithm 2 computes control_points by averaging right and left lane points extracted from the OpenDRIVE geometry, then builds a Catmull-Rom spline through those exact control points. Since Catmull-Rom is an interpolating spline, it passes precisely through every control point, so comparing spline points with those same original points cannot yield anything other than accuracy = 100% and R² = 1. These results are properties of the interpolation scheme, not evidence that the converted road matches the OpenDRIVE geometry between control points. To support the premise that "the converted geometry is identical to the original" (Section V-B), the authors must validate against points not used to build the spline: for example, dense samples along the original OpenDRIVE reference line inside clothoid or arc sections, with per-road error distributions and worst-case deviations. Without such a held-out check, the claim of perfect accuracy is unsupported.
  2. [Section V-B, Table I] The comparison of pass rates between "OpenCat" and "SensoDat" conflates several independent factors: the road representation (OpenDRIVE vs. Catmull-Rom spline), the simulator (BeamNG vs. Udacity), the driving model (PID autopilot vs. Dave-2), and the road-validity filters (e.g., the 250×250 bounding box and no-intersection criteria applied to the converted roads, which may not have been applied to the original SensoDat scenarios). The paper attributes the 25–50 percentage point pass-rate difference to ADAS/benchmark coupling, but any of these factors, or their interaction, could explain the gap. The causal claim that "benchmarks built with specific ADAS models hinder their effective usage for regression testing" requires experiments that isolate these variables, for instance running the same model on both original and converted roads in the same simulator, or at least documenting and matching the validity filters and failure criteria across both conditions.
  3. [Section III-B, cited against Section V-B] The manuscript itself notes that Catmull-Rom splines can overshoot when control points are unevenly spaced. Because no dense comparison between the spline and the original OpenDRIVE geometry is reported, the higher pass rates in Udacity could simply result from geometric alteration (e.g., smoothing of sharp clothoid corners) rather than from an interoperability benefit. The paper needs to rule out this alternative explanation by providing quantitative evidence of geometric fidelity between control points; otherwise the principal conclusion is not established and the contribution could be better framed as a simplification or smoothing operation.
minor comments (4)
  1. [Section V-A] The visual validation in Figure 4 shows only a single road; a quantitative summary across all 32,580 conversions (e.g., distribution of error metrics, worst-case examples) would be more informative than one representative case.
  2. [Table I caption] The caption should explicitly state that the SensoDat pass percentages are the original reported values from BeamNG with a PID autopilot, while the OpenCat pass percentages are from Udacity with Dave-2, so that readers do not mistake the two columns for outcomes under identical conditions.
  3. [Section IV-A, Algorithm 2] The condition on line 4, `if len (right_lane_points) or len (left_lane_points) < min_points`, appears to be missing parentheses: as written it tests the truthiness of `len(right_lane_points)` and separately compares `len(left_lane_points)` to `min_points`, which allows a road with zero right-lane points and at least four left-lane points to pass the guard and fail later. It should be `if len(right_lane_points) < min_points or len(left_lane_points) < min_points`.
  4. [Section IV-A] The description of the spline generator says it is configured with alpha = 0.5 and "produce one spline point per segment"; the meaning of this sampling density is unclear and could affect both the visual representation and the effective road geometry used in simulation. Clarify how many points are generated per segment and whether the Udacity road generator uses those points directly.

Circularity Check

2 steps flagged · score 6.0 of 10

RQ1's perfect accuracy is tautological: the Catmull-Rom spline is checked against the same control points used to build it, so 100% and R²=1 are properties of interpolation, and Section V-B's 'converted geometry is identical' premise inherits this circularity.

  1. self definitional [Section IV-A (Algorithm 2) and Section V-A (Eqs. 3-4)]
    "control_points← compute_centerline(right_lane_points, left_lane_points) ... spline_points← spline_generator.generate_spline() ... In addition to visual validation, we also verify our results using the following metrics ... The conversion achieved perfect accuracy (accuracy = 100%) and R-squared (R2=1), confirming the correctness and quality of OpenCat."

    Algorithm 2 derives control_points from the OpenDRIVE lane points and builds the Catmull-Rom spline through them; Section III-B states that the spline 'passes directly through each control point (no approximation)'. Eq. (3) then measures the distance between spline points and the 'original points', i.e., the same control points used to define the spline. At each control point the interpolation residual is zero by construction, so Accuracy=100% and R²=1 are mathematical identities of the interpolating spline, not independent confirmation that the converted road matches OpenDRIVE geometry between control points. Algorithm 1 extracts only geometry-section attributes, so interior clothoid/arc points are never compared.

  2. other [Section V-B (RQ2), paragraph after Table I]
    "While the converted geometry is identical to the original, as established in RQ1, the significant difference in test pass/fail results can be attributed to the interaction between the representation format, the ADAS under test (e.g., Dave-2), and the simulator environment."

    This sentence makes the RQ1 equivalence load-bearing: the attribution of the 25-50 percentage-point pass-rate gap to ADAS/simulator coupling instead of geometric alteration rests entirely on the self-referential 100%/R²=1 result from Step 1. No comparison is performed at points sampled inside OpenDRIVE geometry sections, and Section III-B itself concedes that Catmull-Rom splines 'can overshoot' when control points are unevenly spaced. The RQ2 conclusion is therefore not independently established; it inherits the circularity of the RQ1 validation rather than providing a separate empirical check of geometric fidelity.

full rationale

The central circularity is in RQ1: a Catmull-Rom spline is an interpolating curve, so the paper's accuracy metric, which compares the spline with the very points fed into the spline generator, is satisfied trivially. That makes the headline '100% accuracy' and R²=1 a property of the construction, not a verification that converted roads reproduce OpenDRIVE geometry between control points. This matters because Section V-B uses 'the converted geometry is identical to the original, as established in RQ1' to interpret the large pass-rate difference as evidence that SensoDat is coupled to a specific ADAS model. The RQ2 simulation itself is an external empirical comparison (converted roads run with Dave-2 in Udacity versus SensoDat's original BeamNG runs), so the benchmark-coupling conclusion is not itself a fitted input or a self-citation chain; however, its causal interpretation depends on the circular geometry-identity premise. The paper's self-citations (e.g., the Biagiola et al. framework by a co-author) provide tooling and prior validation rather than the load-bearing mathematical premise, so they do not raise the score further. Overall: partial circularity, score 6.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on an interpolation-by-construction accuracy metric, on assumptions about SensoDat's road structure, and on comparability of pass/fail labels across two different simulators and ADAS models. Free parameters are the spline tension and sampling choices. No invented entities are introduced.

free parameters (3)
  • Catmull-Rom tension alpha = 0.5
    Chosen in Algorithm 2 without sensitivity analysis; controls how tightly the spline bends between control points and can change road curvature and pass rates.
  • Spline sampling density = 1 spline point per segment
    Configuration chosen in Section IV-A; determines how finely the road is represented and how faithfully the simulator sees curves.
  • Minimum control points = 4
    Algorithm 2 line 3; roads with fewer than 4 points are skipped, which can exclude short roads from conversion.
assumptions (4)
  • domain assumption SensoDat roads have no junctions and are processed as independent roads.
    Section III-A states this; if false, per-road conversion would miss connectivity that affects drivability.
  • domain assumption The centerline can be computed by averaging right and left lane boundary points.
    Algorithm 2 line 7; OpenDRIVE's actual reference line is defined by the planView, not necessarily the midpoint of lane edges, so asymmetric lane widths would shift the converted path.
  • domain assumption Original SensoDat pass/fail labels from BeamNG with a PID autopilot are directly comparable to Udacity/Dave-2 outcomes under the OOB metric.
    Section V-B compares percentages from two different simulators and ADAS models without controlling for simulator physics, road-validity filters, or model capabilities.
  • ad hoc to paper Catmull-Rom interpolation with the selected parameters preserves OpenDRIVE geometry between control points.
    Adopted in Section IV without independent mid-segment verification; the accuracy metric only checks points used to build the spline.

how reviews work

0 comments
Cite this review

Pith. "Pith review of OpenCat: Improving Interoperability of ADS Testing." pith.science (2026). https://pith.science/paper/UFQJHVLH

@misc{pith2026250207719,
  author       = {Pith},
  title        = {Pith review of: OpenCat: Improving Interoperability of ADS Testing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UFQJHVLH}},
  note         = {Machine review of arXiv:2502.07719}
}
read the original abstract

Testing Advanced Driving Assistance Systems (ADAS), such as lane-keeping functions, requires creating road topologies or using predefined benchmarks. However, the test cases in existing ADAS benchmarks are often designed in specific formats (e.g., OpenDRIVE) and tailored to specific ADAS models. This limits their reusability and interoperability with other simulators and models, making it challenging to assess ADAS functionalities independently of the platform-specific details used to create the test cases. This paper evaluates the interoperability of SensoDat, a benchmark developed for ADAS regression testing. We introduce OpenCat, a converter that transforms OpenDRIVE test cases into the Catmull-Rom spline format, which is widely supported by many current test generators. By applying OpenCat to the SensoDat dataset, we achieved high accuracy in converting test cases into reusable road scenarios. To validate the converted scenarios, we used them to evaluate a lane-keeping ADAS model using the Udacity simulator. Both the simulator and the ADAS model operate independently of the technologies underlying SensoDat, ensuring an unbiased evaluation of the original test cases. Our findings reveal that benchmarks built with specific ADAS models hinder their effective usage for regression testing. We conclude by offering insights and recommendations to enhance the reusability and transferability of ADAS benchmarks for more extensive applications.

Figures

Figures reproduced from arXiv: 2502.07719 by the authors.

Figure 1
Figure 1. Elements of OpenDRIVE [45]. 𝑓 : R 𝑑 → [−25◦ , +25◦ ]. Here, 𝑑 denotes the dimension of the input image 𝒙 ∈ R 𝑑 (e.g., for an image of size 140 × 320, 𝑑 = 44, 800 pixels), and the output is a scalar value 𝑦 repre￾senting the predicted steering angle. This angle lies within the range −25◦ , +25◦ [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Catmull-Rom spline interpolation with four points. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. Comparative view of OpenDRIVE road, generated spline, and test case. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

58 extracted references · 54 canonical work pages

  1. [1]

    A Survey on Automated Driving System Testing: Landscapes and Trends,

    S. Tang, Z. Zhang, Y. Zhang, J. Zhou, Y. Guo, S. Liu, S. Guo, Y.-F. Li, L. Ma, Y. Xue, and Y. Liu, “A Survey on Automated Driving System Testing: Landscapes and Trends,”ACM Trans. Softw. Eng. Methodol., vol. 32, no. 5, Jul. 2023

    work page 2023

  2. [2]

    A survey of autonomous driving: Common practices and emerging technologies,

    E. Yurtsever, J. Lambert, A. Carballo, and K. Takeda, “A survey of autonomous driving: Common practices and emerging technologies,” IEEE access, vol. 8, pp. 58443–58469, 2020

    work page 2020

  3. [3]

    Panoptic perception for autonomous driving: A survey,

    Y. Li and L. Xu, “Panoptic perception for autonomous driving: A survey,” 2024. [Online]. Available: https://arxiv.org/abs/2408.15388

    arXiv 2024

  4. [4]

    A survey of deep learning techniques for autonomous driving,

    S. Grigorescu, B. Trasnea, T. Cocias, and G. Macesanu, “A survey of deep learning techniques for autonomous driving,”Journal of Field Robotics, vol. 37, no. 3, pp. 362–386, 2020

    work page 2020

  5. [5]

    Testing Machine Learning based Systems: A Systematic Mapping,

    V. Riccio, G. Jahangirova, A. Stocco, N. Humbatova, M. Weiss, and P. Tonella, “Testing Machine Learning based Systems: A Systematic Mapping,” Empirical Software Engineering, 2020

    work page 2020

  6. [6]

    High-definition maps: Comprehensive survey, challenges, and future perspectives,

    G. Elghazaly, R. Frank, S. Harvey, and S. Safko, “High-definition maps: Comprehensive survey, challenges, and future perspectives,”IEEE Open Journal of Intelligent Transportation Systems, vol. 4, pp. 527–550,

    work page

  7. [7]

    Technical challenges in autonomous ve- hicle development,

    A. Banafa, “Technical challenges in autonomous ve- hicle development,” https://www.linkedin.com/pulse/ technical-challenges-autonomous-vehicle-development-banafa-ypfhc/, 2024, accessed: 2024-11-03

    work page 2024

  8. [8]

    ThirdEye: Attention Maps for Safe Autonomous Driving Systems,

    A. Stocco, P. J. Nunes, M. d’Amorim, and P. Tonella, “ThirdEye: Attention Maps for Safe Autonomous Driving Systems,” inProceedings of 37th IEEE/ACM International Conference on Automated Software Engineering, ser. ASE ’22. IEEE/ACM, 2022

    work page 2022

Show all 58 references
  1. [9]

    Misbehaviour prediction for autonomous driving systems,

    A. Stocco, M. Weiss, M. Calzana, and P. Tonella, “Misbehaviour prediction for autonomous driving systems,” in Proceedings of ACM 42nd International Conference on Software Engineering, ser. ICSE ’20. ACM, 2020, p. 12 pages

    2020

  2. [10]

    Model vs system level testing of autonomous driving systems: a replication and extension study,

    A. Stocco, B. Pulfer, and P. Tonella, “Model vs system level testing of autonomous driving systems: a replication and extension study,” Empirical Software Engineering, vol. 28, no. 3, p. 73, May 2023

    2023

  3. [11]

    Confidence-driven weighted retraining for predicting safety-critical failures in autonomous driving systems,

    A. Stocco and P. Tonella, “Confidence-driven weighted retraining for predicting safety-critical failures in autonomous driving systems,” Journal of Software: Evolution and Process, 2021. [Online]. Available: https://doi.org/10.1002/smr.2386

    2021 doi

  4. [12]

    Assessing quality metrics for neural reality gap input mitigation in autonomous driving testing,

    S. C. Lambertenghi and A. Stocco, “Assessing quality metrics for neural reality gap input mitigation in autonomous driving testing,” in Proceedings of the IEEE 17th International Conference on Software Testing, Verification and Validation, ser. ICST ’24. IEEE, 2024, p. 12 pages

    2024

  5. [13]

    Benchmarking image perturbations for testing automated driving assistance systems,

    S. C. Lambertenghi, H. Leonhard, and A. Stocco, “Benchmarking image perturbations for testing automated driving assistance systems,” in Proceedings of the IEEE 18th IEEE International Conference on Software Testing, Verification and Validation, ser. ICST ’25. IEEE, 2025, p. 12 pages

    2025

  6. [14]

    Mind the Gap! A Study on the Transferability of Virtual Versus Physical-World Testing of Autonomous Driving Systems,

    A. Stocco, B. Pulfer, and P. Tonella, “Mind the Gap! A Study on the Transferability of Virtual Versus Physical-World Testing of Autonomous Driving Systems,”IEEE Transactions on Software Engineering, vol. 49, no. 04, pp. 1928–1940, apr 2023

    1928

  7. [15]

    Testing autonomous ve- hicle software in the virtual prototyping environment,

    B. Kim, Y. Kashiba, S. Dai, and S. Shiraishi, “Testing autonomous ve- hicle software in the virtual prototyping environment,”IEEE Embedded Systems Letters, vol. 9, no. 1, pp. 5–8, 2017

    2017

  8. [16]

    ollner, “Testing and validating high level components for automated driving: Simulation framework for traffic scenarios,

    M. R. Zofka, S. Klemm, F. Kuhnt, T. Schamm, and J. M. Z"ollner, “Testing and validating high level components for automated driving: Simulation framework for traffic scenarios,” inProceedins of the IEEE Intelligent Vehicles Symposium, 2016, pp. 144–150

    2016

  9. [17]

    Identifying and explaining safety-critical scenarios for autonomous vehicles via key features,

    N. Neelofar and A. Aleti, “Identifying and explaining safety-critical scenarios for autonomous vehicles via key features,”ACM Trans. Softw. Eng. Methodol., vol. 33, no. 4, Apr. 2024. 7

    2024

  10. [18]

    Towards Reliable AI: Adequacy Metrics for Ensuring the Quality of System-level Testing of Autonomous Vehicles,

    ——, “Towards Reliable AI: Adequacy Metrics for Ensuring the Quality of System-level Testing of Autonomous Vehicles,” inProceedings of the IEEE/ACM 46th International Conference on Software Engineering (ICSE). ACM, 2024

    2024

  11. [19]

    PAFOT: A Position- Based Approach for Finding Optimal Tests of Autonomous Vehicles,

    V. Crespo-Rodriguez, Neelofar, and A. Aleti, “PAFOT: A Position- Based Approach for Finding Optimal Tests of Autonomous Vehicles,” in Proceedings of the ACM/IEEE 5th International Conference on Automation of Software Test (AST 2024), ser. AST ’24. ACM, 2024, p. 159–170

    2024

  12. [20]

    Epitester: Testing autonomous vehicles with epigenetic algorithm and attention mechanism,

    C. Lu, S. Ali, and T. Yue, “Epitester: Testing autonomous vehicles with epigenetic algorithm and attention mechanism,”IEEE Transactions on Software Engineering, pp. 1–19, 2024

    2024

  13. [21]

    DeepQTest: Testing Autonomous Driving Systems with Reinforcement Learning and Real-world Weather Data,

    C. Lu, T. Yue, M. Zhang, and S. Ali, “DeepQTest: Testing Autonomous Driving Systems with Reinforcement Learning and Real-world Weather Data,” 2023. [Online]. Available: https://arxiv.org/abs/2310.05170

    2023 arXiv

  14. [22]

    Safety assessment of vehicle characteristics variations in autonomous driving systems,

    Q. Pan, T. Wang, P. Arcaini, T. Yue, and S. Ali, “Safety assessment of vehicle characteristics variations in autonomous driving systems,”

  15. [23]

    An empirical comparison of combinatorial testing and search-based testing in the context of automated and autonomous driving systems,

    F. Klück, Y. Li, J. Tao, and F. Wotawa, “An empirical comparison of combinatorial testing and search-based testing in the context of automated and autonomous driving systems,”Information and Software Technology, vol. 160, p. 107225, 2023

    2023

  16. [24]

    Available: https://arxiv.org/abs/2311.14461

    [Online]. Available: https://arxiv.org/abs/2311.14461

    arXiv

  17. [25]

    Ambiegen: A search- based framework for autonomous systems testingimage 1,

    D. Humeniuk, F. Khomh, and G. Antoniol, “Ambiegen: A search- based framework for autonomous systems testingimage 1,”Science of Computer Programming, vol. 230, p. 102990, 2023

    2023

  18. [26]

    Utilizing genetic algorithms for generating critical scenarios for testing autonomous driving functions,

    F. Klück, D. Sumann, and F. Wotawa, “Utilizing genetic algorithms for generating critical scenarios for testing autonomous driving functions,” in 2024 IEEE International Conference on Artificial Intelligence Testing (AITest), 2024, pp. 73–80

    2024

  19. [27]

    Crag – a combinatorial testing-based generator of road geometries for ads testing,

    P. Arcaini and A. Cetinkaya, “Crag – a combinatorial testing-based generator of road geometries for ads testing,” Science of Computer Programming, vol. 238, p. 103171, 2024

    2024

  20. [28]

    Reality bites: Assessing the realism of driving scenarios with large language models,

    J. Wu, C. Lu, A. Arrieta, T. Yue, and S. Ali, “Reality bites: Assessing the realism of driving scenarios with large language models,” inProceedings of the IEEE/ACM 2024 First International Conference on AI Foundation Models and Software Engineering (FORGE). ACM, 2024, p. 40–51

    2024

  21. [29]

    Simulation-based safety testing of automated driving systems,

    F. Khan, H. Anwar, and D. Pfahl, “Simulation-based safety testing of automated driving systems,” in Product-Focused Software Process Improvement. Springer, 2024, pp. 133–138

    2024

  22. [30]

    Parameter coverage for testing of autonomous driving systems under uncertainty,

    T. Laurent, S. Klikovits, P. Arcaini, F. Ishikawa, and A. Ventresque, “Parameter coverage for testing of autonomous driving systems under uncertainty,” ACM Trans. Softw. Eng. Methodol., vol. 32, no. 3, Apr. 2023

    2023

  23. [31]

    Efficient domain augmentation for autonomous driving testing using diffusion models,

    L. Baresi, D. Y. X. Hu, A. Stocco, and P. Tonella, “Efficient domain augmentation for autonomous driving testing using diffusion models,” in Proceedings of the IEEE 47th International Conference on Software Engineering, ser. ICSE ’25. IEEE, 2025

    2025

  24. [32]

    A process for scenario prioritization and selection in simulation- based safety testing of automated driving systems,

    ——, “A process for scenario prioritization and selection in simulation- based safety testing of automated driving systems,” inProduct-Focused Software Process Improvement. Springer, 2024, pp. 89–99

    2024

  25. [33]

    Sensodat: Simulation-based sensor dataset of self-driving cars,

    C. Birchler, C. Rohrbach, T. Kehrer, and S. Panichella, “Sensodat: Simulation-based sensor dataset of self-driving cars,” in Proceedings of the 21st International Conference on Mining Software Repositories, ser. MSR ’24. New York, NY, USA: Association for Computing Machinery, ...

    2024

  26. [34]

    Testing in the evolving world of dl systems: Insights from python github projects,

    Q. Ali, O. Riganelli, and L. Mariani, “Testing in the evolving world of dl systems: Insights from python github projects,” in2024 IEEE 24th International Conference on Software Quality, Reliability and Security (QRS), 2024, pp. 25–35

    2024

  27. [35]

    Sctrans: Constructing a large public scenario dataset for simulation testing of autonomous driving systems,

    J. Dai, B. Gao, M. Luo, Z. Huang, Z. Li, Y. Zhang, and M. Yang, “Sctrans: Constructing a large public scenario dataset for simulation testing of autonomous driving systems,” inProceedings of the IEEE/ACM 46th International Conference on Software Engineering, ser. ICSE ’24. New...

  28. [36]

    Deepscenario: An open driving scenario dataset for autonomous driving system testing,

    C. Lu, T. Yue, and S. Ali, “Deepscenario: An open driving scenario dataset for autonomous driving system testing,” in2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR), 2023, pp. 52–56

    2023

  29. [37]

    Evaluating the impact of flaky simulators on testing autonomous driving systems,

    M. H. Amini, S. Naseri, and S. Nejati, “Evaluating the impact of flaky simulators on testing autonomous driving systems,” Empirical Softw. Engg., vol. 29, no. 2, feb 2024

    2024

  30. [38]

    BeamNG GmbH,

    BeamNG.tech, “BeamNG GmbH,” https://beamng.tech/, 2024, online; accessed 2024-11-19

    2024

  31. [39]

    Digital twins are not monozygotic–cross-replicating adas testing in two industry-grade automotive simulators,

    M. Borg, R. B. Abdessalem, S. Nejati, F.-X. Jegeden, and D. Shin, “Digital twins are not monozygotic–cross-replicating adas testing in two industry-grade automotive simulators,” inICST ’21. IEEE, 2021

    2021

  32. [40]

    OpenDRIVE 2010 and beyond – status and future of the de facto standard for the description of road networks,

    M. Dupuis, M. Strobl, and H. Grezlikowski, “OpenDRIVE 2010 and beyond – status and future of the de facto standard for the description of road networks,” inProceeding of the Driving Simulation Conference Europe, 2010, pp. 231–242

    2010

  33. [41]

    A class of local interpolating splines,

    E. E. Catmull and R. Rom, “A class of local interpolating splines,” Computer Aided Geometric Design , pp. 317–326, 1974. [Online]. Available: https://api.semanticscholar.org/CorpusID:118383557

    1974

  34. [42]

    Self-driving car simulator,

    Udacity, “Self-driving car simulator,” 2023, accessed: 2024-11-04. [Online]. Available: https://github.com/udacity/self-driving-car-sim

    2023

  35. [43]

    End to end learning for self-driving cars

    M. Bojarski, D. Del Testa, D. Dworakowski, B. Firner, B. Flepp, P. Goyal, L. D. Jackel, M. Monfort, U. Muller, J. Zhang, X. Zhang, J. Zhao, and K. Zieba, “End to end learning for self-driving cars.”CoRR, vol. abs/1604.07316, 2016

    2016 arXiv

  36. [44]

    Udacity self-driving car simulator,

    Udacity, “Udacity self-driving car simulator,” https://github.com/ udacity/self-driving-car-sim, 2021, accessed: [2024-01-15]

    2021

  37. [45]

    A framework for automated driving system testable cases and scenarios,

    U. D. of Transportation, “A framework for automated driving system testable cases and scenarios,” https://rosap.ntl.bts.gov/view/dot/38824/ dot_38824_DS1.pdf, 2018

    2018

  38. [46]

    The use of clothoid templates in highway design,

    K. Baass, “The use of clothoid templates in highway design,” in Transportation Forum, vol. 1, 1984, pp. 47–52

    1984

  39. [47]

    Validation of parametric opendrive road space models,

    B. Schwab and T. H. Kolbe, “Validation of parametric opendrive road space models,”ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences , vol. X-4/W2-2022, pp. 257–264,

    2022

  40. [48]

    ASAM OpenDRIVE ®,

    Association for Standardization of Automation and Measuring Systems (ASAM), “ASAM OpenDRIVE ®,” https://www.asam.net/standards/ detail/opendrive/, 2023, accessed: 2024-11-04

    2023

  41. [49]

    Catmull-rom splines,

    S. Documentation, “Catmull-rom splines,” 2024, accessed: 2024-11-19. [Online]. Available: https://splines.readthedocs.io/en/latest/euclidean/ catmull-rom.html

    2024

  42. [50]

    Opencat,

    Q. Ali, “Opencat,” 2024, accessed: 2024-11-19. [Online]. Available: https://github.com/lakhanqurban/OpenCat

    2024

  43. [51]

    libopendrive,

    M. Pagel et al., “libopendrive,” 2024, accessed: 2024-11-04. [Online]. Available: https://github.com/pageldev/libOpenDRIVE

    2024

  44. [52]

    Does road diversity really matter in testing automated driving systems? – a registered report,

    S. Klikovits, V. Riccio, E. Castellano, A. Cetinkaya, A. Gambi, and P. Arcaini, “Does road diversity really matter in testing automated driving systems? – a registered report,” 2022. [Online]. Available: https://arxiv.org/abs/2209.05947

    2022 arXiv

  45. [53]

    Two is better than one: digital siblings to improve autonomous driving testing,

    M. Biagiola, A. Stocco, V. Riccio, and P. Tonella, “Two is better than one: digital siblings to improve autonomous driving testing,”Empirical Software Engineering, vol. 29, p. 72, 2024. [Online]. Available: https://doi.org/10.1007/s10664-024-10458-4

    2024 doi

  46. [54]

    Automatically testing self-driving cars with search-based procedural content generation,

    A. Gambi, M. Mueller, and G. Fraser, “Automatically testing self-driving cars with search-based procedural content generation,” inProceedings of the ACM 28th SIGSOFT International Symposium on Software Testing and Analysis, 2019, pp. 318–328. 8

    2019

  47. [56]

    Opendrive viewer,

    S. Pagel, “Opendrive viewer,” https://sebastian-pagel.net/odrviewer/, ac- cessed: 2024-11-04

    2024

  48. [2022]

    Available: https://isprs-annals.copernicus.org/articles/ X-4-W2-2022/257/2022/

    [Online]. Available: https://isprs-annals.copernicus.org/articles/ X-4-W2-2022/257/2022/

    2022

  49. [2023]

    Available: https://api.semanticscholar.org/CorpusID: 259915452

    [Online]. Available: https://api.semanticscholar.org/CorpusID: 259915452

  50. [2024]

    Available: https://doi.org/10.1145/3597503.3623350

    [Online]. Available: https://doi.org/10.1145/3597503.3623350

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.