ICST Tool Competition 2025 -- Self-Driving Car Testing Track

Christian Birchler; Mattia Fazzini; Sebastiano Panichella; Stefan Klikovits

REVIEW 3 major objections 6 minor 9 references

ICST Tool Competition 2025 -- Self-Driving Car Testing Track

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read In the first tool competition on self-driving-car testing, the ITS4SDC tool proved best at selecting fault-revealing simulation tests.

desk verdict Useful SDC tool-competition report with real artifacts, but the 'best tool' claim overreaches: the time-to-fault difference is within sampling noise; only the precision result is likely real. read the letter →

arxiv 2502.09982 v1 pith:RSZXYLVY submitted 2025-02-14 cs.SE

Christian Birchler , Stefan Klikovits , Mattia Fazzini , Sebastiano Panichella This is my paper · ORCID

classification cs.SE

keywords self-drivingcartestingtestselectionregressionsimulation-basedtoolcompetitionSensoDatfault-to-selectionratiocost-effectiveness

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper reports the first tool competition on regression test selection for simulation-based testing of self-driving cars. Five submitted tools and a random-selection baseline picked test cases from 36 pre-executed SensoDat collections, and the results were scored with two cost-effectiveness metrics. The paper's central claim is that ITS4SDC is the best-performing tool: it achieves the best mean fault-to-selection ratio (0.805) and the best mean simulation time-to-fault ratio (65.39). A sympathetic reader should care because the finding suggests that a prediction-based selector can find more failing simulation tests per unit of simulation time than random selection or the other entrants.

What carries the argument

The engine of the comparison is the competition's evaluation protocol: a containerized interface through which each tool receives an initialization set and then must select test cases from a suite, with all interactions logged by an automated evaluator. Every tool is run on the same 36 SensoDat collections, each treated as an independent test suite; within each suite, 80% of the tests are given for initialization (training) and the remaining 20% are the selection targets. SensoDat is a repository of pre-executed simulation test cases for self-driving cars with pass/fail labels, generated by three different test generators. The cost-effectiveness metrics that carry the ranking are the fault-to-selection ratio (precision of the selection) and the simulation time-to-fault ratio (execution cost per revealed fault), and the paper's findings report the averaged statistics over the 36 runs.

What would settle it

Re-run the identical protocol but hold out entire SensoDat collections instead of splitting each collection 80/20, and compare ITS4SDC with the random baseline on the untouched suite; if the winner's 0.805 fault-to-selection ratio collapses toward the baseline's 0.379, the in-collection split is what carried the result.

Watch

Extended reading notes

Core claim

According to the competition's results, ITS4SDC outperforms both the random baseline and the other four tools on the two cost-effectiveness metrics defined in the study. Across the 36 SensoDat test suites, ITS4SDC selected on average 67.8 test cases per suite and achieved a mean fault-to-selection ratio of 0.8048, meaning about four of every five selected tests failed; the random baseline selected 88.2 tests and achieved 0.3790. ITS4SDC also achieved the best mean simulation time-to-fault ratio of 65.39 seconds of simulation per fault, versus 156.23 for random selection and 72.96 for the next best tool, DRVN Tool. The authors conclude from these numbers that ITS4SDC is the best-performing tool in this first edition.

Load-bearing premise

The entire ranking rests on the assumption that the pre-computed failure labels in the SensoDat collections are a correct and complete oracle for SDC faults; if those labels are noisy or generator-specific, the measured ratios and the tool order could change.

Editorial extensions

If this is right

A prediction-based test selector can be substantially more cost-effective than random selection in simulation-based SDC testing, since the winner's precision was more than double the baseline's.
Tools that select fewer, mostly failing test cases can cut simulation time per fault by more than half compared with random selection.
Model complexity has observable cost consequences: the two tools with the slowest selection (DETOUR and DRVN Tool) are also the ones with costly or complex models, while ITS4SDC selects in under a second on average.
The defined metrics can be reused in future editions to compare new SDC test selection tools against these first-edition results.
Because no clear diversity difference was found, selection cost and fault precision, not road-curvature diversity, drove the ranking.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The 80/20 split is taken from inside each SensoDat collection, so the tools never face a fully unseen test suite; whether ITS4SDC's advantage survives whole-campaign holdout is an open question.
The pass/fail labels come from the original SensoDat creators; if those labels are oracle-dependent, a different oracle could reorder the tools, even though all tools were treated equally.
A practical takeaway implied but not stated by the authors is that for cost-sensitive regression testing, a lightweight predictor with negligible initialization and sub-second selection is more attractive than a slower model with similar precision.
The competition protocol could be extended to test selection across generators (training on Ambiegen campaigns, selecting from Frenetic campaigns), which would probe whether the learned predictors generalize across test generation distributions.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

This is a genuinely useful competition report. It is the first SDC-specific edition of the ICST tool competition, and the authors did the organizational work right: public GitHub repo, Docker images for the five tools, Zenodo archive of results, gRPC interface, and a random baseline. The 36 SensoDat collections are a reasonable testbed, and the paper is transparent about the simple metrics it uses. That transparency is a real strength.

What is actually new is the empirical comparison itself. The finding that ITS4SDC achieves a mean fault-to-selection ratio of 0.80 versus the random baseline's 0.38 is the kind of result the community wants. The authors are also candid that most tools select far fewer tests than random, which suggests learned predictors are doing something.

The soft spots are real. The headline conclusion, 'the best-performing tool is ITS4SDC,' rests on two means with no significance testing, no confidence intervals, and no repeated runs. The stress-test note is right: the time-to-fault difference between ITS4SDC (65.39) and DRVN (72.96) is trivial relative to the standard deviations (33.46 and 40.18). A rough two-sample comparison gives t ≈ 0.87. That difference is not statistically established, and Finding 4 should not be presented as a meaningful advantage. The fault-to-selection difference (0.8048 vs 0.7424) looks more substantial—rough t ≈ 2.5—so the paper does have one robust result. But the 'best' claim depends on both metrics, so it needs to be softened.

The single 80/20 split per collection is another concern. With between-collection variance this high, a different split could change the ranking. The paper should report repeated splits or at least show that the precision result is stable. The time-to-fault metric is also not protected against differences in test length, and RQ2 is answered by speculation rather than by analyzing the submitted tools' actual features.

None of this kills the paper. It is a competition report, not a definitive scientific study, and the artifacts alone justify publication. Who is this for? Researchers working on SDC regression testing and participants in tool competitions. They will use the platform and benchmark. It deserves a serious referee, but the authors should add basic significance testing, report confidence intervals, and revise the conclusion to match the evidence.

Recommendation: send to peer review with a request for statistical analysis and a more careful claim.

Referee Report

3 major / 6 minor

Summary. This paper reports the first ICST Tool Competition on self-driving car testing, held with five participating tools (DRVN, DETOUR, ITS4SDC, Graph Selector, CertiFail) plus a random baseline. The competition platform provides a gRPC/Protocol Buffers interface, Docker packaging, and a public GitHub repository; evaluation is performed on 36 pre-executed SensoDat test collections using regression-test-selection metrics such as selection count, initialization time, selection time, simulation time-to-fault ratio, fault-to-selection ratio, and curvature diversity. The paper's central result, stated in Section V, is that ITS4SDC is the best-performing tool, based on the best mean fault-to-selection ratio (0.8048) and best mean simulation time-to-fault ratio (65.39) over N=36 collections.

Significance. If the empirical result is statistically supported, the paper provides a useful, reproducible evaluation infrastructure for SDC regression test selection: the Docker/gRPC interface, the public benchmark artifacts, and the comparison of five independently submitted tools are concrete contributions that lower the entry barrier for future competition participants. The paper ships the evaluation data on Zenodo and the platform on GitHub, which is a genuine strength. However, the headline conclusion rests on means without significance testing, robustness checks, or repeated runs, and the paper's second research question (RQ2) is not actually answered. With appropriate statistical strengthening and a more carefully scoped conclusion, the paper would be a valuable competition report for the SDC testing community.

major comments (3)

[Section III (Findings 4–5) and Section V] The claim that ITS4SDC is the best-performing tool is not statistically established for the simulation time-to-fault ratio. From Table III, ITS4SDC has mean 65.39 (std 33.46) and DRVN has mean 72.96 (std 40.18) over N=36 collections. The raw difference is 7.57, while a rough two-sample standard error is sqrt((33.46^2 + 40.18^2)/36) = 8.71, giving t ≈ 0.87, which is far from significant. No confidence intervals, paired tests, or adjustment for multiple comparisons are reported anywhere in Section III or Section II-G, so Finding 4, and therefore the overall winner declaration in Section V, is fragile. I recommend adding paired significance tests or bootstrap confidence intervals for all metrics, and rewording the conclusion to say that ITS4SDC achieved the best observed mean values rather than that it is definitively best-performing.
[Section II-F and Section II-G] The evaluation uses a single 80/20 split per collection and no repeated executions of the tools, although several tools are likely stochastic. The reported standard deviations therefore capture between-collection variance only, not run-to-run variability or split sensitivity. With only 36 collections and high variance in the time-to-fault metric, the ranking could change under a different split or seed. The paper should report repeated splits (or bootstrap resampling over collections) and, where applicable, repeated runs of each tool, to demonstrate that the observed ranking, especially the ITS4SDC versus DRVN comparison, is stable.
[Section I (RQ2) and Section IV] The paper poses RQ2 as 'What are the features leveraged by the top-performing tools?' but never answers it with evidence. Section IV speculates that the tools use 'most likely models, which predict the test outcomes' and that DETOUR and DRVN have 'rather complex' models, but no feature analysis, ablation, or inspection of the submitted tools is provided. Either the RQ should be answered with concrete measurements (e.g., type of model, input features, selection heuristic reported by each participant) or RQ2 should be removed or explicitly deferred to the individual tool papers.

minor comments (6)

[Abstract and Section I] There are several typos: 'approache' should be 'approach', 'SDCS' should be 'SDCs', and Figure 1's caption says 'Intercation' instead of 'Interaction'.
[Table III] The column headers 'time to fault ration' and 'fault to selection ration' should be spelled 'ratio'.
[Table II] The generator names are inconsistent: 'frenetic v', 'freneticV', and 'FreneticV' are used in different rows, which makes the table harder to read and could confuse the mapping to the three generators described in Section II-C.
[References] Reference [10] lists the fourth author as 'R. S.'; this appears to be an incomplete or corrupted author name and should be corrected.
[Section II-D] The definition of curvature diversity says 'return its mean value' without specifying whether the mean is taken over road segments of a single test case or over the selected test cases in a collection; please clarify.
[Section II-F] The procedure does not state what information the initialization data contains (e.g., whether failure labels are provided for the 80% training portion). This is important for interpreting the results and should be stated explicitly.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the tool ranking is an empirical measurement, not a derivation from fitted inputs or self-citations.

full rationale

The paper's central claim (Section V: 'the best-performing tool is ITS4SDC') is an empirical statement about measurements on the SensoDat benchmark. The fault-to-selection and simulation-time-to-fault ratios are computed from tool outputs, and no parameter of the evaluation is fitted to produce the conclusion; the random baseline is an independent yardstick. The benchmark and platform are cited from prior work with overlapping authors, but those citations supply data and infrastructure, not a theorem that entails the ranking. The comparison among five independently submitted tools and a random selector is externally falsifiable. Concerns that the single 80/20 split and lack of significance testing make the time-to-fault lead fragile are statistical-validity concerns, not circularity: the result is not equivalent to its inputs by construction, and the ranking could in principle fail under a different split or benchmark. No circular step can be quoted because none exists.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The ranking depends on the benchmark, its failure labels, the 80/20 split, and the organizers' metric choices. Each of these is a domain assumption rather than a derived fact; none are justified with external validation or sensitivity analysis.

free parameters (1)

Random baseline selection probability p = 0.5
Section II-A.3: the baseline is a random selector with p=0.5; the choice is arbitrary and no sensitivity analysis is provided.

assumptions (4)

domain assumption SensoDat pre-executed test cases and their failure labels are a valid ground truth for regression test selection on SDCs.
Section II-C introduces SensoDat as the benchmark, and all metrics count failures against these labels; if the oracle is noisy or labels are generator-dependent, the tool ranking is not meaningful.
domain assumption The 80/20 split within each SensoDat collection gives tools enough initialization data and produces an unbiased evaluation set.
Section II-F states all tools use 80% of each collection for initialization and the remaining 20% for evaluation; this assumes the split is representative and that no leakage occurs through shared collection conditions.
domain assumption The five metrics listed in Section II-D, especially simulation time-to-fault ratio and fault-to-selection ratio, adequately capture test-selection cost-effectiveness.
The paper itself calls the metrics 'rather simple' (Section II-D); no validation links them to downstream regression testing cost or real-world SDC failures.
domain assumption Random selection with probability 0.5 is a meaningful baseline for comparison.
Section II-A.3 describes the example random selector; no theoretical or empirical justification is given that p=0.5 represents a reasonable default selector.

how reviews work

0 comments

Cite this review

Pith. "Pith review of ICST Tool Competition 2025 -- Self-Driving Car Testing Track." pith.science (2026). https://pith.science/paper/RSZXYLVY

@misc{pith2026250209982,
  author       = {Pith},
  title        = {Pith review of: ICST Tool Competition 2025 -- Self-Driving Car Testing Track},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RSZXYLVY}},
  note         = {Machine review of arXiv:2502.09982}
}

read the original abstract

This is the first edition of the tool competition on testing self-driving cars (SDCs) at the International Conference on Software Testing, Verification and Validation (ICST). The aim is to provide a platform for software testers to submit their tools addressing the test selection problem for simulation-based testing of SDCs, which is considered an emerging and vital domain. The competition provides an advanced software platform and representative case studies to ease participants' entry into SDC regression testing, enabling them to develop their initial test generation tools for SDCS. In this first edition, the competition includes five tools from different authors. All tools were evaluated using (regression) metrics for test selection as well as compared with a baseline approache. This paper provides an overview of the competition, detailing its context, framework, participating tools, evaluation methodology, and key findings.

Figures

Figures reproduced from arXiv: 2502.09982 by the authors.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

9 extracted references · 7 canonical work pages

[1]

11em plus .33em minus .07em 4000 4000 100 4000 4000 500 `\.=1000 = #1 \@IEEEnotcompsoconly \@IEEEcompsoconly #1 * [1] 0pt [0pt][0pt] #1 * [1] 0pt [0pt][0pt] #1 * \| ** #1 \@IEEEauthorblockNstyle \@IEEEcompsocnotconfonly \@IEEEauthorblockAstyle \@IEEEcompsocnotconfonly \@IEEEcompsocconfonly \@IEEEauthordefaulttextstyle \@IEEEcompsocnotconfonly \@IEEEauthor...

work page
[2]

Khatiri, S

S. Khatiri, S. Panichella, and P. Tonella, ``Simulation-based test case generation for unmanned aerial vehicles in the neighborhood of real flights,'' in IEEE Conference on Software Testing, Verification and Validation, ICST 2023, Dublin, Ireland, April 16-20, 2023 . 1em plus 0.5em minus 0.4em IEEE , 2023, pp. 281--292. [Online]. Available: https://doi.or...

work page arXiv 2023
[3]

Yoo and M

S. Yoo and M. Harman, ``Regression testing minimization, selection and prioritization: a survey,'' Softw. Test. Verification Reliab., vol. 22, no. 2, pp. 67--120, 2012. [Online]. Available: https://doi.org/10.1002/stv.430

doi:10.1002/stv.430 2012
[4]

Birchler, S

C. Birchler, S. Klikovits, M. Fazzini, and S. Panichella, ``Icst 2025 sdc tool competition results,'' Jan. 2025. [Online]. Available: https://doi.org/10.5281/zenodo.14733136

work page doi:10.5281/zenodo.14733136 2025
[5]

Birchler, C

C. Birchler, C. Rohrbach, T. Kehrer, and S. Panichella, ``Sensodat: Simulation-based sensor dataset of self-driving cars,'' in 21st IEEE/ACM International Conference on Mining Software Repositories, MSR 2024, Lisbon, Portugal, April 15-16, 2024 , D. Spinellis, A. Bacchelli, and E. Constantinou, Eds. 1em plus 0.5em minus 0.4em ACM , 2024, pp. 510--514. [On...

work page arXiv 2024
[9]

Panichella, A

S. Panichella, A. Gambi, F. Zampetti, and V. Riccio, `` SBST tool competition 2021,'' in International Workshop on Search-Based Software Testing. 1em plus 0.5em minus 0.4em IEEE , 2021, pp. 20--27. [Online]. Available: https://doi.org/10.1109/SBST52555.2021.00011

work page arXiv 2021
[10]

Castellano, S

E. Castellano, S. Klikovits, A. Cetinkaya, and P. Arcaini, ``Freneticv at the SBST 2022 tool competition,'' in 15th IEEE/ACM International Workshop on Search-Based Software Testing, SBST@ICSE 2022, Pittsburgh, PA, USA, May 9, 2022 . 1em plus 0.5em minus 0.4em IEEE , 2022, pp. 47--48. [Online]. Available: https://doi.org/10.1145/3526072.3527532

work page arXiv 2022
[11]

Aghababaeyan, M

Z. Aghababaeyan, M. Abdellatif, L. C. Briand, R. S., and M. Bagherzadeh, ``Black-box testing of deep neural networks through test case diversity,'' IEEE Trans. Software Eng. , vol. 49, no. 5, pp. 3182--3204, 2023. [Online]. Available: https://doi.org/10.1109/TSE.2023.3243522

work page arXiv 2023

Show all 9 references

[12]

Birchler, T

C. Birchler, T. K. Mohammed, P. Rani, T. Nechita, T. Kehrer, and S. Panichella, ``How does simulation-based testing for self-driving cars match human perception?'' Proc. ACM Softw. Eng. , vol. 1, no. FSE , pp. 929--950, 2024. [Online]. Available: https://doi.org/10.1145/3643768

2024 doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

11em plus .33em minus .07em 4000 4000 100 4000 4000 500 `\.=1000 = #1 \@IEEEnotcompsoconly \@IEEEcompsoconly #1 * [1] 0pt [0pt][0pt] #1 * [1] 0pt [0pt][0pt] #1 * \| ** #1 \@IEEEauthorblockNstyle \@IEEEcompsocnotconfonly \@IEEEauthorblockAstyle \@IEEEcompsocnotconfonly \@IEEEcompsocconfonly \@IEEEauthordefaulttextstyle \@IEEEcompsocnotconfonly \@IEEEauthor...

work page

[2] [2]

Khatiri, S

S. Khatiri, S. Panichella, and P. Tonella, ``Simulation-based test case generation for unmanned aerial vehicles in the neighborhood of real flights,'' in IEEE Conference on Software Testing, Verification and Validation, ICST 2023, Dublin, Ireland, April 16-20, 2023 . 1em plus 0.5em minus 0.4em IEEE , 2023, pp. 281--292. [Online]. Available: https://doi.or...

work page arXiv 2023

[3] [3]

Yoo and M

S. Yoo and M. Harman, ``Regression testing minimization, selection and prioritization: a survey,'' Softw. Test. Verification Reliab., vol. 22, no. 2, pp. 67--120, 2012. [Online]. Available: https://doi.org/10.1002/stv.430

doi:10.1002/stv.430 2012

[4] [4]

Birchler, S

C. Birchler, S. Klikovits, M. Fazzini, and S. Panichella, ``Icst 2025 sdc tool competition results,'' Jan. 2025. [Online]. Available: https://doi.org/10.5281/zenodo.14733136

work page doi:10.5281/zenodo.14733136 2025

[5] [5]

Birchler, C

C. Birchler, C. Rohrbach, T. Kehrer, and S. Panichella, ``Sensodat: Simulation-based sensor dataset of self-driving cars,'' in 21st IEEE/ACM International Conference on Mining Software Repositories, MSR 2024, Lisbon, Portugal, April 15-16, 2024 , D. Spinellis, A. Bacchelli, and E. Constantinou, Eds. 1em plus 0.5em minus 0.4em ACM , 2024, pp. 510--514. [On...

work page arXiv 2024

[6] [9]

Panichella, A

S. Panichella, A. Gambi, F. Zampetti, and V. Riccio, `` SBST tool competition 2021,'' in International Workshop on Search-Based Software Testing. 1em plus 0.5em minus 0.4em IEEE , 2021, pp. 20--27. [Online]. Available: https://doi.org/10.1109/SBST52555.2021.00011

work page arXiv 2021

[7] [10]

Castellano, S

E. Castellano, S. Klikovits, A. Cetinkaya, and P. Arcaini, ``Freneticv at the SBST 2022 tool competition,'' in 15th IEEE/ACM International Workshop on Search-Based Software Testing, SBST@ICSE 2022, Pittsburgh, PA, USA, May 9, 2022 . 1em plus 0.5em minus 0.4em IEEE , 2022, pp. 47--48. [Online]. Available: https://doi.org/10.1145/3526072.3527532

work page arXiv 2022

[8] [11]

Aghababaeyan, M

Z. Aghababaeyan, M. Abdellatif, L. C. Briand, R. S., and M. Bagherzadeh, ``Black-box testing of deep neural networks through test case diversity,'' IEEE Trans. Software Eng. , vol. 49, no. 5, pp. 3182--3204, 2023. [Online]. Available: https://doi.org/10.1109/TSE.2023.3243522

work page arXiv 2023