Pith. sign in

REVIEW 5 major objections 5 minor 60 references

Using Causal Inference to Test Systems with Hidden and Interacting Variables: An Evaluative Case Study

T0 review · 5 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Causal testing can produce reliable test outcomes for systems with interacting and unobservable variables, as demonstrated on the CARLA driving simulator.

desk verdict A legitimate extension of causal testing to hidden and interacting variables, but the instrumental-variable case study does not support the paper's headline claim. read the letter →

arxiv 2504.16526 v2 pith:EPFIB4AV submitted 2025-04-23 cs.SE

Michael Foster , Robert M. Hierons , Donghwan Shin , Neil Walkinshaw , Christopher Wild This is my paper · ORCID
classification cs.SE
keywords CausalTestingInferenceSoftwareInstrumentalVariablesEffectModificationUnobservableAutonomousDrivingCARLA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that Causal Testing—checking software requirements by estimating causal effects from observational run data—can be extended to systems where variables interact and where some relevant variables never appear in the logs. The authors add two devices from causal inference: an interaction term in the estimator to account for effect modification (when the effect of one variable depends on the value of another), and an instrumental-variable ratio to neutralize unobserved confounders. They evaluate the approach on three requirements of the CARLA driving simulator, using two autonomous driving systems and their built-in scenarios as the test data. The approach yields the expected verdicts for all three requirements: exact infraction penalties, a real but unexpected vehicle-model effect on infractions, and a speed regression between CARLA versions—while using less, and less controlled, data than statistical metamorphic testing needs. If the result holds generally, expensive nondeterministic systems could be tested from pre-existing logs, and regression testing could proceed even when logging infrastructure is incomplete.

What carries the argument

The load-bearing object is the hand-drawn causal directed acyclic graph (DAG) in Figure 2, which specifies which system variables may directly affect which others and which variables are unobserved. The DAG is used twice: identification, where backdoor paths are found and closed by stratification or adjustment, and estimation, where linear regression is fitted with the identified adjustment set. For RE1 the adjustment takes the form of an interaction term $\text{CompletionScore} \times \text{OutsideLane}$, which absorbs the effect modification coming from the leaderboard's multiplicative penalty formula. For RE3 the DAG licenses an instrumental-variable design: route length (the instrument) affects simulation time, which in turn affects system time, while route length has no direct arrow to system time and no arrow from the unobserved pedestrian and vehicle counts; the causal effect is then the IV ratio $\beta_{\text{RouteLength}\to\text{SystemTime}} / \beta_{\text{RouteLength}\to\text{SimulationTime}}$. The DAG thus performs the role of test oracle: the test verdict is a comparison between the estimated causal effect and the effect the DAG says the system should have.

What would settle it

Record the numbers of spawned pedestrians and NPC vehicles during the CARLA runs (as the paper does to build its gold standard), estimate the simulation-time-to-system-time effect by ordinary adjustment, and compare it with the instrumental-variable estimate; if the two differ by substantially more than the confidence-interval overlap, then the exclusion restriction or linearity assumption behind the instrument fails. A more direct check is to add the term $\text{RouteLength}\to\text{SystemTime}$ to the regression and test whether its coefficient is nonzero, since a nonzero direct effect of the instrument on the outcome invalidates the IV ratio in Equation (4c).

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that two established causal-inference concepts remove the two standing limitations of prior Causal Testing work: interacting variables and unobservable variables. For interaction, the paper shows that adding the product term $\text{CompletionScore} \times \text{OutsideLane}$ to the regression estimator isolates the direct effect of completion score on driving score: all five infraction-penalty test cases pass exactly, while omitting the interaction term makes two correct test cases fail for the trained driver. For hidden variables, the paper uses route length as an instrumental variable and estimates the causal effect of simulation time on system time by the ratio $\beta_{\text{RouteLength}\to\text{SystemTime}} / \beta_{\text{RouteLength}\to\text{SimulationTime}}$. The resulting estimates flag CARLA v0.9.11 as slower than v0.9.10.1 for all four driving agents, matching a gold-standard adjustment obtained by modifying the code to log the hidden pedestrian and vehicle counts; and the same analysis pipeline surfaces a vehicle-model effect on infractions that standard testing would not have revealed. The paper's conclusion is that reliable test outcomes are obtainable without large highly controlled datasets or code instrumentation, provided the user's causal model satisfies the identifying assumptions.

Load-bearing premise

The load-bearing premise is that the manually drawn causal map (Figure 2) is correct, and for the instrumental-variable requirement in particular that route length has no direct effect on system time, that no hidden confounder affects route length, and that the relationships among the variables are linear.

Editorial extensions

If this is right

  • One observational dataset can be reused across multiple requirements and multiple driving agents, so adding test properties does not require collecting new controlled data for each one.
  • Adding interaction terms to the estimator prevents false test failures caused by effect modification, letting a tester query a direct causal effect instead of an aggregated association.
  • Instrumental-variable estimation makes regression testing possible when confounders are not logged, as long as a valid instrument such as route length is available and the linearity assumptions hold.
  • The same pipeline can expose unexpected behaviours—such as the ego-vehicle model changing infraction rates and a CARLA version regression absent from the changelog—that would be missed by scenario-based testing or by plain association.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Outside the paper's data, the closeness between the unadjusted and gold-standard estimates in RE3 suggests the hidden confounders happened to have little bias here; the reliable verdict in this case does not mean adjustment can be skipped in general, since the analyst usually cannot tell which estimate is closer.
  • The same instrumental-variable recipe should transfer to other simulators or cyber-physical systems where a controllable proxy (scenario length, episode size, route distance) drives the treatment and has no plausible direct path to the outcome; replicating the CARLA result on a second simulator with instrumented ground truth would test that transfer.
  • Since the hand-drawn DAG is the test oracle, the method's scope is bounded by the domain knowledge in that DAG; learning the graph from the same data would undermine the oracle because the learned structure would encode the very faults under test.
  • An IV estimate that is causally valid but less precise will be most useful when the requirement is qualitative ('new version is not slower') rather than when a precise effect size is the acceptance criterion.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper extends Causal Testing, a previously proposed approach for testing nondeterministic software, to two new settings: systems with interacting variables and systems with unobservable variables. It introduces effect modification terms and instrumental variable (IV) methods into an existing causal testing tool and evaluates the approach on the CARLA autonomous driving simulator. Three requirements are tested: (RE1) the infraction penalty formula, (RE2) the causal effect of ego-vehicle model on infractions, and (RE3) the effect of CARLA version on simulation performance. For RE1, an interaction-adjusted regression recovers the documented penalties; for RE2, three of four agents show a significant effect of vehicle model; for RE3, IV, gold-standard adjustment, and no-adjustment estimates are compared. The paper concludes that reliable test outcomes can be obtained even when variables interact and are not recorded, and provides a replication package.

Significance. If the conclusions were fully supported, the paper would be a useful extension of Causal Testing to realistic settings with hidden confounders and interacting variables, and the case study would offer practical guidance for ADS testing. The study is clearly described, uses a realistic high-fidelity simulator, and ships a public replication package. However, the RE3 evidence is not fully supportive of the central claims: the IV estimates disagree with the paper's own gold standard on one agent and flip a pass/fail decision, while the no-adjustment estimates are nearly identical to the gold standard, so the case study does not demonstrate the claimed advantage of IV in this setting. The contribution is therefore plausible but requires substantial re-analysis and a more cautious framing.

major comments (5)
  1. [Section 5.5.2 / Table 3 / Section 6.3] The claim that 'IV estimates still produce reliable test outcomes' is contradicted by the paper's own gold-standard comparison for the CARLA Garage privileged driver. The IV confidence intervals for v0.9.10.1 and v0.9.11 are [7.389, 8.100] and [8.114, 8.617], which do not overlap, so the test fails; the gold-standard intervals [6.883, 7.441] and [7.398, 8.231] do overlap, so the test passes. Thus the IV method flips a pass/fail decision for one of the four drivers, and the paper should not state that IV gives reliable test outcomes without accounting for this.
  2. [Section 5.5.2 / Figure 2] The IV identifying assumptions for RE3 are asserted from Figure 2 but never tested. Specifically, RouteLength must have no direct effect on SystemTime, no unobserved common cause with SimulationTime or SystemTime, and all relations must be linear. The paper itself notes that linearity 'cannot be known for sure', and the large discrepancies between IV and gold-standard estimates for CARLA Garage privileged (0.589 on v0.9.10.1 and 0.569 on v0.9.11) suggest that at least one assumption is violated. The authors should provide empirical checks, such as a test for a direct RouteLength-to-SystemTime path or a comparison of first-stage strength, or explicitly limit the RE3 conclusions.
  3. [Section 5.5.2 / Table 3 / Section 6.3] The no-adjustment estimates are almost identical to the gold-standard estimates (median error 0.001, versus 0.31 for IV), so the case study data do not exhibit the confounding that IV is supposed to remove. This is especially clear for the CARLA Garage drivers, where manual code inspection shows NPCvehicles is constant, making Equations (5) and (6) effectively equivalent. The statement in Section 6.3 that IV 'adjusts for the bias' is therefore not demonstrated; the data do not show a bias that IV removes.
  4. [Section 5.3.2 / Equation (2) / Table 1] The RE1 estimator in Equation (2) has the same functional form as the leaderboard formula in Equation (1), so the exact matches and degenerate confidence intervals in the top half of Table 1 are a consistency check of the estimator against CARLA's documented formula rather than an independent demonstration that effect modification is correctly handled. This does not invalidate the useful contrast with Equation (3), but the paper should present RE1 as a sanity check and rest the RQ1 evidence primarily on the failures observed when the interaction term is omitted.
  5. [Section 5.4.2 / Table 2] RE2 is described as testing the effect of ego-vehicle model on the number of infractions, but the outcome used in the regression is the numeric infraction penalty (1.0, 0.7, 0.65, 0.6, 0.5), which is a coding of a categorical variable. The conclusions, including the unexpected result for the TCP privileged driver, could depend on this scoring choice. The paper should either justify the penalty scale as the intended outcome or report a robustness analysis using a count or binary outcome.
minor comments (5)
  1. [Section 4.2] The word 'charactesized' should be 'characterized'.
  2. [References [3] and [4]] The access dates are given as 'Accessed 2024-19-03', which is not a valid date; these should be corrected to a standard form such as 'Accessed 2024-03-19'.
  3. [Section 5.3.2] The text says 'ComplectionScore' in the discussion of the interaction term; this should be 'CompletionScore'.
  4. [Sections 5.5.1 and 5.5.2] The expected effect for RE3 is stated as 'not positive' in Section 5.5.1 but as 'zero' in the Test Outcomes paragraph of Section 5.5.2; these statements should be reconciled.
  5. [Section 5.5.2 / Equation (4)] The symbols for the path coefficients in Equations (4a)-(4c) are typeset inconsistently with the notation used elsewhere in the paper; using a single consistent coefficient notation would improve readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: RE1 is a conformance check using an external oracle, and RE2/RE3 are evaluated with independent estimators against a separately constructed gold standard.

full rationale

This paper is an evaluative case study, not a closed derivation, and its load-bearing results do not reduce to their own inputs. The closest candidate is RE1: the regression estimator in Equation (2) has the same functional form as the CARLA leaderboard formula in Equation (1), with the expected infraction penalty replaced by the coefficient alpha_1. That is a deliberately specified conformance test, not a self-definitional prediction. The expected effect is taken from the external leaderboard oracle; the coefficient is estimated from CARLA execution data, and the match is conditional on the implementation following the formula. A buggy penalty would produce a different estimate and a failing test, so the comparison is informative rather than tautological. The contrast between Equation (2) and the misspecified Equation (3) is an algebraic illustration that omitting an interaction biases the coefficient; it does not derive the method's reliability from the method's own assumptions. RE2 is an independent regression with a zero-effect oracle. RE3 uses the standard IV estimator in Equation (4c) and compares it with a gold standard in Equation (5) obtained by instrumenting TCP and by manual code inspection for CARLA Garage; the IV estimates are computed from separate data and can disagree with the gold standard (e.g., the Garage-privileged test outcome flips from pass to fail). The paper's reliance on the authors' prior Causal Testing papers ([15], [16]) is lineage for the DAG-construction and test-case vocabulary, while the claims here are tested on fresh CARLA data and standard causal-inference theorems. Section 7 explicitly flags the internal-validity risk that the chosen requirements favour the technique, and Section 6.3 reports that unadjusted estimation was closer to the gold standard than IV (errors 0.001 vs 0.31). These are validity and correctness concerns, not evidence that any estimate is defined in terms of its own target. No load-bearing self-citation chain or ansatz smuggled via citation is present.

Assumptions & free parameters 1 free parameters · 6 assumptions · 0 invented entities

The central claim rests on a hand-drawn causal DAG and several untested domain assumptions, especially in the instrumental variable analysis. No invented entities are introduced. The only explicit hand-picked number is the 5% tolerance for RE1 test outcomes.

free parameters (1)
  • Test tolerance for RE1 pass/fail = within 5%
    A test case fails if the estimate is not within 5% of the expected penalty. This threshold is chosen by the authors and is not derived from any principle (Section 5.3.2, Test Outcomes).
assumptions (6)
  • domain assumption Causal DAG in Figure 2 correctly represents all causal relationships, including absence of edges.
    Hand-drawn from documentation and domain knowledge; used for all identification steps in Sections 5.3-5.5. If an edge is missing or extra, the adjustment sets and IV estimates are invalid.
  • domain assumption RouteLength is a valid instrument for SimulationTime on SystemTime: no direct effect on SystemTime, independent of unobserved confounders, and relevant.
    Required for the IV ratio in Equation (4a-4c) in Section 5.5.2; not empirically verified in the paper.
  • domain assumption The relationships in RE3 are linear, so the ratio of regression coefficients recovers the causal effect.
    The authors state 'While we cannot know for sure that the relationships are linear' (Section 5.5.2) but proceed with the ratio interpretation.
  • domain assumption The 'gold standard' adjustment, using recorded pedestrians and NPC counts from modified TCP and code inspection of CARLA Garage, is accurate.
    The gold standard is an artificial construct described in Section 5.2; if it is wrong, comparisons of IV and no-adjustment estimates in Table 3 are invalid.
  • domain assumption Test data satisfies the positivity assumption.
    Stated in Section 3.2 as fundamental to causal testing; no verification is provided for the CARLA dataset.
  • ad hoc to paper Infraction can be represented as a numeric outcome using its penalty values.
    In Section 5.4.2, the outcome Infraction is coded as the numeric penalty for linear regression, which assumes interval-scale properties for an ordinal categorical variable.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Using Causal Inference to Test Systems with Hidden and Interacting Variables: An Evaluative Case Study." pith.science (2026). https://pith.science/paper/EPFIB4AV

@misc{pith2026250416526,
  author       = {Pith},
  title        = {Pith review of: Using Causal Inference to Test Systems with Hidden and Interacting Variables: An Evaluative Case Study},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EPFIB4AV}},
  note         = {Machine review of arXiv:2504.16526}
}
read the original abstract

Software systems with large parameter spaces, nondeterminism and high computational cost are challenging to test. Recently, software testing techniques based on causal inference have been successfully applied to systems that exhibit such characteristics, including scientific models and autonomous driving systems. One significant limitation is that these are restricted to test properties where all of the variables involved can be observed and where there are no interactions between variables. In practice, this is rarely guaranteed; the logging infrastructure may not be available to record all of the necessary runtime variable values, and it can often be the case that an output of the system can be affected by complex interactions between variables. To address this, we leverage two additional concepts from causal inference, namely effect modification and instrumental variable methods. We build these concepts into an existing causal testing tool and conduct an evaluative case study which uses the concepts to test three system-level requirements of CARLA, a high-fidelity driving simulator widely used in autonomous vehicle development and testing. The results show that we can obtain reliable test outcomes without requiring large amounts of highly controlled test data or instrumentation of the code, even when variables interact with each other and are not recorded in the test data.

Figures

Figures reproduced from arXiv: 2504.16526 by the authors.

Figure 1
Figure 1. General setup for IVs. The unobserved variable [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. The causal DAG for all three requirements, with the [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

60 extracted references · 30 canonical work pages

  1. [1]

    Non determinism of Walker AI contr ollers

    Accessed 2023-05-05. Non determinism of Walker AI contr ollers. https://github.com/carla-simulator/carla/issues/349 3

    work page 2023

  2. [2]

    CARLA changelog

    Accessed 2024-02-27. CARLA changelog . https://github.com/carla-simulator/carla/blob/maste r/CHANGELOG.md Using Causal Inference to Test Systems with Hidden and Inter acting Variables: An Evaluative Case Study

    work page 2024

  3. [3]

    CARLA Autonomous Driving Leaderb oard

    Accessed 2024-19-03. CARLA Autonomous Driving Leaderb oard. https://leaderboard.carla.org

    work page 2024

  4. [4]

    CARLA simulator

    Accessed 2024-19-03. CARLA simulator. https://github.com/carla-simulator/carla

    work page 2024

  5. [5]

    Aitor Arrieta, Jon Ayerdi, Miren Illarramendi, Aitor Ag irre, Goiuria Sagardui, and Maite Arratibel. 2021. Using machine learning to build t est oracles: an industrial case study on elevators dispatching algorithms . In 2021 IEEE/ACM International Conference on Automation of Software Test (A ST). IEEE, 30–39

    work page 2021

  6. [6]

    Baah, Andy Podgurski, and Mary Jean Harrold

    George K. Baah, Andy Podgurski, and Mary Jean Harrold. 20 11. Mitigating the Confounding Effects of Program Dependences for Effective Fault Local- ization. In Proceedings of the 19th ACM SIGSOFT Symposium and the 13th Eu- ropean Conference on Foundations of Software Engineering (Szeged, Hungary) (ESEC/FSE ’11) . Association for Computing Machinery, New York...

    arXiv

  7. [7]

    Baah, Andy Podgurski, and Mary Jean Harrold

    George K. Baah, Andy Podgurski, and Mary Jean Harrold. 20 10. Causal Inference for Statistical Fault Localization. In Proceedings of the 19th In- ternational Symposium on Software Testing and Analysis (Trento, Italy) (IS- STA ’10) . Association for Computing Machinery, New York, NY, USA, 73 –84. https://doi.org/10.1145/1831708.1831717

    arXiv

  8. [8]

    Zhuofu Bai, Gang Shu, and Andy Podgurski. 2015. NUMFL: Lo calizing Faults in Numerical Software Using a Value-Based Causal Model. In 2015 IEEE 8th Inter- national Conference on Software Testing, Verification and Validation (ICST). IEEE, 1–10. https://doi.org/10.1109/ICST.2015.7102597

    work page arXiv 2015

Show all 60 references
  1. [9]

    Elias Bareinboim and Judea Pearl. 2016. Causal inferenc e and the data-fusion problem. Proceedings of the National Academy of Sciences of the United States of America 113, 27 (2016), 7345–7352

    2016

  2. [10]

    Barr, Mark Harman, Phil McMinn, Muzammil Shahba z, and Shin Yoo

    Earl T. Barr, Mark Harman, Phil McMinn, Muzammil Shahba z, and Shin Yoo. 2015. The Oracle Problem in Software Testing: A Sur - vey. IEEE Transactions on Software Engineering 41, 5 (2015), 507–525. https://doi.org/10.1109/TSE.2014.2372785

    2015

  3. [11]

    Boyuan Chen and Zhen Ming (Jack) Jiang. 2021. A Survey of Soft- ware Log Instrumentation. Comput. Surveys 54, 4 (may 2021), 1–34. https://doi.org/10.1145/3448976

    2021 doi

  4. [12]

    Chen, Shing C

    Tsong Y. Chen, Shing C. Cheung, and Shiu Ming Yiu. 1998. Metamorphic testing: A new approach for generating next test cases. Technical Report HKUST-CS98-01. The Hong Kong University of Science and Technology

    1998

  5. [13]

    Poskitt, Jun Sun, Sridhar Ade pu, and Fan Zhang

    Yuqi Chen, Christopher M. Poskitt, Jun Sun, Sridhar Ade pu, and Fan Zhang

  6. [14]

    Kwang Ting Cheng and A. S. Krishnakumar. 1993. Automati c functional test generation using the extended finite state machine mode l. In Proceedings of the 30th International Design Automation Conference (Dallas, Texas, USA) (DAC ’93). Association for Computing Machinery, New Yo...

    1993

  7. [15]

    Clark, Michael Foster, Benedikt Prifling, Nei l Walkinshaw, Robert M

    Andrew G. Clark, Michael Foster, Benedikt Prifling, Nei l Walkinshaw, Robert M. Hierons, Volker Schmidt, and Robert D. Turner. 2023. Testin g Causality in Sci- entific Modelling Software. ACM Trans. Softw. Eng. Methodol. 33, 1, Article 10 (nov 2023), 42 pages. https://doi.org/10...

    2023 doi

  8. [16]

    Clark, Michael Foster, Neil Walkinshaw, and R obert M

    Andrew G. Clark, Michael Foster, Neil Walkinshaw, and R obert M. Hi- erons. 2023. Metamorphic Testing with Causal Graphs. In 2023 IEEE Conference on Software Testing, Verification and Validatio n (ICST) . 153–164. https://doi.org/10.1109/ICST57152.2023.00023

    2023

  9. [17]

    Alexey Dosovitskiy, German Ros, Felipe Codevilla, Ant onio Lopez, and Vladlen Koltun. 2017. CARLA: An Open Urban Driving Simulator. In Proceedings of the 1st Annual Conference on Robot Learning (Proceedings of M achine Learning Research, Vol. 78). PMLR, 1–16

    2017

  10. [18]

    Hierons, a nd Neil Walkin- shaw

    Michael Foster, Christopher Wild, Robert M. Hierons, a nd Neil Walkin- shaw. 2024. Causal Test Adequacy. In 2024 IEEE Conference on Software Testing, Verification and Validation (ICST) . IEEE, 161–172. https://doi.org/10.1109/icst60714.2024.00023

    2024

  11. [19]

    Freedman

    R.S. Freedman. 1991. Testability of software componen ts. IEEE Transactions on Software Engineering 17, 6 (1991), 553–564. https://doi.org/10.1109/32.87281

    1991 doi

  12. [20]

    Carlo A Furia, Richard Torkar, and Robert Feldt. 2023. T owards causal analysis of empirical software engineering data: The impact of progr amming languages on coding competitions. ACM Transactions on Software Engineering and Method- ology 33, 1 (2023), 1–35

    2023

  13. [21]

    Luca Giamattei, Antonio Guerriero, Roberto Pietrantu ono, and Stefano Russo. 2024. Causality-driven Testing of Autonomous Drivi ng Systems. ACM Trans. Softw. Eng. Methodol. 33, 3, Article 74 (2024), 35 pages. https://doi.org/10.1145/3635709

    2024 doi

  14. [22]

    Luca Giamattei, Antonio Guerriero, Roberto Pietrantu ono, and Stefano Russo

  15. [23]

    Clark Glymour, Kun Zhang, and Peter Spirtes. 2019. Revi ew of causal discovery methods based on graphical models. Frontiers in genetics 10 (2019), 524

    2019

  16. [24]

    Reynolds

    Ross Gore and Paul F. Reynolds. 2012. Reducing confound ing bias in predicate-level statistical debugging metrics. In 2012 34th Inter- national Conference on Software Engineering (ICSE) . IEEE, 463–473. https://doi.org/10.1109/ICSE.2012.6227169

    2012

  17. [25]

    Ralph Guderlei and Johannes Mayer. 2007. Statistical M etamorphic Testing Test- ing Programs with Random Output by Means of Statistical Hypothesis Tests and Metamorphic Testing. In Seventh International Conference on Quality Software (QSIC 2007). 404–409. https://doi.org/10.11...

    2007

  18. [26]

    Jia Cheng Han and Zhi Quan Zhou. 2020. Metamorphic Fuzz T esting of Au- tonomous Vehicles. In Proceedings of the IEEE/ACM 42nd International Con- ference on Software Engineering Workshops (Seoul, Republic of Korea) (IC- SEW’20). Association for Computing Machinery, New York, N...

    2020

  19. [27]

    Fitash Ul Haq, Donghwan Shin, and Lionel Briand. 2022. E fficient Online Test- ing for DNN-Enabled Systems Using Surrogate-Assisted and M any-Objective Optimization. In Proceedings of the 44th International Conference on Software En- gineering (ICSE ’22). Association for Computi...

    2022

  20. [28]

    Helmbold and C.E

    D.P. Helmbold and C.E. McDowell. 1996. A Taxonomy of Rac e Conditions. J. Parallel and Distrib. Comput. 33, 2 (1996), 159–164. https://doi.org/10.1006/jpdc.1996.0034

    1996

  21. [29]

    Miguel A Hernán and James M Robins. 2020. Causal Inference: What if. Chapman & Hall/CRC, Boca Raton

    2020

  22. [30]

    Bernhard Jaeger, Kashyap Chitta, and Andreas Geiger. 2 023. Hidden Biases of End-to-End Driving Models. In 2023 IEEE/CVF International Conference on Com- puter Vision (ICCV) . IEEE. https://doi.org/10.1109/iccv51070.2023.00757

    2023

  23. [31]

    Zhengmin Jiang, Jia Liu, Peng Sun, Ming Sang, Huiyun Li, and Yi Pan. 2024. Generation of Risky Scenarios for Testing Automated Drivin g Visual Percep- tion Based on Causal Analysis. IEEE Transactions on Intelligent Transportation Systems 25, 11 (2024), 15991–16004. https://doi...

    2024

  24. [32]

    Brittany Johnson, Yuriy Brun, and Alexandra Meliou. 20 20. Causal testing: un- derstanding defects’ root causes. In Proceedings of the ACM/IEEE 42nd Interna- tional Conference on Software Engineering . 87–99

  25. [33]

    Prabhjot Kaur, Samira Taghavi, Zhaofeng Tian, and Weis ong Shi. 2021. A Survey on Simulators for Testing Self-Driving Cars. In 2021 Fourth Interna- tional Conference on Connected and Autonomous Driving (Met roCAD). 62–70. https://doi.org/10.1109/MetroCAD51599.2021.00018

    2021

  26. [34]

    Seongmin Lee, Dave Binkley, Robert Feldt, Nicolas Gold , and Shin Yoo. 2021. Causal program dependence analysis. arXiv preprint arXiv:2104.09107 (2021)

    2021 arXiv

  27. [35]

    Daniel Malinsky and David Danks. 2018. Causal discover y algorithms: A prac- tical guide. Philosophy Compass 13, 1 (2018), e12470

    2018

  28. [36]

    John McConnell and Stephan Lindner

    K. John McConnell and Stephan Lindner. 2019. Estimatin g treatment effects with machine learning. Health Services Research 54, 6 (oct 2019), 1273–1282. https://doi.org/10.1111/1475-6773.13212

    2019

  29. [37]

    Changhai Nie and Hareton Leung. 2011. A survey of combin atorial testing. ACM Comput. Surv. 43, 2, Article 11 (feb 2011), 29 pages. https://doi.org/10.1145/1883612.1883618

    2011

  30. [38]

    Nitsche, R.H

    P. Nitsche, R.H. Welsh, A. Genser, and P.D. Thomas. 2018 . A novel, modular val- idation framework for collision avoidance of automated veh icles at road junc- tions. In 2018 21st International Conference on Intelligent Transpo rtation Systems (ITSC). 90–97. https://doi.org/10...

    2018

  31. [39]

    Justin Norden, Matthew O’Kelly, and Aman Sinha. 2019. E f- ficient Black-box Assessment of Autonomous Vehicle Safety. https://doi.org/10.48550/ARXIV.1912.03618

    2019 doi arXiv

  32. [40]

    Sheila F O’Brien and Qi Long Yi. 2016. How do I interpret a confidence interval? Transfusion 56, 7 (2016), 1680–1683

    2016

  33. [41]

    Judea Pearl. 2009. Causality. Cambridge university press, Cambridge

    2009

  34. [42]

    Andy Podgurski and Yiğit Küçük. 2020. CounterFault: Va lue-Based Fault Lo- calization by Modeling and Predicting Counterfactual Outc omes. In 2020 IEEE International Conference on Software Maintenance and Evol ution (ICSME). IEEE, 382–393

    2020

  35. [43]

    Poskitt, Yuqi Chen, Jun Sun, and Yu Jiang

    Christopher M. Poskitt, Yuqi Chen, Jun Sun, and Yu Jiang . 2023. Find- ing Causally Different Tests for an Industrial Control Syste m. In 2023 IEEE/ACM 45th International Conference on Software Engine ering (ICSE) . 2578–

    2023

  36. [44]

    Paul Ralph et al. 2021. Empirical Standards for Softwar e Engineering Research. arXiv:2010.03525 [cs.SE]

    2021

  37. [45]

    Per Runeson, Martin Host, Austen Rainer, and Bjorn Regn ell. 2012. Case study research in software engineering: Guidelines and examples . John Wiley & Sons

    2012

  38. [46]

    Gang Shu, Boya Sun, Andy Podgurski, and Feng Cao. 2013. M fl: Method-level fault localization with causal inference. In 2013 IEEE Sixth International Confer- ence on Software Testing, Verification and Validation. IEEE, IEEE, 124–133

    2013

  39. [47]

    Julien Siebert. 2023. Applications of statistical cau sal inference in software engineering. Information and Software Technology 159 (jul 2023), 107198. https://doi.org/10.1016/j.infsof.2023.107198

    2023

  40. [48]

    Dan Siroker and Pete Koomen. 2015. A/B testing: The most powerful way to turn clicks into customers . John Wiley & Sons. Michael Foster, Robert M. Hierons, Donghwan Shin, Neil Walki nshaw, and Christopher Wild

    2015

  41. [49]

    Jian Sun, He Zhang, Huajun Zhou, Rongjie Yu, and Ye Tian. 2022. Scenario-Based Test Automation for Highly Automated Vehic les: A Re- view and Paving the Way for Systematic Safety Assurance. IEEE Trans- actions on Intelligent Transportation Systems 23, 9 (2022), 14088–14103. ht...

    2022

  42. [50]

    Shuncheng Tang, Zhenya Zhang, Yi Zhang, Jixiang Zhou, Y an Guo, Shuang Liu, Shengjian Guo, Yan-Fu Li, Lei Ma, Yinxing Xue, and Yang L iu. 2023. A Survey on Automated Driving System Testing: Landscapes an d Trends. ACM Trans. Softw. Eng. Methodol. 32, 5, Article 124 (jul 2023),...

    2023 doi

  43. [51]

    Caldiera V

    C. Caldiera V. Basili and D. H. Rombach. 1994. Goal question metric paradigm . Vol. 2. Wiley. 528–532 pages

    1994

  44. [52]

    Weinberg

    Clarice R. Weinberg. 2007. Can DAGs Clarify Effect Modifi cation? Epidemiology 18, 5 (sep 2007), 569–572. https://doi.org/10.1097/ede.0b013e318126c11d

    2007 doi

  45. [53]

    S Wright. 1920. The Relative Importance of Heredity and Environment in De- termining the Piebald Pattern of Guinea-Pigs. Proc Natl Acad Sci U S A 6, 6 (jun 1920), 320–332

    1920

  46. [54]

    Penghao Wu, Xiaosong Jia, Li Chen, Junchi Yan, Hongyang Li, and Yu Qiao. 2022. Trajectory-guided Control Prediction for End-to-end Auto nomous Driving: A Simple yet Strong Baseline. arXiv: 2206.08129 [cs.CV]

    2022 arXiv

  47. [55]

    Xinhai Zhang, Jianbo Tao, Kaige Tan, Martin Törngren, J osé Manuel Gaspar Sánchez, Muhammad Rusyadi Ramli, Xin Tao, Magnus Gyllenham mar, Franz Wotawa, Naveen Mohan, Mihai Nica, and Hermann Felbinger. 20 23. Find- ing Critical Scenarios for Automated Driving Systems: A Sys tem...

    2023

  48. [56]

    Ziyuan Zhong, Yun Tang, Yuan Zhou, Vania de Oliveira Nev es, Yang Liu, and Baishakhi Ray. 2021. A Survey on Scenario-Based Tes t- ing for Automated Driving Systems in High-Fidelity Simulat ion. https://doi.org/10.48550/ARXIV.2112.00964

    2021 doi arXiv

  49. [57]

    Tahereh Zohdinasab, Vincenzo Riccio, Alessio Gambi, a nd Paolo Tonella. 2023. Efficient and Effective Feature Space Exploration for TestingDeep Learning Sys- tems. ACM Trans. Softw. Eng. Methodol. 32, 2, Article 49 (mar 2023), 38 pages. https://doi.org/10.1145/3544792

    2023 doi

  50. [2020]

    In Proceedings of the 34th IEEE/ACM International Confer- ence on Automated Software Engineering (ASE ’19)

    Learning-guided network fuzzing for testing cyber-p hysical sys- tem defences. In Proceedings of the 34th IEEE/ACM International Confer- ence on Automated Software Engineering (ASE ’19) . IEEE Press, 962–973. https://doi.org/10.1109/ASE.2019.00093

  51. [2025]

    Information and Software Technology 178 (Feb

    Causal reasoning in Software Quality Assurance: A sys tematic review. Information and Software Technology 178 (Feb. 2025), 107599. https://doi.org/10.1016/j.infsof.2024.107599

    2025

  52. [2590]

    https://doi.org/10.1109/ICSE48619.2023.00215

    2023

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.