{"id":"f0775971-fe2d-4aaf-bf51-4d6f3ae5afb2","arxiv_id":"2504.16526","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Causal testing with effect modification and instrumental variables can produce reliable test outcomes for software with interacting and unobservable variables, as shown in a CARLA case study.","lead":"Researchers added two causal-inference techniques to software testing: interaction terms for interacting variables and instrumental variables for unobservable ones. They tested the approach on the CARLA self-driving simulator and report reliable outcomes, with caveats about the strength of the evidence.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The RE3 IV result conflicts with the paper's own gold standard: IV estimates deviate by up to ~0.59 seconds (median 0.31) and flip a pass/fail decision, so the asserted identifying assumptions are not validated.","rationale":"The reader's weakest assumption already identifies the RE3 DAG and IV identifying assumptions as the main risk, and I agree: the paper's central claim about unobservable variables depends on RouteLength being a valid instrument. I sharpen the point by noting that the paper's own gold-standard comparison shows IV estimates with median error 0.31 and one test outcome that differs under the paper's comparison rule. The no-adjustment estimator being closer to the gold standard weakens the demonstration that hidden confounders actually needed handling. This is an internal-validity problem rather than a disagreement with the causal-testing paradigm. The paper remains a useful proof-of-concept with a replication package, but the 'reliable test outcomes for hidden variables' claim should be conditional on validating the identifying assumptions. Since the reader already returned a CONDITIONAL verdict, I recommend no change to that verdict.","tokens_in":21140,"tokens_out":10015,"duration_ms":104955,"concrete_test":"Using the replication-package RE3 data, fit SystemTime = beta0 + beta1*SimulationTime + beta2*RouteLength + beta3*Pedestrians + beta4*NPCvehicles + epsilon separately for each CARLA version. If the 95% confidence interval for beta2 excludes zero, the exclusion restriction behind Equation (4c) fails. Then re-estimate the IV effect using two-stage least squares with an intercept and compare it with the paper's ratio-of-coefficients estimates and with the gold standard. If the IV estimates shift beyond the reported confidence intervals, or if beta2 is significant, the RE3 conclusions and the 'reliable outcomes for hidden variables' claim are not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that IV methods give reliable test outcomes for unobservable variables rests on the identifying assumptions for RE3: RouteLength must have no direct effect on SystemTime, no unobserved common cause with SystemTime or SimulationTime, and all relations must be linear. Section 5.5.2 asserts these from Figure 2 but never tests them. The paper's own gold-standard comparison in Table 3 shows the IV estimates deviate substantially: for CARLA Garage privileged, IV gives 7.751 vs the gold-standard 7.162 on v0.9.10.1 and 8.383 vs 7.814 on v0.9.11. Under the paper's non-overlap criterion, IV fails all four drivers, while the gold standard passes the Garage privileged driver. Moreover, the no-adjustment estimator, which requires no unobserved variables, is closer to the gold standard (median error 0.001), so the RE3 data do not actually exhibit confounding that IV removes. The load-bearing assumption that RouteLength is a valid instrument is therefore not just unvalidated; the evidence in the paper suggests it is violated or at least that the case study cannot distinguish IV bias from sampling variation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper extends Causal Testing, a previously proposed approach for testing nondeterministic software, to two new settings: systems with interacting variables and systems with unobservable variables. It introduces effect modification terms and instrumental variable (IV) methods into an existing causal testing tool and evaluates the approach on the CARLA autonomous driving simulator. Three requirements are tested: (RE1) the infraction penalty formula, (RE2) the causal effect of ego-vehicle model on infractions, and (RE3) the effect of CARLA version on simulation performance. For RE1, an interaction-adjusted regression recovers the documented penalties; for RE2, three of four agents show a significant effect of vehicle model; for RE3, IV, gold-standard adjustment, and no-adjustment estimates are compared. The paper concludes that reliable test outcomes can be obtained even when variables interact and are not recorded, and provides a replication package.","tokens_in":21365,"tokens_out":10510,"duration_ms":101890,"significance":"If the conclusions were fully supported, the paper would be a useful extension of Causal Testing to realistic settings with hidden confounders and interacting variables, and the case study would offer practical guidance for ADS testing. The study is clearly described, uses a realistic high-fidelity simulator, and ships a public replication package. However, the RE3 evidence is not fully supportive of the central claims: the IV estimates disagree with the paper's own gold standard on one agent and flip a pass/fail decision, while the no-adjustment estimates are nearly identical to the gold standard, so the case study does not demonstrate the claimed advantage of IV in this setting. The contribution is therefore plausible but requires substantial re-analysis and a more cautious framing.","major_comments":[{"comment":"The claim that 'IV estimates still produce reliable test outcomes' is contradicted by the paper's own gold-standard comparison for the CARLA Garage privileged driver. The IV confidence intervals for v0.9.10.1 and v0.9.11 are [7.389, 8.100] and [8.114, 8.617], which do not overlap, so the test fails; the gold-standard intervals [6.883, 7.441] and [7.398, 8.231] do overlap, so the test passes. Thus the IV method flips a pass/fail decision for one of the four drivers, and the paper should not state that IV gives reliable test outcomes without accounting for this.","section":"Section 5.5.2 / Table 3 / Section 6.3"},{"comment":"The IV identifying assumptions for RE3 are asserted from Figure 2 but never tested. Specifically, RouteLength must have no direct effect on SystemTime, no unobserved common cause with SimulationTime or SystemTime, and all relations must be linear. The paper itself notes that linearity 'cannot be known for sure', and the large discrepancies between IV and gold-standard estimates for CARLA Garage privileged (0.589 on v0.9.10.1 and 0.569 on v0.9.11) suggest that at least one assumption is violated. The authors should provide empirical checks, such as a test for a direct RouteLength-to-SystemTime path or a comparison of first-stage strength, or explicitly limit the RE3 conclusions.","section":"Section 5.5.2 / Figure 2"},{"comment":"The no-adjustment estimates are almost identical to the gold-standard estimates (median error 0.001, versus 0.31 for IV), so the case study data do not exhibit the confounding that IV is supposed to remove. This is especially clear for the CARLA Garage drivers, where manual code inspection shows NPCvehicles is constant, making Equations (5) and (6) effectively equivalent. The statement in Section 6.3 that IV 'adjusts for the bias' is therefore not demonstrated; the data do not show a bias that IV removes.","section":"Section 5.5.2 / Table 3 / Section 6.3"},{"comment":"The RE1 estimator in Equation (2) has the same functional form as the leaderboard formula in Equation (1), so the exact matches and degenerate confidence intervals in the top half of Table 1 are a consistency check of the estimator against CARLA's documented formula rather than an independent demonstration that effect modification is correctly handled. This does not invalidate the useful contrast with Equation (3), but the paper should present RE1 as a sanity check and rest the RQ1 evidence primarily on the failures observed when the interaction term is omitted.","section":"Section 5.3.2 / Equation (2) / Table 1"},{"comment":"RE2 is described as testing the effect of ego-vehicle model on the number of infractions, but the outcome used in the regression is the numeric infraction penalty (1.0, 0.7, 0.65, 0.6, 0.5), which is a coding of a categorical variable. The conclusions, including the unexpected result for the TCP privileged driver, could depend on this scoring choice. The paper should either justify the penalty scale as the intended outcome or report a robustness analysis using a count or binary outcome.","section":"Section 5.4.2 / Table 2"}],"minor_comments":[{"comment":"The word 'charactesized' should be 'characterized'.","section":"Section 4.2"},{"comment":"The access dates are given as 'Accessed 2024-19-03', which is not a valid date; these should be corrected to a standard form such as 'Accessed 2024-03-19'.","section":"References [3] and [4]"},{"comment":"The text says 'ComplectionScore' in the discussion of the interaction term; this should be 'CompletionScore'.","section":"Section 5.3.2"},{"comment":"The expected effect for RE3 is stated as 'not positive' in Section 5.5.1 but as 'zero' in the Test Outcomes paragraph of Section 5.5.2; these statements should be reconciled.","section":"Sections 5.5.1 and 5.5.2"},{"comment":"The symbols for the path coefficients in Equations (4a)-(4c) are typeset inconsistently with the notation used elsewhere in the paper; using a single consistent coefficient notation would improve readability.","section":"Section 5.5.2 / Equation (4)"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of the venue and the replication package is a strength. The main issue is that the RE3 conclusions are stronger than the data support: the IV results conflict with the gold standard on one of four drivers, and the no-adjustment baseline is essentially as accurate. If the authors revise the paper to present RE3 as a cautionary or mixed result, validate or relax the IV assumptions, and soften the abstract and RQ3 claims accordingly, the contribution could be acceptable. I would not reject outright, but the current version overclaims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear Colleague,\n\nThis is a useful paper, but the abstract oversells the hard part. The genuinely new thing is applying effect modification and IV methods inside causal testing, and the CARLA case study is a good testbed. The replication package and the effort to instrument TCP to get a gold standard are real contributions. The effect-modification half (RE1) works, though it's something of a tautology: the estimator is the same functional form as the leaderboard formula, so of course it fits. The more interesting find is RE2, where the ego-vehicle matters more than expected, including a scenario artifact. That's a legitimate use of causal testing.\n\nThe weak spot is RE3, the unobservable-variable claim. The IV assumptions are asserted rather than validated. The gold-standard comparison in Table 3 shows the IV estimates deviate by up to ~0.6 seconds and flip the test outcome for the Garage privileged driver. Meanwhile, the no-adjustment estimate is almost exactly the gold standard. That means this dataset doesn't exhibit the confounding that IV is supposed to remove. The paper's own conclusion says accuracy varies, but the abstract and RQ3 answer go further than the evidence supports.\n\nI think the stress-test note is correct on the load-bearing point: the case study cannot show that IV works for unobservables because the unobservables don't actually bias the naive estimate here. That doesn't destroy the paper; it means the demonstration is incomplete. A better design would deliberately induce hidden confounding, or use an overidentifying test to check the instrument. The paper also treats the infraction penalty as a numeric variable in RE2, which is questionable, but it's a minor issue.\n\nWho's this for? The causal-testing and simulation-based testing crowd. It deserves a serious referee, but the authors need to temper the claims and report the IV/gold discrepancies frankly. I'd send it to review with major-revision expectations.\n\nBest,","headline":"A legitimate extension of causal testing to hidden and interacting variables, but the instrumental-variable case study does not support the paper's headline claim.","tokens_in":21873,"tokens_out":3888,"would_cite":true,"duration_ms":36334,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Causal testing can produce reliable test outcomes for systems with interacting and unobservable variables, as demonstrated on the CARLA driving simulator.","keywords":["Causal Testing","Causal Inference","Software Testing","Instrumental Variables","Effect Modification","Unobservable Variables","Autonomous Driving","CARLA"],"falsifier":"Record the numbers of spawned pedestrians and NPC vehicles during the CARLA runs (as the paper does to build its gold standard), estimate the simulation-time-to-system-time effect by ordinary adjustment, and compare it with the instrumental-variable estimate; if the two differ by substantially more than the confidence-interval overlap, then the exclusion restriction or linearity assumption behind the instrument fails. A more direct check is to add the term $\\text{RouteLength}\\to\\text{SystemTime}$ to the regression and test whether its coefficient is nonzero, since a nonzero direct effect of the instrument on the outcome invalidates the IV ratio in Equation (4c).","tokens_in":20926,"feed_emoji":"🚗","tokens_out":7865,"duration_ms":73303,"temperature":0.7,"pith_summary":"This paper argues that Causal Testing—checking software requirements by estimating causal effects from observational run data—can be extended to systems where variables interact and where some relevant variables never appear in the logs. The authors add two devices from causal inference: an interaction term in the estimator to account for effect modification (when the effect of one variable depends on the value of another), and an instrumental-variable ratio to neutralize unobserved confounders. They evaluate the approach on three requirements of the CARLA driving simulator, using two autonomous driving systems and their built-in scenarios as the test data. The approach yields the expected verdicts for all three requirements: exact infraction penalties, a real but unexpected vehicle-model effect on infractions, and a speed regression between CARLA versions—while using less, and less controlled, data than statistical metamorphic testing needs. If the result holds generally, expensive nondeterministic systems could be tested from pre-existing logs, and regression testing could proceed even when logging infrastructure is incomplete.","feed_headline":"Causal testing handles hidden, interacting variables","feed_subtitle":"Interaction terms and instruments yield reliable verdicts from uncontrolled CARLA simulator logs.","key_machinery":"The load-bearing object is the hand-drawn causal directed acyclic graph (DAG) in Figure 2, which specifies which system variables may directly affect which others and which variables are unobserved. The DAG is used twice: identification, where backdoor paths are found and closed by stratification or adjustment, and estimation, where linear regression is fitted with the identified adjustment set. For RE1 the adjustment takes the form of an interaction term $\\text{CompletionScore} \\times \\text{OutsideLane}$, which absorbs the effect modification coming from the leaderboard's multiplicative penalty formula. For RE3 the DAG licenses an instrumental-variable design: route length (the instrument) affects simulation time, which in turn affects system time, while route length has no direct arrow to system time and no arrow from the unobserved pedestrian and vehicle counts; the causal effect is then the IV ratio $\\beta_{\\text{RouteLength}\\to\\text{SystemTime}} / \\beta_{\\text{RouteLength}\\to\\text{SimulationTime}}$. The DAG thus performs the role of test oracle: the test verdict is a comparison between the estimated causal effect and the effect the DAG says the system should have.","core_discovery":"On its own terms, the paper establishes that two established causal-inference concepts remove the two standing limitations of prior Causal Testing work: interacting variables and unobservable variables. For interaction, the paper shows that adding the product term $\\text{CompletionScore} \\times \\text{OutsideLane}$ to the regression estimator isolates the direct effect of completion score on driving score: all five infraction-penalty test cases pass exactly, while omitting the interaction term makes two correct test cases fail for the trained driver. For hidden variables, the paper uses route length as an instrumental variable and estimates the causal effect of simulation time on system time by the ratio $\\beta_{\\text{RouteLength}\\to\\text{SystemTime}} / \\beta_{\\text{RouteLength}\\to\\text{SimulationTime}}$. The resulting estimates flag CARLA v0.9.11 as slower than v0.9.10.1 for all four driving agents, matching a gold-standard adjustment obtained by modifying the code to log the hidden pedestrian and vehicle counts; and the same analysis pipeline surfaces a vehicle-model effect on infractions that standard testing would not have revealed. The paper's conclusion is that reliable test outcomes are obtainable without large highly controlled datasets or code instrumentation, provided the user's causal model satisfies the identifying assumptions.","pith_inferences":["Outside the paper's data, the closeness between the unadjusted and gold-standard estimates in RE3 suggests the hidden confounders happened to have little bias here; the reliable verdict in this case does not mean adjustment can be skipped in general, since the analyst usually cannot tell which estimate is closer.","The same instrumental-variable recipe should transfer to other simulators or cyber-physical systems where a controllable proxy (scenario length, episode size, route distance) drives the treatment and has no plausible direct path to the outcome; replicating the CARLA result on a second simulator with instrumented ground truth would test that transfer.","Since the hand-drawn DAG is the test oracle, the method's scope is bounded by the domain knowledge in that DAG; learning the graph from the same data would undermine the oracle because the learned structure would encode the very faults under test.","An IV estimate that is causally valid but less precise will be most useful when the requirement is qualitative ('new version is not slower') rather than when a precise effect size is the acceptance criterion."],"forward_implications":["One observational dataset can be reused across multiple requirements and multiple driving agents, so adding test properties does not require collecting new controlled data for each one.","Adding interaction terms to the estimator prevents false test failures caused by effect modification, letting a tester query a direct causal effect instead of an aggregated association.","Instrumental-variable estimation makes regression testing possible when confounders are not logged, as long as a valid instrument such as route length is available and the linearity assumptions hold.","The same pipeline can expose unexpected behaviours—such as the ego-vehicle model changing infraction rates and a CARLA version regression absent from the changelog—that would be missed by scenario-based testing or by plain association."],"supporting_citations":[{"why":"Supplies the original Causal Testing methodology and the DAG-construction convention this paper extends.","marker":"[15]"},{"why":"Source for the definitions of effect modification, instrumental variables, and the positivity assumption.","marker":"[29]"},{"why":"Source for backdoor-path identification and the instrumental-variable rationale.","marker":"[41]"},{"why":"Introduces CARLA, the high-fidelity driving simulator used as the case system.","marker":"[17]"},{"why":"Defines the infraction penalties and driving score formula used as expected effects in RE1.","marker":"[3]"},{"why":"Establishes path coefficients and the ratio argument underlying the IV estimator in Equation (4c).","marker":"[53]"},{"why":"Provides TCP, one of the two subject ADSs, with the driving scenarios used for data collection.","marker":"[54]"},{"why":"Provides CARLA Garage, the other subject ADS, with its own scenario set.","marker":"[30]"},{"why":"Defines statistical metamorphic testing, the baseline with which Causal Testing is compared in RQ2.","marker":"[25]"}],"fun_headline_variants":["Causal testing tames hidden, interacting variables","Instruments and effect modifiers expand causal test scope","Hide and seek: causal testing with unobserved variables","Causal testing without full observability or controlled data","Unobserved variables no longer block causal testing"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the manually drawn causal map (Figure 2) is correct, and for the instrumental-variable requirement in particular that route length has no direct effect on system time, that no hidden confounder affects route length, and that the relationships among the variables are linear.","fun_headline_variants_meta":{"raw":{"variants":["Causal testing tames hidden, interacting variables","Instruments and effect modifiers expand causal test scope","Hide and seek: causal testing with unobserved variables","Causal testing without full observability or controlled data","Unobserved variables no longer block causal testing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000194,"raw_usage":{"total_tokens":1381,"prompt_tokens":1002,"completion_tokens":379,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":618,"completion_tokens_details":{"reasoning_tokens":307}},"tokens_in":618,"tokens_out":379,"duration_ms":4191,"temperature":1.0,"reasoning_tokens":307,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:01:01.485676+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Record the numbers of spawned pedestrians and NPC vehicles during the CARLA runs (as the paper does to build its gold standard), estimate the simulation-time-to-system-time effect by ordinary adjustment, and compare it with the instrumental-variable estimate; if the two differ by substantially more than the confidence-interval overlap, then the exclusion restriction or linearity assumption behind the instrument fails. A more direct check is to add the term $\\text{RouteLength}\\to\\text{SystemTime}$ to the regression and test whether its coefficient is nonzero, since a nonzero direct effect of the instrument on the outcome invalidates the IV ratio in Equation (4c).","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source for the definitions of effect modification, instrumental variables, and the positivity assumption."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source for backdoor-path identification and the instrumental-variable rationale."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces CARLA, the high-fidelity driving simulator used as the case system."},{"cited_title":"CARLA Autonomous Driving Leaderb oard","cited_arxiv_id":null,"evidence_quote":"Defines the infraction penalties and driving score formula used as expected effects in RE1."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes path coefficients and the ratio argument underlying the IV estimator in Equation (4c)."}],"review_version":1}