{"id":"c54bbbd3-f66f-438b-beb2-437ee118b49a","arxiv_id":"2608.13404","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Iterative LLM repair of Terraform code causes a security regression in about 3.3% of scenarios under strict counting, most often through resource restructuring, with iteration 3 as the recommended stopping point.","lead":"This paper measures whether telling an AI model to fix errors in its generated cloud configuration code can break security settings that previously passed. Across 5,968 repair timelines, it finds about 3.3% of scenarios show an unambiguous security regression, with most apparent regressions being artifacts of how checks apply to multiple resources.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Timeline coherence is the load-bearing assumption: if consecutive records in a non-RAG timeline come from different executions, the strict 3.3% regression rate counts artifacts, not repair steps.","rationale":"The reader's weakest_assumption is the same one I consider most load-bearing: Definition 1's validity depends on timeline coherence. I examined alternatives—single-run variance, model-mix confounding, and strict-mode definitional choices—and none cuts as deeply. Single runs affect precision, but the authors already frame 3.3% as an order-of-magnitude estimate. Model attribution affects RQ3 comparisons but not the existence of strict regressions if timelines are coherent. Strict mode may miss partial degradations, but that makes it conservative in one direction and does not manufacture regressions. Timeline mixing, by contrast, can both create and destroy the events that constitute the dependent variable. The paper itself flags this in Section 7, so this is not a re-interpretation of opaque data; it is weighing a disclosed limitation that the authors do not quantify. The concrete test would settle it: reconstructing timelines with execution boundaries and recomputing the strict rate. If the rate persists, the qualitative and quantitative claims are much stronger; if it collapses, the paper should be revised to state an upper bound. My verdict remains CONDITIONAL, matching the reader: the study is methodologically transparent and the qualitative phenomenon is plausible, but the headline point estimate should not be accepted as precise until timeline coherence is demonstrated. Therefore no change to the reader's verdict is needed.","tokens_in":103,"tokens_out":11547,"duration_ms":183453,"concrete_test":"Use the replication package's raw per-iteration logs to rebuild every non-RAG timeline twice: once with the paper's deduplication rules, and once with execution boundaries identified from timestamps/model tags/run IDs, keeping only transitions whose adjacent records belong to the same identified execution. Recompute the strict scenario regression rate under the second construction and compare it with 3.3%. If the rate falls below roughly 1% or its confidence interval includes zero, the headline claim is not supported; if the logs lack the metadata to identify execution boundaries, the timeline-coherence assumption is untestable and 3.3% should be reported only as an upper bound, not a point estimate.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.1's headline strict-mode figure—194 scenarios (3.3%) with regression—is a transition-level measurement. Definition 1 (Section 4.2) requires consecutive iterations from one coherent repair run; otherwise a pass-to-fail transition is not a repair step. Section 4.1 describes keeping 'the most recent execution' when a scenario ran more than once under a non-RAG configuration, and Section 7's Internal threats concede that 'some timelines may mix iterations from separate executions.' The same-iteration exclusion rule in Section 4.2 can leave a timeline whose adjacent records come from different attempts, especially when duplicate records exist at an iteration index. Mixing two non-regressing executions can manufacture a false strict regression (e.g., pair iteration i from a run where the check passed with iteration i+1 from another run where it failed), while also being able to hide genuine ones. Every transition-level statistic—5.2% transition rate, root-cause distribution, churn/volatility contrasts, self-correction and oscillation rates—inherits this risk. The paper reports CIs computed over timelines as independent units, so the intervals do not cover this error. This is load-bearing because the contribution is precisely the raw per-iteration trajectory; cumulative-best metrics are monotone by construction and would be unaffected. No analysis quantifies how many timelines are mixed.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies whether iterative LLM-driven repair of Terraform Infrastructure-as-Code (IaC) can turn a previously passing CIS/Checkov security check into a failing one. Using 5,968 scenario timelines from the IaC-Eval benchmark and 4,440 iteration transitions with Checkov results on both sides, it reports that 13.8% of scenarios (24.8% of transitions) exhibit at least one regression under standard detection, while strict detection — which counts only exclusive check failures — reduces the rate to 3.3% of scenarios (5.2% of transitions). The paper attributes most standard-mode events to multi-resource ambiguity, identifies resource restructuring as the dominant root cause (79.0%), and reports that regression transitions have higher code churn and check volatility, that 36.6% of standard-mode regressions self-correct, and that iteration 3 is the optimal stopping point. The central claim is that cumulative-best metrics hide a real but modest per-iteration security cost of repair.","tokens_in":17467,"tokens_out":7819,"duration_ms":75572,"significance":"If the measurement is sound, this is the first IaC-domain evidence that cumulative-best reporting masks per-iteration security regressions in LLM repair loops, and the standard/strict distinction is a useful methodological contribution for resource-level static analysis. The strict-mode 3.3% scenario rate is a genuinely conservative baseline, and the authors are explicit about their thresholds and about what is descriptive versus causal. The paper also ships a replication package, which supports reproducibility of the derived tables and figures. The main qualitative finding — that repair iterations can break a previously passing check and that cumulative-best metrics hide this — is plausible and important. However, the headline numbers rest on timeline coherence and on a regression definition that the paper itself does not fully validate, so the precise point estimates cannot be certified as stated.","major_comments":[{"comment":"The transition-level definition of regression requires consecutive iterations from one coherent repair run, but the deduplication rule keeps only the most recent execution when a non-RAG scenario ran more than once, and the same-iteration exclusion in §4.2 does not prevent adjacent records from different executions. The threats section concedes that 'some timelines may mix iterations from separate executions.' Since every prevalence, root-cause, churn, volatility, self-correction, and oscillation statistic is computed on these transitions, a mixed timeline can manufacture or hide a pass-to-fail transition that is not a repair step. The strict 3.3% scenario rate is itself derived from such transitions. The authors should quantify how many timelines contain records from more than one execution and re-run all transition-level analyses on the subset of provably single-execution timelines; until that is done, the headline point estimate is not supported.","section":"§4.1, §4.2, §7 (Internal)"},{"comment":"Standard mode counts a regression whenever a check is in the passed set at iteration i and in the failed set at iteration i+1, without requiring that the check was not already failing at iteration i. For a multi-resource check that is partly failing at iteration i, the same per-resource status can satisfy both conditions, so an unchanged partial failure is counted as a regression. This inflates the 13.8% scenario rate and the 24.8% transition rate and weakens the claim that standard mode is an inclusive upper bound on genuine regressions; it is an upper bound plus a constant-partial-failure artifact. The authors should either require that the check was not in the failed set at iteration i, or track failures per resource instance, and report how many standard-mode events are constant partial failures.","section":"§4.2, Definition 1"},{"comment":"All non-RAG strategy comparisons and the RAG-versus-non-RAG comparison are confounded by unknown model identity, because the non-RAG logs do not record the generating model and the model mix differs by strategy (Chain-of-Thought is 79% Gemini, Few-Shot 73%, Zero-Shot 46%). The paper acknowledges this, but it still presents Chain-of-Thought as the lowest-regression non-RAG strategy and uses that ordering in the Discussion's recommendation to prefer it where retrieval is unavailable. Since the model effect is the strongest measured factor in the RAG data (OR = 17.29), the observed ordering is exactly what differing model mixes would produce. The strategy-level findings should be reported only as descriptive of the deployed configurations, and the practical recommendation to prefer Chain-of-Thought should be removed or explicitly labeled as confounded.","section":"§5.3, §7, Discussion"}],"minor_comments":[{"comment":"Transition-level confidence intervals and Mann-Whitney tests treat the 4,440 transitions as independent even though transitions are nested within scenario timelines; within-timeline autocorrelation is acknowledged in the threats section but not quantified or corrected. Adding a cluster-robust analysis or explicitly reporting this as a limitation of the transition-level tests would strengthen the paper.","section":"§4.5, §7"},{"comment":"Table 3 reports only standard-mode scenario regression rates, while the text makes important strict-mode comparisons (including the claim of zero RAG strict regressions). Adding strict-mode counts and rates for each configuration row would let readers verify the mode-dependent reversals.","section":"Table 3, §5.3"},{"comment":"The normalized per-transition rates (22.8%, 26.6%, 29.5%, and 20.8%) are stated without a table of the per-transition denominators; a supplemental table or appendix would allow readers to recompute these rates from the reported event counts.","section":"§6.1, Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The central phenomenon is plausible and the authors are unusually forthcoming about their limitations, but the headline numbers cannot be certified without addressing the timeline-mixing problem. Because the replication package is promised and the raw logs should allow a quantification of mixed timelines, this is fixable within a revision rather than a reason for rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a genuine first look at per-iteration security trajectories in LLM-driven IaC repair, and the central qualitative finding—repair loops sometimes break previously-passing checks, and cumulative-best reporting hides it—holds up. The precise numbers are softer than they look.\n\nWhat is new: nobody has tracked individual Checkov checks across repair iterations in Terraform before. The standard/strict distinction is a real methodological contribution; it cleanly separates true exclusive failures from multi-resource bookkeeping, and the strict-mode robustness check gives you confidence that most standard-mode \"regressions\" are artifacts. The root-cause taxonomy (restructuring dominating) is plausible, and the code churn / volatility results are consistent with the degradation literature in general code.\n\nWhere I wince: the timeline-coherence problem is real and the authors admit it. For non-RAG runs, deduplication keeps the most recent execution, and Section 7 concedes consecutive records may come from different attempts. That means every transition-level rate—including the 3.3% strict figure—could be counting artifacts. The confidence intervals are computed over timelines as if they were independent coherent units, so they do not cover this error. The paper discloses this, which I respect, but \"disclosed limitation\" and \"quantified limitation\" are different things. There is no estimate of how many timelines are mixed.\n\nAlso: single runs per configuration leave variance unquantified; the non-RAG strategy comparisons are confounded by unknown model mix; and the root-cause classifier was not validated against human labels. These are all in the threats section, so the authors are not hiding them, but they do cap the precision of the numbers.\n\nNone of that kills the paper. The strict rate being 3.3% or 2% or 5% does not change the practical message: stop iterating around iteration 3, monitor check volatility, and treat cumulative-best metrics with suspicion. The qualitative claims rest on multiple independent signals (churn effect, volatility effect, self-correction rate) that would be hard to manufacture through timeline mixing alone.\n\nWho this is for: anyone building LLM feedback loops for IaC, and empirical SE researchers studying repair dynamics. It deserves a serious referee. If I were the editor, I would send it out and ask the authors to quantify the mixed-timeline problem—at minimum, report the fraction of non-RAG timelines that were deduplicated and run the headline strict rate on the subset of RAG timelines where model attribution is known.\n\nVerdict: engage with it. It's a good paper with an honest, fixable soft spot.","headline":"The first per-iteration look at security regressions in LLM-driven IaC repair supports the qualitative claim, but the strict 3.3% figure is an order-of-magnitude estimate because the authors admit timelines may mix separate executions.","tokens_in":18013,"tokens_out":2308,"would_cite":true,"duration_ms":21730,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Iterative LLM repair of infrastructure-as-code can break a previously passing security check, and the paper's conservative estimate is about 3.3% of repair scenarios.","keywords":["Infrastructure as Code","security regression","LLM code repair","iterative feedback loop","CIS compliance","Checkov","Terraform","cumulative-best metrics"],"falsifier":"Re-run a sample of scenarios while recording every attempt with full model attribution and an unbroken per-scenario order, then recompute the strict-mode transition rate directly from those true per-attempt sequences; if the rate departs from the reported 3.3% beyond the confidence interval, the headline estimate is an artifact of timeline reconstruction rather than a property of iterative repair.","tokens_in":17006,"feed_emoji":"🛡️","tokens_out":7585,"duration_ms":67063,"temperature":0.7,"pith_summary":"This paper asks whether the standard practice of feeding validator errors back to a large language model until generated infrastructure code passes can, in the act of fixing, break security properties that were already satisfied. The authors track individual CIS security checks across up to five repair rounds in roughly six thousand scenario timelines, counting every transition in which a check that passed at one iteration fails at the next. They find that repair does introduce such regressions, but that the defensible rate is modest: about 3.3% of scenarios under a strict counting that excludes multi-resource ambiguity, versus 13.8% under an inclusive counting. The result matters because prior work reports only cumulative-best compliance, which is non-decreasing by construction and therefore cannot see this security cost.","feed_headline":"LLM repair breaks a security check in 3.3% of scenarios","feed_subtitle":"Per-iteration tracking shows cumulative-best metrics hide the cost, concentrated in resource restructuring.","key_machinery":"The load-bearing object is the transition-level regression detector built from Checkov check IDs mapped to CIS controls, together with the standard/strict detection distinction. Each consecutive pair of iterations $(i, i+1)$ with Checkov results forms a transition; a regression is a check that passes at $i$ and fails at $i+1$. The strict variant filters out multi-resource ambiguity by requiring the check to be exclusively passed at $i$ and exclusively failed at $i+1$, and this distinction is what turns a dramatic 13.8% standard-mode result into a conservative 3.3% strict-mode claim. Supporting machinery includes a root-cause taxonomy (resource restructuring, configuration drift, argument removal, unclassified) and the check volatility measure $V = |C_{new}| + |C_{removed}| + |C_{flipped}|$, which is the strongest regression signal in the data.","core_discovery":"On the paper's own terms, the central discovery is that iterative LLM-driven repair of Terraform configurations produces genuine security regressions, but most apparent regressions are measurement artifacts. A security regression is defined at the transition level: a check ID that is in the passed set at iteration $i$ and in the failed set at iteration $i+1$. Because one Checkov check can apply to several resources, a restructured configuration can make one resource fail while another still passes; the paper's strict mode requires the check to be exclusively passed at $i$ and exclusively failed at $i+1$, and this drops the scenario-level rate from 13.8% to 3.3% (and the transition-level rate from 24.8% to 5.2%). Root-cause classification attributes 79.0% of standard-mode regressions to resource restructuring, and regressing transitions carry 2.6x more code churn (Cohen's $d = 0.90$) and 4.9x higher strict-mode check volatility ($d = 1.49$). The authors conclude that cumulative-best reporting hides a real security cost and that iteration 3 is the optimal stopping point.","pith_inferences":["A testable extension: run the same repair loop on CloudFormation or Ansible with per-resource validators; if multi-resource ambiguity is the mechanism, the standard/strict gap should reappear with a similar order-of-magnitude strict rate.","Because non-RAG timelines are not model-attributed, the observed Chain-of-Thought advantage may be a model-composition effect; a model-controlled run with recorded model identity per attempt would settle whether prompting strategy itself reduces regressions.","The clean-timelines-regress-more result could be probed by ablating feedback content (syntax-only versus security-only) to separate the opportunity effect from the feedback-purity effect.","If the 36.6% self-correction rate replicates, single-regression halt rules are premature; an adaptive policy that waits for a check to stay failed across two consecutive iterations may beat a fixed budget."],"forward_implications":["Cumulative-best reporting should be supplemented with per-iteration trajectory metrics; any pipeline that reports only the best compliance over iterations cannot detect the repair-induced regressions this paper measures.","A bounded iteration budget near three rounds captures most compliance gains (83.1% pass rate at iteration 3 versus 83.4% at iteration 4) while limiting accumulated regression risk.","Feedback loops should flag regressions explicitly and encourage minimal edits, since resource restructuring accounts for 79.0% of standard-mode regressions and large rewrites predict them.","Standard and strict detection support different conclusions: RAG looks worse than non-RAG in standard mode but records zero strict-mode regressions, so the choice of detection mode determines model and strategy comparisons.","Check volatility is cheap to compute online and is the strongest predictor of regression ($d = 1.49$ in strict mode), making it a usable halting or flagging signal."],"supporting_citations":[{"why":"Conceptual replication baseline: established that iterative LLM repair degrades security in general-purpose C and Java code, motivating the IaC adaptation.","marker":"[25]"},{"why":"Showed that SAST-based gating can worsen latent security degradation, raising the question this paper tests for Checkov-based IaC feedback.","marker":"[7]"},{"why":"Supplies the IaC-Eval benchmark of 458 Terraform scenarios from which all 5,968 timelines are generated.","marker":"[16]"},{"why":"The static analyzer whose per-resource check outcomes define the pass/fail data and enable check-ID-level regression tracking.","marker":"[5]"},{"why":"The CIS AWS Foundations Benchmark that maps the 30 tracked Checkov check IDs to security controls and categories.","marker":"[9]"},{"why":"Prior IaC feedback-loop study reporting cumulative-best results and exponential feedback decay, the baseline for the plateau and stopping-point analysis.","marker":"[20]"},{"why":"Provides the prior estimate that only 7% of LLM-generated IaC is secure without explicit guidance, framing the stakes for repair.","marker":"[12]"}],"fun_headline_variants":["True security regression rate in LLM repair: 3.3%","Cumulative-best hides real cost: 3.3% regress security","Resource restructuring causes 79% of security check regressions","Optimal stop at iteration 3: security regression analysis","Most security regressions are artifacts; strict rate 3.3%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that each timeline is a coherent sequence of repair attempts for one scenario, so a pass-to-fail move between consecutive log entries reflects one genuine repair step; the authors concede that some timelines may mix iterations from separate executions.","fun_headline_variants_meta":{"raw":{"variants":["True security regression rate in LLM repair: 3.3%","Cumulative-best hides real cost: 3.3% regress security","Resource restructuring causes 79% of security check regressions","Optimal stop at iteration 3: security regression analysis","Most security regressions are artifacts; strict rate 3.3%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00092,"raw_usage":{"total_tokens":4063,"prompt_tokens":1177,"completion_tokens":2886,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":793,"completion_tokens_details":{"reasoning_tokens":2794}},"tokens_in":793,"tokens_out":2886,"duration_ms":18911,"temperature":1.0,"reasoning_tokens":2794,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:40:09.248736+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run a sample of scenarios while recording every attempt with full model attribution and an unbroken per-scenario order, then recompute the strict-mode transition rate directly from those true per-attempt sequences; if the rate departs from the reported 3.3% beyond the confidence interval, the headline estimate is an artifact of timeline reconstruction rather than a property of iterative repair.","supporting_citations":[{"cited_title":"Agyekum and F","cited_arxiv_id":null,"evidence_quote":"The CIS AWS Foundations Benchmark that maps the 30 tracked Checkov check IDs to security controls and categories."},{"cited_title":"Using a Feedback Loop for LLM-based Infrastructure as Code Generation","cited_arxiv_id":"2411.19043","evidence_quote":"Provides the prior estimate that only 7% of LLM-generated IaC is secure without explicit guidance, framing the stakes for repair."}],"review_version":1}