{"id":"6c96f2e6-7659-480b-abc5-c7f30030d717","arxiv_id":"2607.28271","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Locksmith Loop raises COBOL–Java migration branch coverage past input-search plateaus via parity-gated harness mutations, reaching ~97–100% on two open programs and 91.9% on a 4k-line production-like batch program.","lead":"The paper presents Locksmith Loop, an agentic method that mixes multi-algorithm input search with parity-preserving code mutations to raise branch coverage when checking COBOL-to-Java migrations. It matters because enterprises need a deterministic way to trust AI-migrated legacy code when real test data is scarce.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"Parity-preserving harness mutations may validate forced paths rather than real production reachability, so high mock coverage can overstate cutover readiness.","rationale":"The reader correctly isolates the soft spot: mutations expand the harness boundary and are kept when they add coverage and pass COBOL≡Java parity, but production cutover cares about paths reachable under real multi-step state, not only under side-channel pins or injected PERFORMs. That premise is load-bearing for reading the case-study numbers as evidence the migration is validated beyond input-search ceilings, not merely that mutated mocks agree. The method remains coherent as a compatibility oracle and SDLC pre-integration amplifier (explicit 1:1 parity goal, honest threats, recursive Witness Search + skill catalog). No internal contradiction or hidden tautology requires REJECT; missing public migrator/mocks/production code and thin external baselines already justify CONDITIONAL. My concrete test operationalizes the same concern without demanding new program families first: ablate the most artificial mutation class and measure how much of the claimed Mutation lift survives. If it largely survives under dispatcher-only / catalog-feasible stubs, the reader’s worry is overstated; if not, claims should be narrowed. Verdict stays CONDITIONAL; no upgrade to ACCEPT without that (or equivalent) evidence and artifacts.","tokens_in":16002,"tokens_out":734,"duration_ms":32005,"concrete_test":"On the production-like artifact (or CBSTM03A), partition final covered branches into Witness-Search-only vs Mutation-opened. Re-run with call-injection disabled and force-set limited to values already producible by the mock catalog under normal dispatch; report residual branch/paragraph coverage and how many Mutation-only paragraphs become unreachable. If most of the reported Mutation Δ (e.g. +25 paragraphs / +47 transitions) disappears, the cutover-readiness reading of the headline numbers weakens and should be scoped to “forced-entry parity.”","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim rests on Locksmith pushing past multi-algorithm Witness Search plateaus via parity-preserving mutations (dispatcher-arm and call-injection / force-set skills, §III-D–E and §IV), then accepting cases only when the three-axis Parity Gate holds on the symmetrically mutated COBOL mock and generated Java. That is a valid check that the migrated paragraph bodies and stub handling agree once entry is forced. It is a weaker check that the unmodified production control flow—multi-step chains that set switches and coordinated external sequences (§IV-A residual gaps)—would actually reach those paragraphs and exercise the same branches. On CBSTM03A, Mutation alone opened 8 paragraphs and 19 transitions Witness Search could not; on the production-like program it added ΔP=+25 and ΔT=+47 after a Witness Search plateau at P=110/142, T=54/146. If a large share of that red coverage is only live under call-injection or pinned external values that production inputs never arrange, then “91.90% branch / 166/166 PASS” measures mock-harness agreement more than migration readiness of the real routing surface. The paper’s own residual analysis (coordinated fake file/DB sequences and early internal switches) and the construct-validity threat (§V-b) already point at this gap; the load-bearing premise is that kept mutations still stress the migration-relevant control flow rather than mainly an artificially expanded harness.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper proposes the Locksmith Loop, an agentic test-synthesis method for validating 1:1 COBOL-to-Java migrations. A deterministic migrator and mock generator produce paired runtimes; six input-search algorithms (Witness Search) explore mocked inputs until a coverage plateau; an analyzer then identifies Locked Paragraphs and an Authoring Layer proposes parity-preserving harness mutations (e.g., dispatcher-arm and call-injection/force-set skills) applied symmetrically to COBOL mock and Java. Kept mutations must add coverage and pass a three-axis Parity Gate (paragraphs hit, stub log, terminal state). On three programs (430–4,114 SLOC), the loop reports gains past multi-algorithm plateaus: near-complete coverage on two open-source CardDemo programs and 91.90% branch / 95.1% paragraph coverage with 166/166 parity PASS on a production-like batch program. The authors position this as a novel deterministic-oracle pattern for validating agentic migration output.","tokens_in":16414,"tokens_out":1633,"duration_ms":44381,"significance":"If the method’s coverage gains genuinely stress migration-relevant control flow and the parity oracle is accepted as a compatibility criterion, this is a practically significant contribution to legacy modernization and to agentic SE: it combines test amplification, multi-algorithm search, differential testing, and closed-loop LLM authoring under a hard execution gate. Concrete strengths include off-mainframe mocked execution, explicit residual-gap analysis, continuous parity (not only post-hoc), and a large production-shaped run with 166 accepted cases and autonomous operation. The teacher–student split (agent proposes, deterministic student accepts/rejects) is a clear industrial pattern. These are real engineering results, not only a metaphor; the main open question is how far mock-harness coverage under forced entry should be read as cutover readiness.","major_comments":[{"comment":"§III-D–E and §IV (force-set / call-injection): The central claim that Locksmith validates migration parity beyond input-search plateaus rests on keeping parity-preserving mutations that force external returns or paragraph entry on both sides. That checks agreement of paragraph bodies and stub handling once entry is arranged; it is a weaker check that unmodified production routing (multi-step switch setup and coordinated external sequences, §IV-A) would reach those paths. On CBSTM03A, Mutation opened 8 paragraphs and 19 transitions Witness Search could not; on the production run, ΔP=+25 and ΔT=+47 after a Witness Search plateau. Please quantify, per program, what fraction of Mutation-attributed (red) coverage is reachable only under call-injection or pinned externals versus after mutations that only expand the controllable harness interface while still entering via natural dispatch. Witho","section":"§III-D–E, §IV, §IV-A"},{"comment":"Related work cites symbolic-execution COBOL→JUnit equivalence checking [32], [33] as the closest validation line, but §IV reports no head-to-head numbers (coverage, parity failures found, human effort, wall time) against that pipeline or against a pure Witness Search / fuzzing budget-matched baseline without Mutation. The claim of consistently improving beyond input-search plateaus needs an explicit ablation: Witness Search alone (longer budget), Witness Search+Mutation, and ideally the prior automated validation approach on at least the two public programs. Otherwise the incremental value of the agentic Mutation tier remains narrative.","section":"§II, §IV"},{"comment":"§V-a and experimental design: Evidence is three case studies, one internal and not releasable. That is acceptable for a systems paper only if threats are matched by stronger public reproducibility. For the two open-source programs, release (or artifact-describe) the mock harness interface, skill catalog entries that fired, plateau thresholds, and per-phase coverage tables so others can re-run Witness Search and verify Mutation deltas. As written, external validity and auditability of the production 91.90% figure are limited, which is load-bearing for the “scales to production-like” claim.","section":"§IV, §V-a"},{"comment":"§III-F Parity Gate: Acceptance uses three axes (paragraph set, ordered stub log, terminal state). §V-e correctly notes silence on intermediate state, FP rounding, and JVM-only errors. For the migration-validation claim, please state whether any accepted Mutation-tier cases differ in control-flow path shape beyond the paragraph set (e.g., different PERFORM/GO TO trajectories that still hit the same paragraph set), and whether stub_log ordering is complete for SQL/CICS/MQ mocks used in the production-like program. If path multiset or data-layout fingerprints are available but unused, say why; if not, bound what class of migrator bugs the gate can miss.","section":"§III-F, §V-e"}],"minor_comments":[{"comment":"Abstract and §IV: CBSTM03A branch coverage is 74.0% (108/146) while paragraph/transition coverage is very high; the abstract’s “nearly complete coverage on the two open-source programs” should state branch vs paragraph/transition explicitly to avoid over-reading.","section":"Abstract, §IV"},{"comment":"Figure 3 vs Figure 4: Figure 3 reports T=37/38 and P=24/25; the prose sometimes mixes branch probes (146) with transitions (38). Add a single summary table (program × phase × P/T/branch × parity) early in §IV.","section":"§IV, Fig. 3–5"},{"comment":"§III-C: UCB1 exploration constant c and MAP-Elites grid/descriptor definition are not specified; even if “representative not prescribed,” give the values used in the reported runs.","section":"§III-C"},{"comment":"§I-B / Fig. 1: “student” vs “Authoring Layer” colors are described but the Migrator repair path (parity failure → migrator patch) could be shown as a distinct edge; several discrepancy classes in §IV-A are listed without counts.","section":"§I-B, §IV-A"},{"comment":"Minor prose: “EV ALUATE” spacing (§III-D); “necesarry” (§VI); arXiv-style future refs (e.g., COBOL-Coder 2026) should be checked for consistency with the venue bibliography style.","section":"§III-D, §VI"}],"recommendation":"major_revision","confidential_remarks":"Fit for a software engineering journal is good (industrial migration + agentic loop + deterministic oracle). Novelty relative to [32]/[33] and TestGen-LLM-style propose/validate should be sharpened in revision; I would not reject on novelty alone. The internal production program cannot be released—push for a rich public artifact on CardDemo so the Mutation-delta claim is independently checkable. If the authors can add the forced-vs-natural coverage split and a baseline ablation, this could become a strong accept on resubmission."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"Punchline: this is a real systems contribution for COBOL-to-Java parity testing, not a rebrand of EvoSuite. The new piece is the recursive Locksmith loop—six-algorithm Witness Search until plateau, then analyzer-identified Locked Paragraphs opened by parity-preserving mutations (dispatcher-arm / call-injection) applied symmetrically to COBOL mock and generated Java, with a three-axis Parity Gate as the only acceptance oracle. That teacher-student split (agent proposes skills; deterministic runner keeps or reverts) is clean and matches how serious agentic SE should work.\n\nWhat they show is concrete. On CBACT01C they hit near-full paragraph/transition coverage and 96.8% branch. On the production-shaped 4k-line program, Witness Search stalled around 110/142 paragraphs and 54/146 edges; Mutation added +25 paragraphs and +47 edges to 91.9% branch, with 166/166 parity PASS. Residual-gap discussion (multi-step switch chains, coordinated external sequences) is honest, and the threats section correctly flags COBOL-as-oracle as compatibility not correctness.\n\nSoft spots, in proportion. N=3 and one internal codebase limit external validity; no public migrator, mocks, or agent logs, so reproducibility is weak. Baselines are mostly “our six search algorithms plateau together,” not a head-to-head against symbolic COBOL test gen or the IBM-style translate-and-check pipelines they cite. The stress-test concern is real but not fatal: force-set and call-injection can exercise paragraph bodies that production inputs never reach, so red coverage can overstate cutover readiness. The paper already leans that way in §IV–V and frames the goal as pre-integration drift finding, not sign-off. Still, they should quantify how much accepted coverage depends on injection versus natural routing.\n\nMath is light (no formal claims); citation pattern is appropriate and not padded. For anyone doing legacy migration, agentic test amplification, or deterministic oracles around LLM coding, this is worth reading. I would send it to peer review—revise toward more families (CICS/IMS/SQL-heavy), public artifacts on the open programs, and a clearer forced-vs-natural coverage split—not desk-reject.","headline":"Solid industrial method paper: recursive search-plus-harness-mutation with a dual-runtime parity oracle actually moves coverage past multi-algorithm plateaus; the open question is how much of the red coverage is real routing vs forced entry.","tokens_in":17020,"tokens_out":582,"would_cite":true,"duration_ms":18195,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"An agentic loop that mutates harnesses in lockstep can push COBOL–Java migration tests past input-search plateaus while a deterministic parity gate accepts only matching runs.","keywords":["legacy migration","COBOL to Java","agentic testing","differential testing","search-based test generation","parity oracle","harness mutation","branch coverage"],"falsifier":"On a held-out COBOL–Java pair, run Locksmith to high mock coverage, then replay the accepted witnesses and any production-like inputs without the force-set mutations: a systematic rise in parity failures or large drop in true reachable coverage would refute the claim that the loop validates migration readiness.","tokens_in":16851,"feed_emoji":"🔐","tokens_out":918,"duration_ms":24469,"temperature":0.7,"pith_summary":"Enterprises moving COBOL to Java often lack enough real test data to cover corner paths, and ordinary input search stalls once remaining branches depend on state the harness cannot set. This paper proposes the Locksmith Loop: run mocked COBOL and generated Java off-mainframe, search inputs with several algorithms until coverage plateaus, then have an authoring agent propose parity-preserving mutations that open “Locked Paragraphs,” keep a mutation only if it adds coverage and both sides still agree, and resume search inside the newly opened region. On three programs from hundreds to thousands of lines, the loop beat multi-algorithm plateaus, reached near-complete coverage on two open-source cases and about 92% branch coverage on a production-like batch program, and accepted only test cases where Java matched COBOL on path, external effects, and final state. The practical claim is that agentic migration output can be judged continuously by a deterministic oracle rather than by hope or sparse production samples.","feed_headline":"Locksmith loop pushes COBOL–Java tests past search plateaus","feed_subtitle":"Parity-gated harness mutations open locked paths; Java matched COBOL on every accepted case","key_machinery":"The Locksmith Loop: Witness Search (six input-space algorithms until plateau) identifies Locked Paragraphs; an Authoring Layer proposes Mutation Skills (e.g. dispatcher-arm or call-injection) applied to both COBOL mock and Java; the deterministic Parity Gate compares paragraphs hit, ordered stub log, and terminal state and is the sole acceptance oracle.","core_discovery":"The paper claims that alternating multi-algorithm Witness Search with parity-preserving harness mutations—applied symmetrically to a mocked COBOL runtime and its generated Java target, and retained only when coverage rises and a three-axis Parity Gate passes—systematically opens execution regions ordinary input search cannot reach, and that under this regime the migrated Java matched the COBOL reference on every accepted test case across three case studies up to 4,114 source lines.","pith_inferences":["The method’s value may concentrate on lift-and-shift cutover confidence; it deliberately preserves legacy bugs, so semantic correctness work still needs a separate specification oracle.","Production residual gaps behind multi-step internal switches suggest the next bottleneck is coordinated fixture synthesis across long state chains, not more generic fuzzers.","If mutation skills transfer poorly across related gates without human help, skill catalogs and transfer metrics become as important as raw branch counts for unattended runs."],"forward_implications":["Migration teams can treat behavioral parity under a shared harness as a continuous acceptance gate for agent-generated Java, not only a late integration check.","When multi-algorithm input search converges within a few branches, remaining gaps are treated as structural harness limits and become candidates for symmetric mutation rather than more sampling budget.","Engineers shift toward defining outcome goals and deterministic oracles while the loop authors reusable mutation skills and migrator fixes from parity divergences.","The same search-then-mutate pattern is offered for other 1:1 behavioral migrations where both sides can be instrumented and compared under identical witnesses."],"fun_headline_variants":["Locksmith Loop opens COBOL paths input search cannot reach","Parity-gated mutations lift COBOL–Java coverage past plateaus","Witness Search plus harness locks hit 91.9% on legacy port","Java matched COBOL on every accepted Locksmith test case","Agentic loop unlocks locked paragraphs in COBOL migration"],"cache_read_input_tokens":128,"weakest_assumption_plain":"Mutations that force external values or paragraph entry on both sides still count as fair tests of migration equivalence for real production control flow, not mainly artificial paths the live system would never take that way.","fun_headline_variants_meta":{"raw":{"variants":["Locksmith Loop opens COBOL paths input search cannot reach","Parity-gated mutations lift COBOL–Java coverage past plateaus","Witness Search plus harness locks hit 91.9% on legacy port","Java matched COBOL on every accepted Locksmith test case","Agentic loop unlocks locked paragraphs in COBOL migration"]},"model":"grok-4.5","effort":"low","cost_usd":0.0029,"raw_usage":{"total_tokens":1039,"prompt_tokens":779,"num_sources_used":0,"completion_tokens":65,"cost_in_usd_ticks":29004000,"prompt_tokens_details":{"text_tokens":779,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":195,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":779,"tokens_out":65,"duration_ms":4844,"temperature":1.0,"reasoning_tokens":195,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-31T12:37:19.849278+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"On a held-out COBOL–Java pair, run Locksmith to high mock coverage, then replay the accepted witnesses and any production-like inputs without the force-set mutations: a systematic rise in parity failures or large drop in true reachable coverage would refute the claim that the loop validates migration readiness.","supporting_citations":[],"review_version":1}