{"id":"100b1798-b7f0-4d8e-9f38-77a290b6a383","arxiv_id":"2607.04058","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":4,"one_line_summary":"Compilation success and static developer tests are unreliable proxies for semantic correctness of LLM-translated HPC code; metamorphic fuzzing plus differential testing is required to expose input-dependent errors.","lead":"Kaizen is a testing framework that uses source-code mutations, input fuzzing, and differential runs to check whether LLM-translated HPC programs actually compute the same scientific results as the originals. It shows that compilation success and fixed developer tests often hide input-dependent semantic bugs in CUDA-to-OpenMP ports.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"No significant objection identified","rationale":"The paper's strongest claim is empirical and narrowly scoped: compilation and fixed seed tests do not guarantee behavioral correctness of LLM-translated HPC kernels, and metamorphic-plus-differential testing systematically surfaces input-dependent semantic errors. Table 4 shows multiple applications with near-perfect compilability yet zero correctness; the LIF example shows a missing-statement fault that passes all developer seeds yet fails under Kaizen-generated inputs. These results stand even under a conservative reading of mutation fidelity. The taxonomies (9 compile-time, 6 semantic, 27 full-program) and the kernel-vs-full-program gap (Table 7) are additional contributions that do not rest on the contested assumption. Limitations (16-app correctness subset, single-seed analysis for correctness, non-uniform built-in tolerances) are acknowledged and do not reverse the bimodal pattern. The reader's ACCEPT / HIGH / low-risk assessment is therefore appropriate; no adjustment is warranted.","tokens_in":27568,"tokens_out":539,"duration_ms":5134,"concrete_test":"Independently re-execute the LIF differential-testing campaign (Table 1 and Section 3.2) on the original unmutated CUDA kernel and its ChatPORT CL_13B OpenMP translation using the same two fuzzed inputs (10,128,1) and (2048,200,200). If the same output mismatches appear, the central claim holds without relying on mutation semantics.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The reader's weakest assumption (that the 15 mutation operators fully preserve semantics) is real but does not undermine the central claim. The claim is that compilation success and fixed developer tests are insufficient proxies for semantic correctness of LLM-translated HPC code. That claim is established by the bimodal compilability-vs-correctness results (Table 4) and the LIF motivating example (Section 3.2), both of which rest on differential testing of successfully compiled translations against the original CUDA programs under diverse runtime inputs. Even if a mutation occasionally introduced a latent change that seed tests missed, the observed divergences would still demonstrate that fixed tests fail to catch behavioral errors; the attribution of every failure solely to the LLM would be slightly noisier, but the insufficiency of the proxies would remain intact. The paper already validates each mutated variant by compile-and-run against developer seeds before translation (Section 4.2), and the LIF bug is an omission of an original statement rather than a mutation artifact. No other internal inconsistency or unsupported leap threatens the strongest claim.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper presents Kaizen, a metamorphic fuzzing and differential-testing framework for assessing behavioral correctness of LLM-translated HPC programs. It combines semantics-preserving source mutations (15 operators), grammar-based runtime input fuzzing, and cross-platform differential execution with application-appropriate error norms. The evaluation targets CUDA-to-OpenMP translation of HeCBench applications (16 for kernel-level correctness, 47 for full-program compile-time analysis) using three ChatPORT fine-tuned models at kernel and full-program granularity. Four RQs report: (1) a large, application-dependent gap between compilability and semantic correctness (Table 4); (2) nine kernel-level compile-time error categories; (3) six input-dependent semantic error categories exposed only by differential testing; and (4) substantially worse full-program outcomes (up to 27 compile-time categories) than kernel-level translation. A replication package is provided.","tokens_in":27778,"tokens_out":1267,"duration_ms":18526,"significance":"If the results hold, the paper makes a clear and timely contribution to evaluation methodology for LLM-assisted HPC porting: it shows that the dominant proxies (compilation success, token similarity, fixed developer tests) systematically overestimate correctness for scientific codes, and it supplies concrete taxonomies of both compile-time and semantic failures that future translators and fine-tuning efforts can target. Strengths include a well-motivated LIF case study, clearly defined compilability/correctness metrics, multi-LLM and multi-granularity comparison, honest threats discussion, and a public replication package. The work is empirical rather than theoretical, but the evidence is proportionate to the claim and useful to both SE testing and HPC portability communities.","major_comments":[{"comment":"Section 4.2 states that each mutated variant is validated by compile-and-run against developer-provided seed inputs before translation. The paper’s own central claim (RQ1/RQ3) is that those seed tests are insufficient to establish semantic preservation. This creates a methodological tension: residual mutation-induced behavioral changes that seeds miss could be mis-attributed to the LLM. The LIF omission and Table 4 bimodal pattern still support the proxy-insufficiency claim, but the manuscript should either strengthen semantic-preservation evidence (e.g., additional oracles, multi-seed differential checks of mutant vs. original before translation, or a quantified residual-risk discussion) or explicitly bound claims about pure LLM attribution.","section":"Section 4.2 / RQ1–RQ3"},{"comment":"Table 7 and RQ4 compare kernel-level fine-tuned ChatPORT models on full-program translation and conclude that full-program translation is substantially harder. For two of three variants, the base models outperform the kernel-fine-tuned models on full-program compilability/correctness, which the text notes as over-specialization. The hardness claim therefore confounds task difficulty with training specialization. Please separate these effects more cleanly (e.g., emphasize base-model full-program results as the cleaner difficulty baseline, or reframe RQ4 as “kernel-specialized models transfer poorly to full programs” rather than pure granularity hardness).","section":"Section 5.5.4, Table 7, RQ4"},{"comment":"External validity is limited to CUDA→OpenMP and three ChatPORT-family models (Section 5.1–5.2, Section 8). That scope is acceptable for a first study, but several abstract/conclusion statements generalize to “LLM-translated HPC applications” and “LLM-assisted HPC code translations.” Please tighten those claims to the evaluated setting, or add at least a small pilot on another pair (e.g., CUDA→SYCL using the authors’ own SYCL extension) if claiming broader applicability.","section":"Abstract, Section 9, Section 8"}],"minor_comments":[{"comment":"Figure 1 is described as a partial diff of the LIF translation but is not rendered as readable code in the text dump; ensure the published figure shows the omitted ref_time -= dt line clearly side-by-side.","section":"Section 3.2, Figure 1"},{"comment":"Table 4 mixes denominators across Compilability and Correctness columns (different N per seed/campaign). A short footnote explaining when N differs would reduce reader confusion.","section":"Table 4"},{"comment":"Section 5.1 says 18 HeCBench applications include built-in verification in one place and evaluates 16 elsewhere; reconcile the counts.","section":"Section 4.3.2 / Section 5.1"},{"comment":"Typo/consistency: “OpemMP” in Figure 2 caption path labels; “𝐶ℎ𝑎𝑡𝑃𝑂𝑅𝑇” formatting is inconsistent with plain ChatPORT in places.","section":"Figure 2, throughout"},{"comment":"Related work could briefly contrast Kaizen’s source-level metamorphic mutations with EMI-style dead-code insertion for compilers (already cited) to clarify novelty for LLM translation evaluation.","section":"Section 6"}],"recommendation":"minor_revision","confidential_remarks":"The authors evaluate their own prior ChatPORT models; this is disclosed and the results are not uniformly flattering (full-program collapse, zero-correctness apps), so I do not see a conflict that undermines the findings. Fit is strong for an SE/testing or HPC software venue. I would not block on the mutation-validation caveat if the authors add a clear residual-risk paragraph; the proxy-insufficiency result is the load-bearing contribution and is well supported by Table 4 and the LIF example."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The punchline is simple: compilation success and fixed developer tests are bad proxies for semantic correctness of LLM-translated HPC code, and Kaizen is a practical way to show it. The LIF refractory-period omission is a clean motivating case, and Table 4’s bimodal pattern (near-perfect compilability with zero correctness on lif/winograd/flip/nlll versus near-perfect correctness on dense-embedding/cross) makes the point without rhetoric.\n\nWhat is new is the combination applied to heterogeneous HPC translation, plus the taxonomies: nine kernel-level compile categories, 27 full-program categories, and especially the six input-dependent semantic categories (intermediate-variable elimination, execution-model assumption transfer, loop-bound error, missing-statement fault, shared-memory scope mistranslation, multi-dimensional index flattening). Those last six are the real contribution; they are not in ChatPORT, HeteroBugDetect, or the general LLM-translation literature. The paper is careful about self-citation: ChatPORT is the system under test, not circular support. Metrics are defined cleanly, three fine-tuned models and both granularities are compared, and a replication package is promised.\n\nSoft spots are real but proportionate. Correctness measurement is on 16 apps (not all 47), single-seed analysis in places, and application-specific tolerances mean the oracle is not uniform. The 15 mutation operators are validated only against seed tests, so a latent floating-point or UB change could occasionally muddy attribution to the LLM. That does not break the central claim: even with noisier attribution, the divergences still show that fixed tests miss behavioral errors. Full-program results are weak (0–28% compilability), which the paper itself treats as evidence that kernel fine-tuning does not transfer, not as a solved problem.\n\nThis is for people building or evaluating LLM porting tools and for HPC groups who care about silent numerical bugs. It deserves a serious referee. I would bring it to reading group and expect to cite the taxonomies and the evaluation methodology.","headline":"Solid empirical framework and first real taxonomies showing that compile+seed-tests are unreliable for LLM CUDA-to-OpenMP ports; soft spots are scope and attribution noise, not the central claim.","tokens_in":28453,"tokens_out":529,"would_cite":true,"duration_ms":6544,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Compilation success and fixed tests do not guarantee that LLM-translated HPC code is scientifically correct.","keywords":["LLM code translation","HPC portability","metamorphic testing","differential testing","CUDA to OpenMP","semantic correctness","fuzzing"],"falsifier":"Find a Kaizen-generated CUDA variant that still compiles and matches the original on all seed inputs yet produces numerically different results on a later fuzzed input; any such case would break the claim that translation failures can be cleanly attributed to the LLM rather than to the mutations themselves.","tokens_in":28457,"feed_emoji":"🔬","tokens_out":629,"duration_ms":6196,"temperature":0.7,"pith_summary":"Large language models are now used to port scientific codes from CUDA to portable models such as OpenMP, but the usual evaluation yardsticks—whether the code compiles, how similar the tokens look, and whether a handful of developer-written tests pass—do not guarantee that the translated program still computes the right answers. This paper introduces Kaizen, a framework that mutates source programs into semantically equivalent variants, generates diverse runtime inputs with grammar-based fuzzing, and then compares original and translated outputs on those inputs. Across sixteen scientific applications from seven domains and three fine-tuned translators, Kaizen shows that programs which compile and pass the original tests can still produce wrong scientific results under inputs the fixed tests never exercise. The same evaluation also surfaces systematic compile-time error patterns (nine categories for kernels, twenty-seven for full programs) and six categories of input-dependent semantic faults. Full-program translation proves far harder than translating isolated kernels. The practical message is that correctness-oriented evaluation is required before LLM-assisted ports of scientific software can be trusted.","feed_headline":"Compiling is not enough: LLM ports of science code still fail","feed_subtitle":"Fuzzing and differential tests expose wrong answers that fixed benchmarks miss","key_machinery":"Kaizen: a two-level metamorphic-fuzzing and differential-testing pipeline that first generates semantics-preserving source variants (fifteen grammar-based mutation operators validated by compile-and-run checks) and then explores the runtime input space of both original and translated programs, flagging any output divergence beyond application-defined error norms as a correctness failure.","core_discovery":"Neither compilation success nor developer-written tests from static benchmarks reliably ensure semantic correctness of LLM-translated HPC applications. Programs that compile and pass those tests can still produce incorrect scientific results under inputs the fixed tests never exercise; only metamorphic source fuzzing combined with differential testing on diverse runtime inputs systematically exposes the six categories of input-dependent semantic errors that survive compilation.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["LLM HPC ports compile and pass tests yet yield wrong science results","Kaizen fuzzing exposes semantic bugs static tests miss in LLM translations","Compilation fails as correctness proxy for LLM-ported scientific codes","Input-dependent errors survive compile and tests in LLM HPC translations","Full-program LLM ports far harder than kernels to get scientifically right"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"The fifteen source mutations, once they compile and pass the developer seed tests, truly leave the original program’s computational semantics unchanged for every variant.","fun_headline_variants_meta":{"raw":{"variants":["LLM HPC ports compile and pass tests yet yield wrong science results","Kaizen fuzzing exposes semantic bugs static tests miss in LLM translations","Compilation fails as correctness proxy for LLM-ported scientific codes","Input-dependent errors survive compile and tests in LLM HPC translations","Full-program LLM ports far harder than kernels to get scientifically right"]},"model":"grok-4.5","effort":"low","cost_usd":0.004144,"raw_usage":{"total_tokens":1296,"prompt_tokens":812,"num_sources_used":0,"completion_tokens":70,"cost_in_usd_ticks":41440000,"prompt_tokens_details":{"text_tokens":812,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":414,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":812,"tokens_out":70,"duration_ms":3502,"temperature":1.0,"reasoning_tokens":414,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-11T21:57:52.041622+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Find a Kaizen-generated CUDA variant that still compiles and matches the original on all seed inputs yet produces numerically different results on a later fuzzed input; any such case would break the claim that translation failures can be cleanly attributed to the LLM rather than to the mutations themselves.","supporting_citations":[],"review_version":1}