Pith. sign in

REVIEW 3 major objections 5 minor 42 references

Compilation success and fixed tests do not guarantee that LLM-translated HPC code is scientifically correct.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 21:57 UTC pith:HJXX2HGM

load-bearing objection Solid empirical framework and first real taxonomies showing that compile+seed-tests are unreliable for LLM CUDA-to-OpenMP ports; soft spots are scope and attribution noise, not the central claim. the 3 major comments →

arxiv 2607.04058 v1 pith:HJXX2HGM submitted 2026-07-04 cs.SE cs.PL

Kaizen: Metamorphic Fuzzing and Differential Testing for LLM-Translated HPC Applications

Oscar Ludwig , Ninad Anklesaria , Zheming Jin , Swaroop Pophale , Kausar Moshood , Christian J. DeVore , Brandon Gill , Cassius Villareal
show 2 more authors
Keita Teranishi Manish Motwani
This is my paper
classification cs.SE cs.PL
keywords LLM code translationHPC portabilitymetamorphic testingdifferential testingCUDA to OpenMPsemantic correctnessfuzzing
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Large language models are now used to port scientific codes from CUDA to portable models such as OpenMP, but the usual evaluation yardsticks—whether the code compiles, how similar the tokens look, and whether a handful of developer-written tests pass—do not guarantee that the translated program still computes the right answers. This paper introduces Kaizen, a framework that mutates source programs into semantically equivalent variants, generates diverse runtime inputs with grammar-based fuzzing, and then compares original and translated outputs on those inputs. Across sixteen scientific applications from seven domains and three fine-tuned translators, Kaizen shows that programs which compile and pass the original tests can still produce wrong scientific results under inputs the fixed tests never exercise. The same evaluation also surfaces systematic compile-time error patterns (nine categories for kernels, twenty-seven for full programs) and six categories of input-dependent semantic faults. Full-program translation proves far harder than translating isolated kernels. The practical message is that correctness-oriented evaluation is required before LLM-assisted ports of scientific software can be trusted.

Core claim

Neither compilation success nor developer-written tests from static benchmarks reliably ensure semantic correctness of LLM-translated HPC applications. Programs that compile and pass those tests can still produce incorrect scientific results under inputs the fixed tests never exercise; only metamorphic source fuzzing combined with differential testing on diverse runtime inputs systematically exposes the six categories of input-dependent semantic errors that survive compilation.

What carries the argument

Kaizen: a two-level metamorphic-fuzzing and differential-testing pipeline that first generates semantics-preserving source variants (fifteen grammar-based mutation operators validated by compile-and-run checks) and then explores the runtime input space of both original and translated programs, flagging any output divergence beyond application-defined error norms as a correctness failure.

Load-bearing premise

The fifteen source mutations, once they compile and pass the developer seed tests, truly leave the original program’s computational semantics unchanged for every variant.

What would settle it

Find a Kaizen-generated CUDA variant that still compiles and matches the original on all seed inputs yet produces numerically different results on a later fuzzed input; any such case would break the claim that translation failures can be cleanly attributed to the LLM rather than to the mutations themselves.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents Kaizen, a metamorphic fuzzing and differential-testing framework for assessing behavioral correctness of LLM-translated HPC programs. It combines semantics-preserving source mutations (15 operators), grammar-based runtime input fuzzing, and cross-platform differential execution with application-appropriate error norms. The evaluation targets CUDA-to-OpenMP translation of HeCBench applications (16 for kernel-level correctness, 47 for full-program compile-time analysis) using three ChatPORT fine-tuned models at kernel and full-program granularity. Four RQs report: (1) a large, application-dependent gap between compilability and semantic correctness (Table 4); (2) nine kernel-level compile-time error categories; (3) six input-dependent semantic error categories exposed only by differential testing; and (4) substantially worse full-program outcomes (up to 27 compile-time categories) than kernel-level translation. A replication package is provided.

Significance. If the results hold, the paper makes a clear and timely contribution to evaluation methodology for LLM-assisted HPC porting: it shows that the dominant proxies (compilation success, token similarity, fixed developer tests) systematically overestimate correctness for scientific codes, and it supplies concrete taxonomies of both compile-time and semantic failures that future translators and fine-tuning efforts can target. Strengths include a well-motivated LIF case study, clearly defined compilability/correctness metrics, multi-LLM and multi-granularity comparison, honest threats discussion, and a public replication package. The work is empirical rather than theoretical, but the evidence is proportionate to the claim and useful to both SE testing and HPC portability communities.

major comments (3)
  1. [Section 4.2 / RQ1–RQ3] Section 4.2 states that each mutated variant is validated by compile-and-run against developer-provided seed inputs before translation. The paper’s own central claim (RQ1/RQ3) is that those seed tests are insufficient to establish semantic preservation. This creates a methodological tension: residual mutation-induced behavioral changes that seeds miss could be mis-attributed to the LLM. The LIF omission and Table 4 bimodal pattern still support the proxy-insufficiency claim, but the manuscript should either strengthen semantic-preservation evidence (e.g., additional oracles, multi-seed differential checks of mutant vs. original before translation, or a quantified residual-risk discussion) or explicitly bound claims about pure LLM attribution.
  2. [Section 5.5.4, Table 7, RQ4] Table 7 and RQ4 compare kernel-level fine-tuned ChatPORT models on full-program translation and conclude that full-program translation is substantially harder. For two of three variants, the base models outperform the kernel-fine-tuned models on full-program compilability/correctness, which the text notes as over-specialization. The hardness claim therefore confounds task difficulty with training specialization. Please separate these effects more cleanly (e.g., emphasize base-model full-program results as the cleaner difficulty baseline, or reframe RQ4 as “kernel-specialized models transfer poorly to full programs” rather than pure granularity hardness).
  3. [Abstract, Section 9, Section 8] External validity is limited to CUDA→OpenMP and three ChatPORT-family models (Section 5.1–5.2, Section 8). That scope is acceptable for a first study, but several abstract/conclusion statements generalize to “LLM-translated HPC applications” and “LLM-assisted HPC code translations.” Please tighten those claims to the evaluated setting, or add at least a small pilot on another pair (e.g., CUDA→SYCL using the authors’ own SYCL extension) if claiming broader applicability.
minor comments (5)
  1. [Section 3.2, Figure 1] Figure 1 is described as a partial diff of the LIF translation but is not rendered as readable code in the text dump; ensure the published figure shows the omitted ref_time -= dt line clearly side-by-side.
  2. [Table 4] Table 4 mixes denominators across Compilability and Correctness columns (different N per seed/campaign). A short footnote explaining when N differs would reduce reader confusion.
  3. [Section 4.3.2 / Section 5.1] Section 5.1 says 18 HeCBench applications include built-in verification in one place and evaluates 16 elsewhere; reconcile the counts.
  4. [Figure 2, throughout] Typo/consistency: “OpemMP” in Figure 2 caption path labels; “𝐶ℎ𝑎𝑡𝑃𝑂𝑅𝑇” formatting is inconsistent with plain ChatPORT in places.
  5. [Section 6] Related work could briefly contrast Kaizen’s source-level metamorphic mutations with EMI-style dead-code insertion for compilers (already cited) to clarify novelty for LLM translation evaluation.

Circularity Check

0 steps flagged

No significant circularity: empirical evaluation of LLM translators via differential testing against original programs; self-citations name the systems under test, not load-bearing premises.

full rationale

The paper is an empirical software-engineering study. Its central claims (compilation success and fixed developer tests are poor proxies for semantic correctness; six input-dependent semantic-error categories exist; full-program translation is harder) are established by running Kaizen’s metamorphic source mutations, grammar-based input fuzzing, and differential execution of original CUDA versus LLM-translated OpenMP programs on 16 HeCBench applications. Correctness is defined operationally as output agreement (within each application’s built-in tolerance or configurable L1/L2/Max norms) on all fuzzed inputs; no parameters are fitted to produce a desired numerical outcome that is then re-labeled a “prediction.” Self-citations to ChatPORT (and its SYCL extension) simply identify the three fine-tuned LLMs being evaluated; they are not invoked as uniqueness theorems, external mathematical facts, or justifications that force the experimental conclusions. HeteroBugDetect is cited only for methodological comparison of differential-testing techniques. Mutation operators are validated by compile-and-run against developer seed inputs before translation, so downstream divergences are attributed to the LLM rather than to circular re-use of the same data. There are no equations, no self-definitional identities, no fitted-input-called-prediction steps, and no ansatz smuggled via citation. The evaluation is therefore self-contained against external benchmarks and exhibits zero circularity of the kinds enumerated.

Axiom & Free-Parameter Ledger

4 free parameters · 3 axioms · 0 invented entities

As an empirical software-engineering paper the load-bearing content is methodological and observational rather than axiomatic. The free parameters are the concrete experimental knobs (mutation set, budgets, thresholds). The axioms are standard domain assumptions about semantic preservation and floating-point comparison. No new physical or mathematical entities are postulated.

free parameters (4)
  • source-code mutation operator set (M1–M15)
    The 15 operators are hand-designed; different operators could change the distribution of translation failures observed.
  • fuzzing time budgets (5 min source campaigns, 30 min runtime campaigns)
    Chosen for practicality on the available cluster; longer budgets could surface additional bugs.
  • output comparison thresholds / application-specific tolerances
    Correctness uses each HeCBench application's built-in tolerance (e.g., 1e-3 for LIF) or user-set L1/L2/Max norms; the choice directly affects which divergences are counted as failures.
  • full-program translation prompt template
    Engineered on a held-out set of 5 applications; prompt wording is a free design choice that affects measured full-program success rates.
axioms (3)
  • domain assumption The 15 mutation operators plus compile-and-run validation against developer seed inputs preserve the original program's computational semantics.
    Stated in Section 4.2; required to attribute any later failure to the LLM rather than to a broken mutant.
  • domain assumption Configurable error norms (L1/L2/Max or application-built-in tolerances) correctly separate benign floating-point variation from true semantic divergence across CUDA GPU and OpenMP CPU platforms.
    Section 4.3.2; central to the differential-testing oracle.
  • domain assumption HeCBench applications and their developer-written OpenMP host code are suitable oracles for measuring translation correctness.
    Dataset construction in Section 5.1; the paper itself notes that Kaizen sometimes found bugs in the original CUDA code.

pith-pipeline@v1.1.0-grok45 · 31677 in / 2627 out tokens · 25337 ms · 2026-07-11T21:57:52.041622+00:00 · methodology

0 comments
read the original abstract

Large language models (LLMs) are increasingly used to port scientific codes across heterogeneous high-performance computing (HPC) programming models, such as translating CUDA to OpenMP, OpenACC, Kokkos or SYCL. However, current evaluations use compilation success, token-level similarity, or developer-written tests from static benchmarks, which cannot reliably ensure behavioral correctness. We present Kaizen, a metamorphic fuzzing and differential testing framework for evaluating the correctness of LLM-translated HPC code. Kaizen uses metamorphic fuzzing via source-code mutation to generate semantically equivalent programs, grammar-based input fuzzing to explore behavioral diversity, and differential testing to expose semantic divergences between original and translated applications that compile and pass developer-written tests yet produce incorrect scientific results. We evaluate Kaizen on CUDA-to-OpenMP translation of 16 scientific applications from seven domains using three fine-tuned LLMs at kernel-level and full-program granularity. Our evaluation reveals that (1) compilation success is a poor proxy for correctness; (2) LLM-translated programs exhibit systematic compile-time error patterns, with nine categories for kernel-level translation and 27 for full-program translation; (3) semantic errors that survive compilation are often input-dependent and require differential testing to expose; and (4) full-program translation is substantially harder than kernel-level translation. These findings highlight the need for correctness-oriented evaluation of LLM-assisted HPC code translations.

Figures

Figures reproduced from arXiv: 2607.04058 by Brandon Gill, Cassius Villareal, Christian J. DeVore, Kausar Moshood, Keita Teranishi, Manish Motwani, Ninad Anklesaria, Oscar Ludwig, Swaroop Pophale, Zheming Jin.

Figure 1
Figure 1. Figure 1: Partial diff of the CUDA-to-OpenMP translation of Kaizen-modified variant of the Leaky Integrate-and-Fire (LIF) application [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Kaizen framework architecture showing two parallel translation evaluation paths. The kernel-level path extracts and translates [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Prompt template engineered after manually translating a sample of 5 HeCBench programs from CUDA to OpenMP using the [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

42 extracted references · 7 canonical work pages

  1. [1]

    Hartwig Anzt, Axel Huebl, and Xiaoye S. Li. 2024. Then and Now: Improving Software Portability, Productivity, and 100×Performance.Computing in Science & Engineering26, 1 (2024), 61–70. doi:10.1109/MCSE.2024.3387302

    doi:10.1109/mcse.2024.3387302 2024

  2. [2]

    Cornelius Aschermann, Tommaso Frassetto, Thorsten Holz, Patrick Jauernig, Ahmad-Reza Sadeghi, and Daniel Teuchert. 2019. NAUTILUS: Fishing for Deep Bugs with Grammars.Proceedings 2019 Network and Distributed System Security Symposium(2019). https://api.semanticscholar.org/ CorpusID:69790362

    2019

  3. [3]

    Bairavasundaram, Garth R

    Lakshmi N. Bairavasundaram, Garth R. Goodson, Shankar Pasupathy, and Jacob Schindler. 2008. An Analysis of Latent Sector Errors in Disk Drives. InSIGMETRICS. 289–300

    2008

  4. [4]

    Tomer Bitan, Tal Kadosh, Erel Kaplan, Shira Meiri, Le Chen, Peter Morales, Niranjan Hasabnis, and Gal Oren. 2025. UniPar: A Unified LLM-Based Framework for Parallel and Accelerated Code Translation in HPC. In2025 IEEE High Performance Extreme Computing Conference (HPEC). IEEE. doi:10.1109/HPEC67600.2025.11196677

    doi:10.1109/hpec67600.2025.11196677 2025

  5. [5]

    Carter Edwards, Christian R

    H. Carter Edwards, Christian R. Trott, and Daniel Sunderland. 2014. Kokkos: Enabling manycore performance portability through polymorphic memory access patterns.J. Parallel and Distrib. Comput.74, 12 (2014), 3202–3216. Domain-Specific Languages and High-Level Frameworks for High-Performance Computing. doi:10.1016/j.jpdc.2014.07.003

    doi:10.1016/j.jpdc.2014.07.003 2014

  6. [6]

    Le Chen, Bin Lei, Dunzhi Zhou, Pei-Hung Lin, Chunhua Liao, Caiwen Ding, and Ali Jannesari. 2025. Fortran2CPP: Automating Fortran-to-C++ Translation using LLMs via Multi-Turn Dialogue and Dual-Agent Integration. InarXiv preprint arXiv:2412.19770

    Pith/arXiv arXiv 2025

  7. [7]

    1998.Metamorphic Testing: A New Approach for Generating Next Test Cases

    Tsong Yueh Chen, Shing-Chi Cheung, and Siu-Ming Yiu. 1998.Metamorphic Testing: A New Approach for Generating Next Test Cases. Technical Report HKUST-CS98-01. Department of Computer Science, The Hong Kong University of Science and Technology, Hong Kong

    1998

  8. [8]

    Fernando Chirigati, Dennis Shasha, and Juliana Freire. 2013. ReproZip: Using Provenance to Support Computational Reproducibility. InUSENIX Workshop on the Theory and Practice of Provenance (TaPP)

    2013

  9. [9]

    Steven Cho, Stefano Ruberto, and Valerio Terragni. 2025. Metamorphic Testing of Large Language Models for Natural Language Processing. In2025 IEEE International Conference on Software Maintenance and Evolution (ICSME). 174–186. doi:10.1109/ICSME64153.2025.00025 Kaizen : Metamorphic Fuzzing and Differential Testing for LLM-Translated HPC Applications 27

    doi:10.1109/icsme64153.2025.00025 2025

  10. [10]

    Dearing, Yiheng Tao, Xingfu Wu, Zhiling Lan, and Valerie Taylor

    Matthew T. Dearing, Yiheng Tao, Xingfu Wu, Zhiling Lan, and Valerie Taylor. 2024. LASSI: An LLM-based Automated Self-Correcting Pipeline for Translating Parallel Scientific Codes. InProceedings of the IEEE International Conference on Cluster Computing Workshops (CLUSTER Workshops). IEEE, 136–143

    2024

  11. [11]

    James Demmel and Huan D. Nguyen. 2016.Efficient Reproducible Floating Point Summation and BLAS. Technical Report UCB/EECS-2016-121. EECS Department, UC Berkeley

    2016

  12. [12]

    Bernholdt, and Sunita Chandrasekaran

    Jose Monsalve Diaz, Swaroop Pophale, Oscar Hernandez, David E. Bernholdt, and Sunita Chandrasekaran. 2018. OpenMP 4.5 Validation and Verification Suite for Device Offload. InEvolving OpenMP for Evolving Architectures, Bronis R. de Supinski, Pedro Valero-Lara, Xavier Martorell, Sergi Mateo Bellido, and Jesus Labarta (Eds.). Springer International Publishin...

    2018

  13. [13]

    Evans and Alberto Savoia

    Robert B. Evans and Alberto Savoia. 2007. Differential testing: a new approach to change detection. InThe 6th Joint Meeting on European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering: Companion Papers(Dubrovnik, Croatia) (ESEC-FSE companion ’07). Association for Computing Machinery, New York, NY, U...

    doi:10.1145/1295014.1295038 2007

  14. [14]

    Ferreira et al

    Kurt B. Ferreira et al. 2008. Characterizing the Impact of System Noise on Scaling of Parallel Applications. InSC ’08: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

    2008

  15. [15]

    Kistler, Richard Naud, and Liam Paninski

    Wulfram Gerstner, Werner M. Kistler, Richard Naud, and Liam Paninski. 2014.Neuronal Dynamics: From Single Neurons to Networks and Models of Cognition. Cambridge University Press. https://neuronaldynamics.epfl.ch/online/Ch1.S3.html

    2014

  16. [16]

    Ganesh Gopalakrishnan et al. 2008. ISP: A Tool for Model Checking MPI Programs. InProceedings of the 13th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming (PPoPP). 285–286

    2008

  17. [17]

    2020.SYCL™2020 Specification (revision 11)

    The Khronos Group Inc. 2020.SYCL™2020 Specification (revision 11). Accessed: 2025-10-25. https://registry.khronos.org/SYCL/specs/sycl- 2020/html/sycl-2020.html

    2020

  18. [18]

    Zheming Jin, Swaroop Pophale, and Keita Teranishi. 2025. Enhancing ChatPORT with CUDA-to-SYCL Kernel Translation Capability. InProceedings of the SC ’25 Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC Workshops ’25). Association for Computing Machinery, New York, NY, USA, 524–533. doi:10.1145/...

    work page doi:10.1145/3731599.3767398 2025

  19. [19]

    Zheming Jin and Jeffrey S. Vetter. 2023. A Benchmark Suite for Improving Performance Portability of the SYCL Programming Model. In2023 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS). 325–327. doi:10.1109/ISPASS57527.2023.00041

    doi:10.1109/ispass57527.2023.00041 2023

  20. [20]

    Upulee Kanewala and James M. Bieman. 2014. Testing scientific software: A systematic literature review.Information and Software Technology56, 10 (2014), 1219–1232. doi:10.1016/j.infsof.2014.05.006

    work page doi:10.1016/j.infsof.2014.05.006 2014

  21. [21]

    Hongliang Liang, Xiaoxiao Pei, Xiaodong Jia, Wuwei Shen, and Jian Zhang. 2018. Fuzzing: State of the Art.IEEE Transactions on Reliability67, 3 (2018), 1199–1218. doi:10.1109/TR.2018.2834476

    work page doi:10.1109/tr.2018.2834476 2018

  22. [22]

    David Luebke. 2008. CUDA: Scalable parallel programming for high-performance scientific computing. In2008 5th IEEE International Symposium on Biomedical Imaging: From Nano to Macro. IEEE (Institute of Electrical and Electronics Engineers), Piscataway, NJ, USA, 836–838. doi:10.1109/ISBI. 2008.4541126

    work page doi:10.1109/isbi 2008

  23. [23]

    McKeeman

    William M. McKeeman. 1998. Differential Testing for Software.Digital Technical Journal10, 1 (1998), 100–107

    1998

  24. [24]

    Manish Motwani, Aakash Kulkarni, Yunhan Qiao, Matthew Davis, and Ziyan Chen. 2025. LLM-Guided Differential Fuzzing for Detecting Platform-Specific Bugs in Scientific Applications. In2025 Conference on AI x Software Engineering (AIxSE). IEEE, 59–66

    2025

  25. [25]

    2020.OpenACC Application Programming Interface Version 3.0

    OpenACC Committee. 2020.OpenACC Application Programming Interface Version 3.0. Accessed: 2025-05-19. https://www.openacc.org/

    2020

  26. [26]

    OpenMP Architecture Review Board. 2024. OpenMP Application Program Interface Version 6.0. https://www.openmp.org/wp-content/uploads/ OpenMP-API-Specification-6-0.pdf

    2024

  27. [27]

    Rangeet Pan, Ali Reza Ibrahimzada, Rahul Krishna, Divya Sankar, Lambert Pouguem Wassi, Michele Merler, Boris Sobolev, Raju Pavuluri, Saurabh Sinha, and Reyhaneh Jabbarvand. 2024. Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code. InProceedings of the IEEE/ACM 46th International Conference on Software Engineeri...

    doi:10.1145/3597503.3639226 2024

  28. [28]

    Swaroop Pophale, Zheming Jin, and Keita Teranishi. 2025. ChatPORT: Fine-Tuned LLM for Easy Code PORTing. InOpenMP: Balancing Productivity and Performance Portability: 21st International Workshop on OpenMP, IWOMP 2025, Charlotte, NC, USA, October 1-3, 2025 Proceedings(Charlotte, NC, USA). Springer-Verlag, Berlin, Heidelberg, 197–211. doi:10.1007/978-3-032-...

    work page doi:10.1007/978-3-032-06343-4_13 2025

  29. [29]

    de Supinski, Wolfgang E

    Joachim Protze, Tobias Hilbrich, Martin Schulz, Bronis R. de Supinski, Wolfgang E. Nagel, and Matthias S. Mueller. 2014. MPI Runtime Error Detection with MUST: A Scalable and Crash-Safe Approach. In2014 43rd International Conference on Parallel Processing Workshops. 206–215. doi:10.1109/ICPPW.2014.37

    work page doi:10.1109/icppw.2014.37 2014

  30. [30]

    Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020. Codebleu: a method for automatic evaluation of code synthesis.arXiv preprint arXiv:2009.10297(2020)

    Pith/arXiv arXiv 2020

  31. [31]

    2021.The Coding Manual for Qualitative Researchers(4th ed.)

    Johnny Saldaña. 2021.The Coding Manual for Qualitative Researchers(4th ed.). SAGE Publications

    2021

  32. [32]

    Geof Sawaya, Michael Bentley, Ian Briggs, Ganesh Gopalakrishnan, and Dong H. Ahn. 2017. FLiT: Cross-platform floating-point result-consistency tester and workload. In2017 IEEE International Symposium on Workload Characterization (IISWC). 229–238. doi:10.1109/IISWC.2017.8167780

    work page doi:10.1109/iiswc.2017.8167780 2017

  33. [33]

    Sergio Segura, Gordon Fraser, Ana B Sanchez, and Antonio Ruiz-Cortés. 2016. A survey on metamorphic testing.IEEE Transactions on software engineering42, 9 (2016), 805–824

    2016

  34. [34]

    Sameer Shende and Allen D. Malony. 2006. Performance Regression Testing for HPC Codes. InInternational Conference on Computational Science (ICCS). 185–193. 28 Ludwig et al

    2006

  35. [35]

    Victoria Stodden et al. 2014. Best Practices for Computational Science: Software Infrastructure and Environments for Reproducible and Extensible Research.Computing in Science & Engineering17, 6 (2014), 21–31

    2014

  36. [36]

    Chengnian Sun, Vu Le, and Zhendong Su. 2016. Finding compiler bugs via live code mutation.SIGPLAN Not.51, 10 (Oct. 2016), 849–863. doi:10.1145/3022671.2984038

    doi:10.1145/3022671.2984038 2016

  37. [37]

    Qiuming Tao, Wei Wu, Chen Zhao, and Wuwei Shen. 2010. An automatic testing approach for compiler based on metamorphic testing technique. In 2010 Asia Pacific Software Engineering Conference. IEEE, 270–279

    2010

  38. [38]

    Ahmed, Azalia Mirhoseini, and Ali Jannesari

    Ali TehraniJamsaz, Arijit Bhattacharjee, Le Chen, Nesreen K. Ahmed, Azalia Mirhoseini, and Ali Jannesari. 2024. CodeRosetta: Pushing the Boundaries of Unsupervised Code Translation for Parallel Programming. InAdvances in Neural Information Processing Systems (NeurIPS), Vol. 37. 100965–100999

    2024

  39. [39]

    Dongwei Xiao, Zhibo Liu, Yuanyuan Yuan, Qi Pang, and Shuai Wang. 2022. Metamorphic testing of deep learning compilers.Proceedings of the ACM on Measurement and Analysis of Computing Systems6, 1 (2022), 1–28

    2022

  40. [40]

    Xuejun Yang, Yang Chen, Eric Eide, and John Regehr. 2011. Finding and understanding bugs in C compilers. InProceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation(San Jose, California, USA)(PLDI ’11). Association for Computing Machinery, New York, NY, USA, 283–294. doi:10.1145/1993498.1993532

    doi:10.1145/1993498.1993532 2011

  41. [41]

    Hanliang Zhang, Cristina David, Meng Wang, Brandon Paulsen, and Daniel Kroening. 2025. Scalable, Validated Code Translation of Entire Projects using Large Language Models.Proc. ACM Program. Lang.9, PLDI, Article 212 (June 2025), 26 pages. doi:10.1145/3729315

    doi:10.1145/3729315 2025

  42. [42]

    Qian Zhang, Jiyuan Wang, and Miryung Kim. 2021. HeteroFuzz: fuzz testing to detect platform dependent divergence for heterogeneous applications. InProceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (Athens, Greece)(ESEC/FSE 2021). Association for Computing Machiner...

    doi:10.1145/3468264.3468610 2021