Pith. sign in

REVIEW 3 major objections 5 minor 70 references

ParBench: A Benchmark for Reliable Evaluation of LLM Parallel Code Translation

T0 review · 3 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ParBench argues that LLM parallel-code translation becomes measurable once the build, run, and verification infrastructure is fixed by declarative specs and only computational kernels are translated, reporting pass@1 between 23.9% and 62.7%

desk verdict A useful benchmark substrate with a candid self-audit, but the headline pass rates ride on weak oracles and one non-significant robustness trend is oversold. read the letter →

arxiv 2607.22588 v1 pith:7ZOZRNWC submitted 2026-06-09 cs.AI cs.DC

Samyak Jhaveri , Erel Kaplan , Tom Yotam , Le Chen , Tomer Bitan , Niranjan Hasabnis , Gal Oren This is my paper
classification cs.AIcs.DC
keywords parallelcodetranslationLLMevaluationbenchmarkkernel-centricdeclarativespecificationsbuild-run-verifyharnessASTsourceaugmentationpass@kanalysisdirectionasymmetry
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ParBench tries to establish that LLM parallel-code translation can be measured reliably if the evaluation isolates one capability: rewriting an existing computational kernel from one parallel API (CUDA, OpenMP, OpenCL, or OpenMP target offload) to another, with all surrounding build, run, and verification infrastructure fixed by declarative specifications. On 142 translation tasks drawn from five open-source HPC suites, the authors report pass@1 of 23.9% for the open-weight model Qwen 3.5 and 62.7% for both GPT-5.4 and the code-specialized GPT-5.3-codex. The two proprietary models are statistically indistinguishable, build-stage API adaptation is the dominant failure mode, and direction asymmetry is consistent across models, with removing explicit GPU constructs easier than introducing them. AST-driven source perturbation at four levels tests robustness to surface-form change; the proprietary models hold steady while the open-weight model declines, a pattern the authors flag as descriptive given the conditional subset design. A sympathetic reader would take away that the field now has a reproducible substrate for scoring parallel translation and for locating exactly where it breaks.

What carries the argument

The load-bearing mechanism is the declarative JSON spec, which partitions each task into prompt_payload (kernel source shown to the model), support_files (headers and build context, read-only), verification_only (reference code withheld), and translation_targets (files the model rewrites), plus build, run, and verification blocks. The harness verifies conjunctively: all declared strategies — exit code, stdout pattern, optional numeric comparison or file hash — must pass for a PASS verdict, and failures are classified as EXTRACT_FAIL, BUILD_FAIL, RUN_FAIL, or VERIFY_FAIL. An AST-driven augmentation engine applies six behavior-intended transforms (condition swapping, arithmetic rewriting, vari

What would settle it

Take the 80 weak-oracle specs, produce translations whose numeric output is deliberately corrupted while preserving exit code and stdout banner, and run them through the harness; if a substantial share receives PASS, the reported pass@1 figures measure 'builds and prints' rather than correct translation.

Watch

Extended reading notes

Core claim

The central claim is that kernel-centric, oracle-defined evaluation produces reliable measurements of LLM parallel API translation: each task is a declarative spec that fixes build commands, run arguments, and conjunctive verification checks, and the model rewrites only the kernel source files. On 142 unique L0 tasks, the open-weight Qwen 3.5 reaches pass@1 23.9% and pass@3 35.2%, while GPT-5.4 and the code-specialized GPT-5.3-codex both reach 62.7% pass@1 and are statistically indistinguishable. Build-stage adaptation is the dominant failure (39.1% of Qwen records), direction difficulty is consistent across models — removing explicit GPU constructs (CUDA-to-OpenMP) is easier than introducin

Load-bearing premise

The load-bearing premise is that the declared verification oracles catch material translation errors: 80 of 87 eval-eligible specs check only exit code plus a stdout banner, so a translation that computes wrong numbers but prints the expected output would be scored a pass — the paper itself calls oracle sufficiency its weakest assumption (Appendix H.5) and concedes in its Evaluation Card that 'PASS means numerically faithful' is an invalid claim.

Editorial extensions

If this is right

  • Because build, run, and verify steps are fixed, a model's inability to reconstruct build systems no longer swamps the measurement; failures can be attributed to API adaptation, multi-file coordination, or verification rather than missing scaffolding.
  • Build-stage API adaptation is the binding constraint: most failures are incomplete API-surface mappings, and resampling to pass@3 rescues only a small share, so improving first-attempt mapping quality matters more than sampling more.
  • Direction rankings are stable across models, implying direction difficulty is a property of the API pair; evaluations should report direction-level rates rather than a single aggregate number.
  • Surface perturbation at L1–L4 leaves proprietary-model pass rates above 85%, compatible with robustness to cosmetic source changes; the open-weight model's decline suggests greater reliance on surface form, though the conditional design limits the claim.
  • A code-specialized model shows no measurable advantage over the general-purpose model on these tasks under the same provider and sampling conditions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Since 80 of 87 eval-eligible specs are verified only by exit code plus a stdout banner, the headline pass rates should be read as 'compiles, runs, and prints expected output'; upgrading oracles to numeric comparison or result hashing — which the paper lists as future work — is a direct, testable way to see how much the numbers move.
  • The paper's pipeline already supports iterative repair but reports only single-attempt results; an immediate extension is to feed build and verify errors back to the model and measure whether the dominant build-stage bottleneck shrinks.
  • The augmentations deliberately exclude semantics-altering transforms, so 'not surface-form memorized' is the most the robustness test can show; extending the engine to loop or data-layout changes would probe deeper but requires revalidating behavior preservation.
  • Because direction difficulty tracks how much explicit memory, synchronization, and launch structure the target API forces the model to introduce, the benchmark suggests a predictor for untested directions: near-renaming pairs would be easy, while directions that add host-device boilerplate would be hard.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. ParBench is an executable, kernel-centric benchmark for LLM cross-API parallel code translation. It wraps 87 baseline-verified kernel/API specifications in declarative JSON that fixes build, run, and verification infrastructure, asks models to translate only kernel files, and scores results with a conjunctive build-run-verify harness. The corpus spans CUDA, OpenMP, OpenCL, and OpenMP-target directions, and the paper adds an AST-driven augmentation engine (L1–L4) to probe robustness to surface-form changes. Three models (Qwen 3.5, GPT-5.4, GPT-5.3-codex) are evaluated on 142 L0 tasks and 2,262 total records. Reported results include pass@1 from 23.9% to 62.7%, a dominant BUILD_FAIL failure mode for Qwen (39.1% of records), direction-dependent pass rates, and augmentation robustness patterns. The paper is unusually transparent about its limitations, including weak oracle coverage, L0-conditional augmentation selection, and unmatched sampling conditions.

Significance. If the headline measurements are accepted, ParBench would be a valuable shared evaluation substrate: it isolates kernel translation from repository reconstruction, ships pinned specifications and per-record results, and its failure taxonomy is useful for diagnosing where translation breaks. The artifact appears reproducible, the internal arithmetic is consistent, and the authors explicitly document the KNOWN_FAIL policy, oracle downgrades, and the Evaluation Card. However, the benchmark's central 'reliability' claim rests on an untested oracle-sufficiency assumption: 80 of 87 eval-eligible specs verify only via stdout-pattern plus exit-code checks, so a translation that prints the expected banner while computing wrong numbers receives PASS. The paper itself labels this the weakest assumption and lists 'A PASS means numerically faithful' as an invalid claim. The augmentation analysis is also weakened by L0-conditional filtering and non-significant trend tests. These issues are load-bearing for the central contribution, so the manuscript needs revision before the benchmark can be recommended for adoption as a reliable measure of parallel translation correctness.

major comments (3)
  1. [Section 4 'Oracle strength'; Appendix H.4 item 7; H.5 item 2] The load-bearing reliability claim is undermined by the declared oracle weakness. Of the 87 eval-eligible specs, only 7 use numeric_comparison or file_hash; the remaining 80 use exit_code plus stdout_pattern. As the paper itself concedes (H.4 item 7), a PASS under these oracles does not imply numerical fidelity: a translation with broken indexing, missing reductions, or dropped synchronization can compile, run, print the expected banner, and be recorded as PASS. Because the headline pass@1 23.9–62.7% and the BUILD_FAIL-dominance results are computed over these weak-oracle records, the benchmark currently measures 'compiles, runs, and prints the expected flow' rather than 'translates the parallel kernel correctly.' The paper's careful 'declared-oracle correctness' phrasing is not enough to support the abstract's claim of measuring preservation of low-level parallel semantics. Please eithe
  2. [Section 5.5 and Appendix E.4, Tables 15–16] The augmentation robustness analysis cannot support the conclusion that 'Qwen 3.5's declining augmentation rates suggest greater reliance on surface-form patterns' (Section 6). The augmentation subset is L0-conditional, so it differs across models (50/142 Qwen pairs vs. 99/142 and 97/142 GPT pairs); Table 16 shows Qwen's decline is on a different, smaller subset. On the balanced 12-kernel CUDA-to-OpenMP subset, L1–L4 rates range 75.0–83.3% and the Cochran–Armitage trend is z = −1.84, p = 0.065, i.e., not significant at α = 0.05. The paper does label these results 'descriptive,' but the abstract and Section 6 present them as an empirical finding. Moreover, 7 of 87 baseline-validated variants fail at L3–L4 (all omp_target), yet these non-behavior-preserving variants still enter the model evaluation as prompt inputs (H.5.8), further contaminating the robustness signal. Please either run aug
  3. [Section 5.3 and Appendix E.4, Table 17] The direction-asymmetry claim is stated more strongly than the statistical evidence supports. The paired McNemar tests for the three standard direction pairs are all non-significant at the Bonferroni-corrected level: CUDA↔OMP p=0.180, CUDA↔OpenCL p=1.000, OMP↔OpenCL p=0.289. Yet Section 5.3 concludes that 'direction difficulty is predominantly a property of the translation task, not the model' and lists direction effects as a key empirical finding. The raw per-direction rates (e.g., 0% for OpenCL→CUDA under Qwen vs. 40.3% for CUDA→OMP) are striking descriptively, but with 17–24 paired tasks per direction and wide Wilson intervals, the sample is consistent with substantial sampling variation. Please either report the asymmetric-confidence intervals and power limitations alongside the claim, or soften the conclusion to a descriptive observation requiring larger paired samples.
minor comments (5)
  1. [Section 4 and Appendix B.4] The text says 'temperature is provider-controlled' for GPT models but does not specify what value the provider uses. Please state, if known, or explicitly mark it as unknown/unreported in the configuration table.
  2. [Section 5.2 and Table 10] The 2,262 valid-record total follows from 626+822+814, but Table 10's footnote explains Qwen's 626 vs. 630. For clarity, add the same four-record exclusion note to the main-text sentence where 2,262 is first introduced.
  3. [Figure 15 and Appendix E.2] Transform frequency counts in Figure 15 and the text (e.g., 59/60 for SwapCondition) include KNOWN_FAIL specs, while baseline validation uses only non-KNOWN_FAIL specs. Please annotate this explicitly near Figure 15, as it is easy to misread as being over the eval-eligible corpus.
  4. [Appendix D.5] The text says 'five kernels lack one or more API variants' but then says 'six API variants absent.' Consider rephrasing to avoid ambiguity about whether this is five kernels with six missing variants.
  5. [Section 3.2, Table 1] The row for HeCBench lists 10 kernels, 25 specs, 23 PASS, 2 KF. The earlier funnel description in Appendix D.4 says 20 kernels were curated, 10 in the current corpus. Please clarify in Table 1 that only 10 of the 20 curated HeCBench kernels are included in the current release.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: ParBench's pass rates, failure taxonomy, and augmentation results are empirical measurements, not quantities defined by fitted inputs or the authors' prior claims.

full rationale

ParBench is a benchmark-construction and empirical-evaluation paper, not a derivational one. No headline quantity (pass@1 23.9–62.7%, BUILD_FAIL share, direction asymmetries, augmentation plateaus) is defined in terms of a fitted parameter or a self-cited theorem. The verification oracles are declared in JSON specs and calibrated from original baseline output (Section 2.1, B.1), which is standard test-oracle construction: the model's translation must compile, run, and match the original program's declared output patterns; this does not make the model's pass rate equal to the oracle by construction. The paper's own Evaluation Card explicitly scopes PASS as 'successful build, execution, and satisfaction of the declared verification oracle, not a proof of full semantic equivalence' (Section 2.4), and Appendix H.4 item 7 states that 'A PASS means the translation is numerically faithful' is an invalid claim. The weakest assumption, oracle sufficiency, is named in Appendix H.5 item 2 and its consequences are quantified in Section 4 ('80 use stdout-pattern plus exit-code checks'); these are validity limitations, not circular steps. The augmentation engine is credited to external work (CodeMorph; Zhang et al.) and validated against the baseline harness; the 7 omp_target variants that fail baseline validation are explicitly excluded from robustness claims (Section 2.2, E.2). The L0-conditional augmentation analysis is disclosed as descriptive with survivorship bias (Sections 5.5, E.4), again a scoping caveat rather than a fitted-input-called-prediction. Self-citations (UniPar, ParaCodex, OMPGPT, OMPar, MonoCoder, etc.) appear as related-work positioning or complementary method papers, not as load-bearing justification for ParBench's measurement design; no uniqueness theorem or ansatz is imported from the authors' prior work. Accordingly, no circular step can be quoted, and the paper's results remain externally checkable through the released artifact and pinned platform.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The paper is a measurement substrate, not a derivation: the ledger entries are the hand-adjudicated choices that define what a PASS means and which records count. The four hand-chosen parameters (L0-conditional filter, oracle downgrades, augmentation fractions, KNOWN_FAIL boundaries) directly shape the headline rates and robustness conclusions. The axioms are the paper's own stated assumptions (Appendix H.5), of which oracle sufficiency is explicitly labeled the weakest, and behavior preservation of the augmentation transforms is partially falsified (7/87 variants). No invented entities: the failure taxonomy and augmentation levels are measurement categories, not new physical or conceptual objects.

free parameters (4)
  • L0-conditional augmentation filter = Qwen: 50/142 pairs; GPT-5.4: 99/142; GPT-5.3-codex: 97/142
    Hand-chosen inclusion rule (at least 1 of 3 L0 samples passes) for L1–L4. It sets different denominators per model and directly determines the augmentation-robustness findings in §5.5 and Table 16.
  • Verification oracle downgrades = 10 specs across cfd, hotspot, myocyte, bfs, nw, nn
    Specs downgraded from strong/medium to weak stdout+exit-code oracles because of FP reduction-order divergence or reference asymmetry (Section 6). These choices set the verification standard for the reported pass rates.
  • Augmentation intensity fractions = L2: f=0.33; L3: f=0.66; L4: f=1.0 of transforms and candidate sites
    Hand-chosen density levels for L2–L4 (Appendix E.1, Table 8); no sensitivity analysis over alternative densities is reported.
  • KNOWN_FAIL exclusion set = 9 specs (7 Rodinia, 2 HeCBench)
    Exclusions for pre-existing toolchain failures are documented per-spec (Appendices D.5, B.4), but the exclusion boundary is hand-adjudicated and changes denominators (Qwen: 82 of 708 records removed).
assumptions (6)
  • domain assumption Declared oracles are sufficient to detect material translation errors
    Appendix H.5, assumption 2, labeled by the authors 'the weakest assumption.' For 80/87 specs the oracle is stdout-pattern + exit-code; wrong numerical output with correct banner passes. This is the load-bearing premise behind every pass-rate figure.
  • domain assumption AST augmentation transforms are behavior-preserving
    Appendix H.5, assumption 8. Baseline validation confirms for 80/87 specs at L1–L4, but 7 omp_target variants fail their own augmented baseline at L3–L4; the affected variants are excluded from robustness claims. The exclusion is post hoc but documented.
  • domain assumption Source implementations are baseline-valid on the reference platform
    Appendix H.5, assumption 1. All 87 non-KNOWN_FAIL specs pass baseline build-run-verify; a latent source bug would be faithfully propagated to 'passing' translations.
  • domain assumption Kernel-centric isolation measures translation capability
    Appendix H.5, assumption 5. Fixing host code and build infrastructure removes repository-reconstruction effects by design, so results cannot speak to multi-component integration competence.
  • domain assumption Single-platform evaluation (RTX 4070, HPC SDK 24.3) is representative
    Appendix H.5, assumption 3. Compiler/runtime behavior differs across GPU architectures and toolchains; the paper defers cross-platform generality to future work.
  • domain assumption Three samples per task characterize stochastic variance
    Appendix H.5, assumption 4. The pass@1-to-pass@3 gaps (11.3/7.0/5.6 pp) are reported, and the paper concedes more samples would tighten estimates.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ParBench: A Benchmark for Reliable Evaluation of LLM Parallel Code Translation." pith.science (2026). https://pith.science/paper/7ZOZRNWC

@misc{pith2026260722588,
  author       = {Pith},
  title        = {Pith review of: ParBench: A Benchmark for Reliable Evaluation of LLM Parallel Code Translation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7ZOZRNWC}},
  note         = {Machine review of arXiv:2607.22588}
}
read the original abstract

Modern compute-intensive software must migrate across a changing ecosystem of accelerators, programming APIs, compiler stacks, and portability layers, including CUDA, OpenMP, OpenCL, and OpenMP target offload. Large language models and autonomous coding agents are increasingly proposed for such migration, but the field lacks reliable ways to measure whether they preserve the low-level parallel semantics that make translations behaviorally valid, including thread indexing, synchronization, memory management, host-device coordination, and API-specific execution structure. We present ParBench, a kernel-centric benchmark framework for evaluating LLM-based parallel API translation under executable, reproducible conditions. ParBench fixes the surrounding build, run, and verification infrastructure through declarative benchmark specifications and asks models to translate only the computational kernels. It draws on multiple open-source HPC suites and covers representative cross-API translation directions among CUDA, OpenMP, OpenCL, and OpenMP target offload. To test whether success reflects robust translation rather than surface-form memorization, ParBench includes AST-driven, intended behavior-preserving, baseline-validated source augmentation. Evaluations on state-of-the-art open and proprietary LLMs show persistent barriers to reliable parallel code translation, including direction asymmetry, multi-file coordination, incomplete API adaptation, and uneven robustness to source-level perturbations. Code is available at https://github.com/Scientific-Computing-Lab/ParBench.

Figures

Figures reproduced from arXiv: 2607.22588 by the authors.

Figure 1
Figure 1. PARBENCH isolates parallel API translation from repository-level confounds. The benchmark pipeline starts by selecting representative parallel APIs and curating executable kernel specifications from multi-API HPC suites. Each specification fixes the build, run, and verification context, while the model is asked to translate only the computational kernel code. AST-driven source augmentations create surface-varied sou… view at source ↗
Figure 2
Figure 2. Per-kernel translation status across all directions and three models (L0, first sample per task, [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Failure taxonomy by translation direction across all three models (L0, first sample per task). [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (20 more)
Figure 4
Figure 4. Figure 4: API co-occurrence heatmap across all 71 surveyed repositories. Cell values indicate the [PITH_FULL_IMAGE:figures/full_fig_p024_4.png]
Figure 5
Figure 5. Figure 5: API coverage across the benchmark survey. OpenMP (50 benchmarks), MPI (45), and [PITH_FULL_IMAGE:figures/full_fig_p025_5.png]
Figure 6
Figure 6. Figure 6: Kernel-level API co-occurrence matrix. Each cell counts the number of individual kernels [PITH_FULL_IMAGE:figures/full_fig_p026_6.png]
Figure 7
Figure 7. Figure 7: Kernel-level API co-occurrence network. Edges shown for [PITH_FULL_IMAGE:figures/full_fig_p027_7.png]
Figure 8
Figure 8. Figure 8: Repository-level vs. kernel-level translation pair counts. Left bars show the number of [PITH_FULL_IMAGE:figures/full_fig_p027_8.png]
Figure 9
Figure 9. Figure 9: Distribution of benchmark types across the 35 repositories that met inclusion criteria. Suites [PITH_FULL_IMAGE:figures/full_fig_p030_9.png]
Figure 10
Figure 10. Figure 10: Distribution of API breadth across all 71 surveyed repositories. The mode is 3 APIs [PITH_FULL_IMAGE:figures/full_fig_p031_10.png]
Figure 11
Figure 11. Figure 11: Benchmark kernel richness (y-axis, capped at 100 for readability) vs. API breadth (x-axis, [PITH_FULL_IMAGE:figures/full_fig_p032_11.png]
Figure 12
Figure 12. Figure 12: Top multi-API repository candidates ranked by kernel count (blue bars, scaled for [PITH_FULL_IMAGE:figures/full_fig_p032_12.png]
Figure 13
Figure 13. Figure 13: HeCBench kernel selection funnel. Starting from 522 kernels spanning CUDA, HIP, [PITH_FULL_IMAGE:figures/full_fig_p033_13.png]
Figure 14
Figure 14. Figure 14: Benchmarks grouped by verification method. Large nodes represent verification ap [PITH_FULL_IMAGE:figures/full_fig_p035_14.png]
Figure 15
Figure 15. Figure 15: Per-kernel transform application frequency across the Rodinia corpus. Rows represent [PITH_FULL_IMAGE:figures/full_fig_p037_15.png]
Figure 16
Figure 16. Figure 16: L0-conditional augmentation outcomes across all three models for the CUDA-to-OpenMP [PITH_FULL_IMAGE:figures/full_fig_p042_16.png]
Figure 17
Figure 17. Figure 17: Per-kernel L0/L1–L4 outcomes for the CUDA-to-OpenMP subset under Qwen 3.5. Cell [PITH_FULL_IMAGE:figures/full_fig_p043_17.png]
Figure 18
Figure 18. Figure 18: Pass@k by translation direction (Qwen 3.5, L0, three samples per task). CUDA–OMP OMP–CUDA CUDA–OCL OCL–CUDA OMP–OCL OCL–OMP CUDA–OMP-T OMP-T–CUDA 0 20 40 60 80 100 83.3 55.6 59.6 19.3 72.5 41.2 95.8 95.8 91.7 62.5 68.4 26.3 82.4 47.1 100 100 Average pass@k (%) pass@k …
Figure 19
Figure 19. Figure 19: Pass@k by translation direction (GPT-5.4, L0, three samples per task). CUDA–OMP OMP–CUDA CUDA–OCL OCL–CUDA OMP–OCL OCL–OMP CUDA–OMP-T OMP-T–CUDA 0 20 40 60 80 100 76.4 55.6 57.9 19.3 82.4 39.2 100 100 83.3 62.5 63.2 26.3 82.4 52.9 100 100 Average pass@k (%) pass@k by …
Figure 20
Figure 20. Figure 20: Pass@k by translation direction (GPT-5.3-codex, L0, three samples per task). 53 [PITH_FULL_IMAGE:figures/full_fig_p053_20.png]
Figure 21
Figure 21. Figure 21: Cross-suite pass rate comparison (L0, Qwen 3.5). Per-suite aggregate pass rates with [PITH_FULL_IMAGE:figures/full_fig_p054_21.png]
Figure 22
Figure 22. Figure 22: Cross-suite pass rate comparison (L0, GPT-5.4). Per-suite aggregate pass rates with [PITH_FULL_IMAGE:figures/full_fig_p054_22.png]
Figure 23
Figure 23. Figure 23: Cross-suite pass rate comparison (L0, GPT-5.3-codex). Per-suite aggregate pass rates with [PITH_FULL_IMAGE:figures/full_fig_p054_23.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

70 extracted references · 1 linked inside Pith

  1. [1]

    Tal Kadosh, Niranjan Hasabnis, Prema Soundararajan, Vy A

    URLhttps://doi.org/10.1007/978-3-031-40744-4_1. Tal Kadosh, Niranjan Hasabnis, Prema Soundararajan, Vy A. V o, Mihai Capota, Nesreen Ahmed, Yuval Pinter, and Gal Oren. Ompar: Automatic parallelization with ai-driven source-to-source compilation, 2024a. URLhttps://arxiv.org/abs/2409.14771. Tal Kadosh, Niranjan Hasabnis, Vy A. V o, Nadav Schneider, Neva Kri...

    arXiv 2024

  2. [2]

    Extension-based match.If explicit filenames are absent, the parser matches code fences by file extension (e.g.,.cu,.cl,.cpp) against the expected target file types

  3. [3]

    Fuzzy match.Partial filename matches and common naming variants are attempted (e.g., matchingkernel.cuto an expectedtranslated_0.cu)

  4. [4]

    If any expected target file cannot be recovered or the extracted content is empty after all tiers, the task is classified as EXTRACT_FAIL

    Elimination-based assignment.When multiple code blocks remain unmatched, they are assigned to the remaining expected files by elimination order. If any expected target file cannot be recovered or the extracted content is empty after all tiers, the task is classified as EXTRACT_FAIL. Across the 2,262 evaluation-eligible records, extraction failures are rar...

    2021

  5. [5]

    Elias Konstantinidis and Yiannis Cotronis

    URLhttps://arxiv.org/abs/2506.11153. Elias Konstantinidis and Yiannis Cotronis. A quantitative roofline model for gpu kernel perfor- mance estimation using micro-benchmarks and hardware metric profiling.Journal of Paral- lel and Distributed Computing, 107:37–56, 2017. ISSN 0743-7315. doi: https://doi.org/10. 1016/j.jpdc.2017.04.002. URL https://www.scienc...

    arXiv 2017

  6. [6]

    Code files are included up to a cumulative 50,000-character limit

    Support / Header Files(optional) – source-directory headers and code files, genericized as Header FileN and Code FileN , with an instruction to inline definitions rather than emit unresolvable #include directives. Code files are included up to a cumulative 50,000-character limit

  7. [7]

    This is the highest-confidence tier

    Explicit filename match.The parser searches for markdown code fences annotated with the expected target filename (e.g., “‘cpp filename=translated_0.cpp). This is the highest-confidence tier

  8. [8]

    H.3 Valid Claims The following conclusions can be drawn from PARBENCHresults when accompanied by the stated conditions:

    Training-data memorization beyond surface form (definitively).Augmentation tests surface-form robustness, not algorithmic memorization (Section H.4). H.3 Valid Claims The following conclusions can be drawn from PARBENCHresults when accompanied by the stated conditions:

Show all 70 references
  1. [9]

    the model understands parallel programming

    Prompt anonymization does not change task difficulty.Stripping comments, generi- cizing filenames, and removing kernel names reduce memorization cues but might also remove helpful context, potentially making PARBENCHresults conservative relative to non-anonymized use. H.6 User...

  2. [11]

    While repository- level co-occurrence with CUDA and OpenMP is high, the kernel-level material is insufficient for the multi-direction evaluation design that PARBENCHrequires

    Low kernel-level coverage: At the kernel level, only 22 kernels provide OpenACC im- plementations across the surveyed repositories (Figure 6)–fewer than OpenCL (27) and substantially fewer than the primary APIs (CUDA: 656, OpenMP: 472). While repository- level co-occurrence wi...

  3. [12]

    Including both would provide diminishing returns in programming-model diversity

    Paradigm overlap with OpenMP target: OpenACC’s directive-based model (#pragma acc parallel loop) occupies a similar conceptual niche to OpenMP’s target offload directives (#pragma omp target teams distribute). Including both would provide diminishing returns in programming-mod...

  4. [13]

    Compiler availability: OpenACC compilation requires the NVIDIA HPC SDK (nvc/nvc++) or GCC with -fopenacc. This is a less universally available toolchain than CUDA (nvcc) or OpenMP (any modern C/C++ compiler), introducing a confounding variable between compiler availability and...

  5. [14]

    Default GCC and Clang installations support only CPU-threaded OpenMP; GPU offloading requires custom builds with target-offload support enabled

    Compiler requirement: OpenMP target compilation for NVIDIA GPUs requires the NVIDIA HPC compiler (nvc/nvc++, part of the NVIDIA HPC SDK 24.3). Default GCC and Clang installations support only CPU-threaded OpenMP; GPU offloading requires custom builds with target-offload suppor...

  6. [15]

    Within PARBENCH’s curated corpus, OpenMP target implementations are available for 12 kernels: 10 from HeCBench, 1 from XSBench, and 1 from RSBench

    Limited benchmark coverage: The kernel-level survey identifies 106 kernels with both OpenMP target and CUDA implementations (Figure 6), concentrated entirely in RAJAPerf. Within PARBENCH’s curated corpus, OpenMP target implementations are available for 12 kernels: 10 from HeCB...

  7. [16]

    available parallel code

    Compilation model difference: OpenMP target generates GPU offload code through the compiler, while CPU OpenMP generates threaded CPU code. A translation from CUDA to OpenMP target preserves GPU execution semantics but changes the syntax entirely; a translation from CUDA to CPU...

    2009

  8. [17]

    This is the dominant build-time failure mode for CUDA-targeting directions across all three models

    Phantom dependencies: The model hallucinates include files (timing.h,./main.h) or helper functions (checkCUDAError, setdevice) that may exist in the model’s training data but are absent from the target program’s file structure. This is the dominant build-time failure mode for ...

  9. [18]

    Cross-file identifier inconsistency: In some translations (observed in GPT-5.3-codex and GPT-5.4 OpenCL →CUDA attempts), the kernel file uses the target API’s naming convention while the wrapper file retains the source API’s function name, producing linker errors (undefined re...

  10. [19]

    CUDA”, “OpenMP

    Runtime kernel compilation failure: For OpenCL-targeting directions, the host C/C++ code compile successfully, but the OpenCL kernel source, compiled at runtime by clBuildProgram, contains errors such as incorrect __local variable scoping and un- declared type aliases. This re...

  11. [20]

    The kernel name and benchmark description arenotincluded (anonymization)

    Translation Task– source and target API display names. The kernel name and benchmark description arenotincluded (anonymization)

  12. [21]

    When target infrastructure context is provided, an explanatory note clarifies that only these files replace existing project files

    Target Files to Produce– list of genericized target filenames (e.g., translated_0.cpp, translated_1.cl). When target infrastructure context is provided, an explanatory note clarifies that only these files replace existing project files

  13. [22]

    Build Command– the anonymized compilation command in a code fence, so the LLM can ensure API and flag compatibility

  14. [23]

    CUDA Toolkit ≥ 11.0

    Build Environment– system dependencies from the target spec (e.g., “CUDA Toolkit ≥ 11.0”, “GCC≥9.0”)

  15. [24]

    Source Code (CUDA)

    Source Code– each source file presented as a numbered subsection ( Source File 1, Source File 2, . . . ) with all C/C++ comments stripped. The section heading includes the source API name (e.g., “Source Code (CUDA)”)

  16. [26]

    DO NOT MODIFY – for reference only

    Target Infrastructure Context(optional) – non-kernel target files (prompt payload entries not intranslation_targets, plus target support headers) provided as read-only reference so the LLM can match expected function signatures and data structures. Headed “DO NOT MODIFY – for ...

  17. [27]

    Kernel identity omission.The kernel name and benchmark description are excluded from the prompt entirely. 45

  18. [28]

    The stripper uses a state-machine parser that preserves string, character, and raw string literals

    Comment stripping.All C/C++ line ( //) and block (/* */) comments are removed from every file shown to the LLM–source files, support files, and target infrastructure files alike. The stripper uses a state-machine parser that preserves string, character, and raw string literals

  19. [29]

    Support files are labeled Header FileN or Code FileN by extension

    Source and support filename genericization.Source files are labeled Source File 1, Source File 2, etc. Support files are labeled Header FileN or Code FileN by extension. Target infrastructure files are labeledInfrastructure FileN

  20. [30]

    An internal mapping restores original filenames when writing LLM output to disk, including any cross-file#includereferences the LLM emits using the generic names

    Target filename genericization.Target output filenames are replaced with translated_0.ext, translated_1.ext, etc., preserving original file extensions. An internal mapping restores original filenames when writing LLM output to disk, including any cross-file#includereferences t...

  21. [31]

    Build command anonymization.Kernel-specific identifiers are removed from the build command: maketarget is reduced tomake (relying on the Makefile default target), and kernel names in other command strings are replaced with a generic placeholder

  22. [32]

    CUDA”, “OpenMP

    API-name retention.The source and target API names (e.g., “CUDA”, “OpenMP”) are intentionallynotanonymized, as they are essential to the translation task specification. G Evaluation Cost and Reproducibility This appendix groups the practical information needed to reproduce the...

    2026

  23. [33]

    Binary declared-oracle pass/fail of kernel-level translation.The harness evaluates whether LLM-translated code compiles, executes, and produces output matching the spec’s verification strategies. Verification applies a conjunction of strategies: all declared checks 47 (exit co...

  24. [34]

    The evaluation matrix covers 142 unique source–target pairs (Section 2.3)

    Translation capability across three primary APIs and ten directions.CUDA, OpenMP, and OpenCL form six bidirectional standard directions; OpenMP Target adds four case-study directions. The evaluation matrix covers 142 unique source–target pairs (Section 2.3)

  25. [35]

    Failure mode taxonomy.Four failure classifications (EXTRACT_FAIL, BUILD_FAIL, RUN_FAIL, VERIFY_FAIL) enable diagnostic analysis ofwherein the pipeline translations fail. Build failures indicate incomplete API-surface adaptation; run failures indicate runtime errors or timeouts...

  26. [36]

    The augmentation campaign applies only to direction–kernel pairs that qualify under the L0-conditional filter

    Surface-form robustness on the L0-conditional subset.Six AST-level transforms (Swap- Condition, ArithmeticTransform, ChangeNames, TypedefExpansion, PointerArithmetic- ToArrayIndex, ChangeFunctionNames) at four intensity levels (L1–L4) test whether pass/fail status is stable un...

  27. [37]

    CUDA-to-OpenMP consistently passes at higher rates than OpenCL-to-CUDA across all three evaluated models (Section 5.3)

    Direction asymmetry.Pass-rate variation across translation directions quantifies structural difficulty differences between API pairs. CUDA-to-OpenMP consistently passes at higher rates than OpenCL-to-CUDA across all three evaluated models (Section 5.3)

  28. [38]

    Cross-model discrimination.The benchmark distinguishes models that differ in translation capability, subject to sampling-condition caveats (Section H.5). In the current evaluation, task-level pairwise analysis on 142 shared L0 tasks yields Cohen’s h≈0.71 between Qwen 3.5 and b...

  29. [39]

    H.2 What PARBENCHDoes Not Measure The following properties are explicitly outside PARBENCH’s measurement scope:

    Per-kernel difficulty heterogeneity.For Qwen 3.5, canonical pass rates range from 72.2% (floydwarshall, iso2dfd) to 0% (10 kernels), enabling fine-grained analysis of which computa- tional patterns–stencils, graph traversal, ODE integration, pointer-heavy data structures–are h...

  30. [40]

    Wall-clock time is captured but is un- reliable for sub-millisecond baselines

    Performance and resource efficiency.No speedup, throughput, occupancy, bandwidth, memory, or energy measurement is performed. Wall-clock time is captured but is un- reliable for sub-millisecond baselines. No kernel-level profiling ( ncu/nsys for CUDA, omp_get_wtime for OpenMP)...

    2026

  31. [41]

    A harness-passing but unreadable translation receives PASS

    Code quality, readability, or maintainability.Translated code is stored in result files but no AST analysis, cyclomatic complexity, or style metrics are computed. A harness-passing but unreadable translation receives PASS

  32. [42]

    There is no partial-credit classifica- tion

    Partial pass credit or oracle-strength gradations.Verification is binary: a numerical result that deviates from the reference by 0.02% beyond the configured tolerance receives VERIFY_FAIL, identical to a completely wrong output. There is no partial-credit classifica- tion

  33. [43]

    Numerical accuracy for most specs.Only 7 of 87 non-KNOWN_FAILspecs (8%) have medium or strong oracles ( numeric_comparison or file_hash). The remaining 80 specs (92%) rely on exit_code and stdout_pattern, which verify that the program runs and prints expected banners but do no...

  34. [44]

    The evaluation pipeline implements iterative repair (multi-turn error feedback with linker analysis via --max-retries), but all reported results use max_retries=1 (zero-shot)

    Iterative self-repair or agentic translation.Each sample is a single LLM call with no feedback loop. The evaluation pipeline implements iterative repair (multi-turn error feedback with linker analysis via --max-retries), but all reported results use max_retries=1 (zero-shot). ...

  35. [45]

    No testing on other GPU architectures (A100, H100), CPU-only environments, or other OS configurations

    Cross-platform portability.All evaluations run on a single platform: NVIDIA RTX 4070 (sm_89), AMD Ryzen 9 7900X, Ubuntu 24.04, HPC SDK 24.3 (Appendix Table 19). No testing on other GPU architectures (A100, H100), CPU-only environments, or other OS configurations

  36. [46]

    Host code, Makefiles, and I/O routines remain fixed

    Repository-level translation.By design, PARBENCHisolates kernel translation from build-system reconstruction. Host code, Makefiles, and I/O routines remain fixed. The benchmark does not measure the LLM’s ability to restructure project files, generate build systems, or coordina...

    2025

  37. [48]

    Model X achieves Y% pass@k on kernel-centric parallel code translation across Z directions

    “Model X achieves Y% pass@k on kernel-centric parallel code translation across Z directions.”Valid when: the exact pass@ k metric is specified (pass@1 vs. pass@3), aug- mentation levels are stated (L0 only vs. L0–L4), directions are enumerated, and confidence intervals are included

  38. [49]

    Build-stage failure is the dominant failure mode for Model X

    “Build-stage failure is the dominant failure mode for Model X.”Valid when: the failure taxonomy breakdown is reported with counts and percentages from the full record set

  39. [50]

    Direction A→B is harder/easier than Direction C→D for Model X

    “Direction A→B is harder/easier than Direction C→D for Model X.”Valid when: backed by per-direction pass rates with Wilson confidence intervals. Claims of statistical signifi- cance should use McNemar’s test on paired kernels with appropriate multiple-comparison correction

  40. [51]

    Model X maintains declared-oracle pass rates across L1–L4 on the L0-conditional subset

    “Model X maintains declared-oracle pass rates across L1–L4 on the L0-conditional subset.”Valid when: the L0-conditional filter is disclosed, the qualifying subset size is stated, the direction restriction is noted, and the analysis is presented as descriptive rather than as a ...

  41. [52]

    Model X discriminates from Model Y on PARBENCH

    “Model X discriminates from Model Y on PARBENCH.”Valid when: (a) all compared models are evaluated on the same task set, (b) McNemar’s paired test is reported with effect size and concordance table, (c) sampling-configuration differences are explicitly disclosed if temperature...

  42. [53]

    Kernel K is harder to translate than Kernel J

    “Kernel K is harder to translate than Kernel J.”Valid when: both kernels have sufficient sample sizes (n≥18recommended) and confidence intervals are reported. H.4 Invalid Claims The following conclusionscannotbe drawn from PARBENCHresults:

  43. [54]

    Model X produces efficient/fast parallel code

    “Model X produces efficient/fast parallel code.”Performance is not measured. A PASS means the code compiles, runs, and satisfies the declared oracle–not that it runs at acceptable speed

  44. [55]

    Model X’s translations are production-ready

    “Model X’s translations are production-ready.”No code review, security audit, or race- condition detection is performed. Parallel code can have latent data races that produce correct output on some executions. 49

  45. [56]

    Model X understands parallel programming

    “Model X understands parallel programming.”PARBENCHmeasures behavioral out- comes, not internal representations. A model could pass by pattern matching without “understanding” synchronization semantics

  46. [57]

    PARBENCHpass rate generalizes to all HPC translation tasks

    “PARBENCHpass rate generalizes to all HPC translation tasks.”The corpus contains 87 non-KNOWN_FAILspecs from five suites, with Rodinia contributing 53 (61%). Domains such as distributed-memory MPI, GPU tensor operations, and FPGA HLS are outside its scope

  47. [58]

    Model X is definitively better than Model Y at parallel translation

    “Model X is definitively better than Model Y at parallel translation.”Only valid under matched sampling conditions. If temperatures, reasoning modes, or provider-side controls differ (as they do between Qwen 3.5 and the Azure models in this paper), the observed gap reflects an...

  48. [59]

    Surface-form robustness proves the model is not memorizing

    “Surface-form robustness proves the model is not memorizing.”Surface-form aug- mentation (variable renaming, syntax sugar) tests robustness to cosmetic code changes. Algorithmic memorization–where the model recognizes the computation and produces a previously seen translation ...

  49. [60]

    A PASSmeans the translation is numerically faithful

    “A PASSmeans the translation is numerically faithful.”For the 80 of 87 specs with weak oracles, verification checks that the program ran and printed expected output banners. Numerical results are not independently validated. Only 7 specs have medium or strong oracles that veri...

  50. [61]

    OpenCL→CUDA is impossible for LLMs

    “OpenCL→CUDA is impossible for LLMs.”Qwen 3.5 achieves 0% but both GPT-5.4 and GPT-5.3-codex achieve 19.3% on the same direction. Zero-rate results for a single model reflect that model’s capability, not an inherent impossibility. H.5 Assumptions PARBENCH’s evaluation results ...

  51. [62]

    If a source implemen- tation has a latent bug, faithful translations of that bug would pass the declared oracle

    Source implementations are baseline-valid on the reference platform.PARBENCHtreats the original benchmark implementations as the reference baseline. If a source implemen- tation has a latent bug, faithful translations of that bug would pass the declared oracle. Mitigation:all ...

  52. [63]

    Declared oracles are sufficient to detect material translations.This is the weakest assumption. For the 80 specs with weak oracles, a translation that produces wrong numerical results but expected program flow (runs to completion, prints expected banners) would receive a false...

  53. [64]

    Different GPU architectures may have different compiler behavior, runtime characteristics, or memory limits

    The single evaluation platform generalizes to other NVIDIA GPU configurations.All results are from one machine (RTX 4070, sm_89). Different GPU architectures may have different compiler behavior, runtime characteristics, or memory limits

  54. [65]

    The pass@1-to-pass@3 gap (11.3 pp for Qwen 3.5, 7.0 pp for GPT-5.4, 5.6 pp for GPT-5.3-codex) suggests moderate within-task variance

    Three samples per task capture meaningful stochastic variance under each provider’s sampling regime.For Qwen 3.5 this uses temperature 0.7; for GPT-5.4 and GPT-5.3-codex sampling is provider-controlled (Section B.4). The pass@1-to-pass@3 gap (11.3 pp for Qwen 3.5, 7.0 pp for G...

  55. [66]

    ParEval-Repo’s 0% at>133 SLoC Davis et al

    Kernel-centric translation isolates parallel programming capability.By fixing host code and build infrastructure, PARBENCHprevents build-system reconstruction from dominating results (cf. ParEval-Repo’s 0% at>133 SLoC Davis et al. [2025]). However, this also means that a model...

    2025

  56. [67]

    The 9 KNOWN_FAILexclusions are legitimately infrastructure failures.Each exclusion is documented with a specific technical cause (CUDA 12 API deprecation, missing libraries, pre-existing build or runtime errors). 50

  57. [68]

    [2009], HeCBench: 2023 Jin and Vetter [2023], XSBench: ANL Tramm et al

    Benchmark suites are representative of real HPC workloads.The five suites are well- established in HPC research (Rodinia: IISWC 2009 Che et al. [2009], HeCBench: 2023 Jin and Vetter [2023], XSBench: ANL Tramm et al. [2014]). However, they over-represent structured stencil and ...

    2009

  58. [69]

    80 of 87 specs pass all L1–L4 augmented baseline verification

    Augmentation transforms are intended to preserve source behavior.Each transform is backed bylibclang AST analysis and validated by 15 unit tests. 80 of 87 specs pass all L1–L4 augmented baseline verification. The 7 failures are omp_target-specific: condition operand swapping (...

  59. [2015]

    ide nt it y

    Springer International Publishing. ISBN 978-3-319-15976-8. Pedro Valero-Lara, Aaron Young, Thomas Naughton III, Christian Engelmann, Al Geist, Jeffrey S. Vetter, Keita Teranishi, and William F. Godoy. Chatmpi: Llm-driven mpi code generation for hpc workloads. InProceedings of ...

    2026

  60. [2024]

    ISBN 978-3-031-69576-6

    Springer-Verlag. ISBN 978-3-031-69576-6. doi: 10.1007/978-3-031-69577-3_9. URL https://doi.org/10.1007/978-3-031-69577-3_9. Le Chen, Nesreen Ahmed, Mihai Capot˘a, Ted Willke, Niranjan Hasabnis, and Ali Jannesari. Pcebench: A multi-dimensional benchmark for evaluating large lan...

    2025

  61. [2025]

    ISBN 9798400720741

    Association for Computing Machinery. ISBN 9798400720741. doi: 10.1145/3754598. 3754669. URLhttps://doi.org/10.1145/3754598.3754669. Matthew T. Dearing, Yiheng Tao, Xingfu Wu, Zhiling Lan, and Valerie Taylor. Lassi: An llm- based automated self-correcting pipeline for translati...

    2024

  62. [2026]

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan

    URLhttps://arxiv.org/abs/2510.00031. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? InThe Twelfth International Conference on Learning Representations,...

    2024

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.