{"id":"6f06878a-a28b-430c-983f-9c5df51c4e07","arxiv_id":"2506.20938","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A benchmark and evaluation showing that LLM-based repository-scale HPC translation works for small programs but fails for larger ones due to build system and cross-file issues.","lead":"ParEval-Repo is a new benchmark for testing whether large language models can translate entire HPC codebases between GPU programming models. Initial results show small programs can be translated, but build systems and cross-file dependencies fail as code grows.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The universal zero-result for larger applications rests on unreported per-cell sample counts; without N or confidence intervals, pass@1=0 cannot be distinguished from a small positive rate missed by a small sample.","rationale":"The paper is a well-constructed benchmark, and the headline finding is consistent with the displayed heatmaps: for every cell that was run for SimpleMOC-kernel, XSBench, and llm.c, pass@1 is 0, and the same holds for code-only scores that use author-provided build systems. This means the failure is in the generated source itself, not just the build system. The inclusion of XSBench as a deliberate contamination control is a good design, and its zero results support the authors' interpretation that public ports do not rescue the models.\n\nThe load-bearing weakness is statistical: the paper defines pass@k with an explicit sample count N (Eq. 1) but never states N, k, or any confidence interval for the zero estimates. The universal negative in Section 8.2 is therefore not quantitatively bounded. If the per-cell N is small, true success rates of a few percent could be missed, which would not overturn the qualitative conclusion that translation is unreliable but would invalidate the categorical 'no combination achieves a pass@k above 0' phrasing. The reader's anti-contamination concern is less load-bearing because data leakage can only increase scores; the positive results are on custom micro-applications, so they are not plausible memorization artifacts, and the negative results cannot be explained by contamination.\n\nSince this weakness is addressable by reporting N and intervals, and since the underlying benchmark is sound, the appropriate disposition remains CONDITIONAL. My independent stress-test therefore does not move the reader's verdict.","tokens_in":18314,"tokens_out":12580,"duration_ms":136671,"concrete_test":"Inspect the ParEval-Repo GitHub repository or experiment scripts to recover the exact N per (task, LLM, technique) cell. For every pass@1=0 cell involving SimpleMOC-kernel, XSBench, and llm.c, compute the 95% Clopper-Pearson confidence interval and the joint probability of the observed zero pattern under a plausible per-attempt success rate, e.g., 0.05. If N is found to be at least 50 for all run cells, the universal zero is robust and the claim stands; if N is 10-25, the paper should be revised to report N and restate the zero result as a sample-bound finding rather than a categorical inability.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central negative result in Section 8.2 — 'no combination of translation technique and LLM achieves a pass@k above 0 for any application larger than microXOR' — is a universal zero. The paper defines pass@k in Eq. 1 with N samples per task, but never reports N, k, or per-cell confidence intervals anywhere in the text; Figure 2 shows only pass@1 and build@1 point estimates. For a single cell, an observed 0 with N samples has a 95% upper bound of roughly 3/N (rule of three). The heatmap granularity (values in increments of 0.02 and 0.04) suggests N may be 25 or 50, but this is not stated. If N is 25, a true pass@1 as high as 0.12 is compatible with zero observed successes; if N is 10, the bound is over 0.25. Many cells for the three larger applications are empty due to context-window overflows, node-hour budgets, or API budget limits, so the support for the universal statement is the aggregate of the remaining cells, and that aggregate is not quantified. The anti-contamination assumption is less load-bearing for this claim: contamination can only inflate pass@k, the positive results occur only on custom micro-applications with no public translations, and the deliberately contaminated XSBench case also yields zero, so leakage is not a plausible driver of the negative finding. The missing sample counts, by contrast, directly bound how strongly the zero can be interpreted as 'LLM translation cannot yet produce working code.'","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"ParEval-Repo is a benchmark suite for evaluating LLM-based translation of entire HPC repositories across GPGPU programming models. The paper defines 16 translation tasks spanning six applications (nanoXOR, microXORh, microXOR, SimpleMOC-kernel, XSBench, llm.c) and three translation pairs (CUDA to OpenMP Offload, CUDA to Kokkos, OpenMP Threads to OpenMP Offload). It evaluates five LLMs under three techniques: a non-agentic file-by-file translator, a top-down agentic pipeline, and SWE-agent. Correctness is measured with pass@k and build@k, and the paper proposes an expected token cost metric E_kappa. The headline finding is that no combination of LLM and technique produces a correct translation for any application larger than microXOR, with build-system generation and cross-file consistency identified as the main obstacles. The paper also contributes a semi-automated error-clustering analysis of build and run failures.","tokens_in":18659,"tokens_out":9491,"duration_ms":97919,"significance":"If the headline negative result is statistically well supported, the paper makes a valuable and timely contribution: it provides a concrete, reproducible benchmark for repository-level HPC translation and suggests that current LLM-based approaches are not yet viable for non-trivial codebases. The benchmark design is thoughtful in several respects: tasks are selected to avoid public reference translations, correctness is checked with real test cases, and code-only versus overall scoring separates source-translation ability from build-system generation. The public release of the benchmark is a strength, and the E_kappa metric is a simple but useful cost-aware comparison tool. The error taxonomy, while secondary, gives actionable insight into recurring failures. However, the strength of the central zero-result claim currently exceeds the statistical evidence: the number of samples per task is never reported, many heatmap cells are empty, and the error classification relies on a manually adjusted clustering step.","major_comments":[{"comment":"The paper defines pass@k in Eq. (1) in terms of N and c_t but never reports N, k, or per-cell confidence intervals anywhere in the text or figures. This is load-bearing for the headline result in §8.2: 'no combination of translation technique and LLM achieves a pass@k above 0 for any application larger than microXOR.' An observed zero with N samples has an approximate 95% upper bound of about 3/N by the rule of three; if N=25, true pass@1 values up to 0.12 are compatible with the reported zeros, and if N=10 the bound exceeds 0.25. The heatmap granularity in Fig. 2 (values in increments such as 0.02, 0.04, and 0.76) suggests N may be 25 or 50, but this is not stated. Please report per-cell N (or raw counts c_t and b_t), state k explicitly, and provide confidence intervals for the zero cells and for nonzero point estimates, such as the 0.76 versus 0.2 comparison for Llama-3.3 on microXORh versus nanoXOR in the CUDA-to-OpenMP code-only panel. Without this information, the universal zero cannot be distinguished from a small positive rate missed by a small sample.","section":"§2.2, Eq. (1); §8.2, Fig. 2"},{"comment":"The universal negative statement overstates the coverage of the completed experiments. The paper correctly distinguishes empty cells ('we do not run that case') from zero cells, but many cells for SimpleMOC-kernel, XSBench, and llm.c are empty because of context-window overflows, node-hour budgets, or API budgets. Examples stated in §8.2 include Gemini and GPT-4o for llm.c, Gemini for XSBench (CUDA to OpenMP Offload), QwQ for XSBench and llm.c across all translation pairs, Llama-3.3 for XSBench and llm.c (CUDA to Kokkos), and SWE-agent for XSBench and llm.c. The sentence 'no combination of translation technique and LLM achieves a pass@k above 0 for any application larger than microXOR' should therefore be scoped to the completed configurations, with the number of completed cells stated and the unrun cells explicitly excluded. As written, it reads as a claim about the full Cartesian product of techniques, LLMs, applications, and translation pairs, which the data do not cover.","section":"§8.2; Fig. 2 and §7.2"},{"comment":"The error taxonomy that supports the conclusions about build-system and cross-file-dependency failures is produced by a semi-automated pipeline whose subjective components are not fully specified. The paper states that DBSCAN hyperparameters were tuned by manual inspection, that the algorithm produced many clusters, and that a manual pass merged clusters, reassigned samples, and assigned labels. The paper does not report the hyperparameter values, the number of clusters before and after merging, the corpus used to train the word2vec embeddings, or a release artifact containing the raw logs and the cluster-to-label mapping. Please provide these details, or make the raw build/run logs and the post-merge cluster assignments available, so that the counts in Fig. 3 and the qualitative findings in §8.3 can be independently audited. This issue is secondary to the pass@k claim, but the error analysis is a stated contribution of the paper.","section":"§6.3; Fig. 3"}],"minor_comments":[{"comment":"In Eq. (1), the summation index p in T is inconsistent with the per-task counts c_t; please use c_p or make the dependence on the summed task explicit.","section":"§2.2, Eq. (1)"},{"comment":"The overall caption of Fig. 2 says 'Correctness metrics for OpenMP Threads to OpenMP Offload tasks,' but subfigures (a)-(d) are CUDA to OpenMP Offload and CUDA to Kokkos; the caption should be corrected to reflect the three translation pairs.","section":"Fig. 2"},{"comment":"The abstract and the results text refer to pass@k, but the presented heatmaps show only pass@1 and build@1. Please clarify whether k=1 is the only reported value, and if larger k were computed, report them as well.","section":"Abstract; §8.2"},{"comment":"The claim that all but one task has no publicly available translation in the target programming model is difficult to verify for commercial training corpora; please state how this absence was checked (e.g., repository and web searches) and acknowledge the residual contamination risk.","section":"§5.1"},{"comment":"The phrase 'consistently significantly lower' is used without statistical testing; either add appropriate tests or use a non-statistical comparator such as 'consistently lower in the measured cells.'","section":"§8.2"},{"comment":"E_kappa is aggregated only over cases with pass@1 greater than 0; please state this limitation directly where the metric is introduced in §6.2 so that readers do not interpret empty cells in Fig. 5 as zero expected cost.","section":"§6.2 and §8.4"},{"comment":"Please fix the typo 'estiamtes' in §8.4 and add axis labels and a color-scale label to Fig. 3 for readability.","section":"§8.4; Fig. 3"}],"recommendation":"major_revision","confidential_remarks":"The missing sample counts are the main barrier to accepting the headline negative result. From the authors' logs, adding per-cell N, k, and confidence intervals should be a reporting fix rather than a request for new experiments, unless the logs reveal that N is too small to support the universal zero. The large number of empty cells is a consequence of budget limits; the authors should be transparent about coverage. The benchmark artifact itself is useful and the negative result is interesting if properly qualified."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the good news: this is the first benchmark I know of that targets repository-level translation between GPU programming models, and it includes build system generation, which prior work (RepoTransBench, LASSI) mostly ignores. The authors evaluate a reasonable spread of LLMs and three translation techniques, and they test actual compilation and execution rather than CodeBLEU similarity. The headline result—no LLM/technique combination gets a pass@k > 0 for any application larger than microXOR—is striking and, if even approximately true, useful for the community. The error clustering analysis is also valuable: it shows build systems and cross-file interfaces are the main bottlenecks, not kernel translation itself.\n\nThe main soft spot is the missing sample count. The paper defines pass@k with N samples per task but never reports N, k, or any confidence interval. A single cell showing 0 with N samples has an upper 95% bound of roughly 3/N; if N is 25—which the heatmap increments suggest—a true pass@1 as high as 0.12 is compatible with the observed zero. That directly weakens the universal zero claim in Section 8.2. The authors should report N per cell and, ideally, confidence intervals or at least a rule-of-three bound. This is an addressable revision, not a fatal flaw, because the qualitative pattern—larger applications fail—is robust across many cells and techniques.\n\nThe stress-test note also raises the anti-contamination assumption. I think that concern lands less well: contamination can only inflate pass@k, and the positive results are on custom micro-apps with no public translations. The deliberately contaminated XSBench case also yields zero, so leakage is not a plausible driver of the negative finding. So I'd not weight that heavily.\n\nOne more minor thing: the E_kappa metric is only computed where pass@1 > 0, so it doesn't apply to the large applications. That's fine, but it means the cost analysis is limited to the trivial cases.\n\nOverall, this is a solid empirical paper. The missing N is the one thing I'd want fixed before relying on the exact zero. The benchmark itself is a real contribution and likely to be used. I'd take it for peer review and would cite it in work on LLM-based HPC translation. It's a good reading-group paper too, mostly for the negative result and the error taxonomy.","headline":"Useful benchmark with a striking negative result, but the missing per-cell sample counts undercut the precision of the headline zero.","tokens_in":19163,"tokens_out":3021,"would_cite":true,"duration_ms":28924,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ParEval-Repo, a new benchmark, shows that current LLMs fail to translate complete HPC repositories beyond tiny micro-applications.","keywords":["LLM program translation","repository-level translation","GPGPU programming models","OpenMP offload","Kokkos","CUDA","benchmark suite","expected token cost"],"falsifier":"A single correct end-to-end translation of an application larger than microXOR—for example, XSBench from CUDA to Kokkos using an LLM-generated CMake build that passes XSBench's reference tests—would refute the paper's core negative result; alternatively, discovering a pre-cutoff public port of SimpleMOC-kernel or llm.c in the target programming model would undermine the contamination guard.","tokens_in":18139,"feed_emoji":"⚙️","tokens_out":5234,"duration_ms":54053,"temperature":0.7,"pith_summary":"ParEval-Repo is a benchmark for testing whether large language models can translate entire HPC software repositories from one GPU programming model to another (CUDA to OpenMP offload, CUDA to Kokkos, and OpenMP threads to OpenMP offload). The paper evaluates five LLMs with three strategies—file-by-file translation, a top-down multi-agent pipeline, and SWE-agent—across six applications ranging from roughly 100 to 3,000 lines. Its central finding is negative: no LLM and translation technique combination achieved a pass@k above 0 for any application larger than the tiny microXOR, and the main failure points were generated build systems and inconsistent interfaces across files. The benchmark matters because it isolates whether LLM translation can scale from single kernels to whole codebases, which is the real porting burden in HPC.","feed_headline":"No LLM can translate a full HPC repository yet","feed_subtitle":"Benchmark of five models and three methods: every attempt fails on codebases larger than microXOR.","key_machinery":"The load-bearing object is the ParEval-Repo task suite itself: six applications deliberately spanning size, file count, and dependency structure (nanoXOR, microXORh, microXOR, SimpleMOC-kernel, XSBench, llm.c) with the contamination guard that all but XSBench lack a public translation in the target programming model. Correctness is measured by build@1 and pass@1, extensions of the pass@k estimator from prior work: build@1 is the chance a single translation compiles, and pass@1 is the chance it compiles and passes the application's reference tests on an A100. The token-economy metric $E_\\kappa$ is the expected number of inference tokens needed for one correct translation, computed as $(1/\\mathrm{pass}@1)$ times the average tokens per generation, and it lets the authors compare cost across LLMs and techniques.","core_discovery":"The paper's claim is that repository-scale translation is not yet viable: across all tested LLMs (Gemini 1.5 Flash, GPT-4o mini, o4 mini, Llama 3.3 70B, QwQ-32B) and all three translation techniques, no configuration achieved a pass@k above 0 for any application larger than microXOR, and even for the micro-applications success was sporadic and model-dependent. Separating 'code-only' scores (using a hand-written ground-truth build system) from 'overall' scores (using the LLM-generated build system) shows that the gap is dominated by build-system generation: LLM-produced Makefiles and CMakeLists.txt fail with configuration errors, invalid compiler flags, and missing targets, while cross-file mismatches in function names and types account for many remaining failures. The paper also introduces $E_\\kappa$, the expected token cost of a correct translation, and reports that non-agentic o4-mini is the cheapest commercial option and non-agentic Llama-3.3 the cheapest open-source option among configurations that succeeded at all.","pith_inferences":["The code-only versus overall gap suggests a directed remedy the paper leaves implicit: generate the source translation with the LLM and synthesize the build system with a separate, build-system-specialized tool, which could push larger tasks from 0 to positive pass@1.","Because commercial LLM training data is opaque, the benchmark's contamination guard is a design guarantee, not an empirical one; a version of ParEval-Repo that creates fresh private repositories with unique identifiers would make memorization impossible to hide.","$E_\\kappa$ generalizes beyond HPC: any code-generation task billed per token could use expected-token-cost as the economic objective, especially in agentic settings where reasoning models emit large overhead.","The finding that SWE-agent cannot handle Makefiles suggests that agentic software-engineering tools tuned to Python ecosystems will need HPC-aware file handling before they can be applied to scientific codebases."],"forward_implications":["LLM-based automated porting is not yet a substitute for human effort on real HPC codebases; at best it works for single-file micro-applications.","Researchers should focus on build-system generation and cross-file interface repair rather than on kernel-level translation, since code-only scores are much higher than overall scores.","Benchmark users should report code-only and overall scores separately, because conflating kernel translation with build-system generation hides where the failure lies.","$E_\\kappa$ provides a practical cost target: future methods must reduce expected tokens to a correct translation, not just raise pass@1, to be economically useful.","The contamination-aware task selection sets a template for future translation benchmarks, with XSBench as a built-in control for measuring how much public ports help."],"supporting_citations":[{"why":"Supplies the pass@k estimator and the prior finding that LLMs can translate existing code but struggle with parallel code generation.","marker":"[18]"},{"why":"Defines the OpenMP programming model that is a target for two of the three translation pairs.","marker":"[20]"},{"why":"Supplies the SimpleMOC-kernel proxy application, a translation task with an external cuRAND dependency.","marker":"[23]"},{"why":"Supplies XSBench, the contamination-control application that has publicly available ports in the target programming models.","marker":"[24]"},{"why":"Defines the Kokkos programming model, the target library for the CUDA-to-Kokkos translation tasks.","marker":"[25]"},{"why":"Supplies the SWE-agent baseline, an agentic software-engineering tool evaluated for full-repository translation.","marker":"[29]"}],"fun_headline_variants":["No LLM cracks full HPC repo translation in new benchmark","LLMs fall short on repository-scale HPC translation tasks","Benchmark: every LLM fails on large HPC codebase translation","Build-system errors sink LLMs in HPC repo translation test"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's anti-contamination premise is that for every task except XSBench no public translation in the target programming model exists, so the LLM cannot simply recite memorized output; this cannot be verified for opaque commercial training corpora, and if a public port is actually in the training data, the reported scores would reflect memorization rather than translation ability.","fun_headline_variants_meta":{"raw":{"variants":["No LLM cracks full HPC repo translation in new benchmark","LLMs fall short on repository-scale HPC translation tasks","Benchmark: every LLM fails on large HPC codebase translation","Build-system errors sink LLMs in HPC repo translation test"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000616,"raw_usage":{"total_tokens":2889,"prompt_tokens":1003,"completion_tokens":1886,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":619,"completion_tokens_details":{"reasoning_tokens":1814}},"tokens_in":619,"tokens_out":1886,"duration_ms":14166,"temperature":1.0,"reasoning_tokens":1814,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:37:49.359995+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A single correct end-to-end translation of an application larger than microXOR—for example, XSBench from CUDA to Kokkos using an LLM-generated CMake build that passes XSBench's reference tests—would refute the paper's core negative result; alternatively, discovering a pre-cutoff public port of SimpleMOC-kernel or llm.c in the target programming model would undermine the contamination guard.","supporting_citations":[{"cited_title":"Davis, Zhaojun Xie, Arjun Rajaram, and Abhinav Bhatele","cited_arxiv_id":null,"evidence_quote":"Supplies the pass@k estimator and the prior finding that LLMs can translate existing code but struggle with parallel code generation."},{"cited_title":"OpenMP Application Program Interface","cited_arxiv_id":null,"evidence_quote":"Defines the OpenMP programming model that is a target for two of the three translation pairs."},{"cited_title":"Tramm, Geoffrey Gunow, Tim He, Kord S","cited_arxiv_id":null,"evidence_quote":"Supplies the SimpleMOC-kernel proxy application, a translation task with an external cuRAND dependency."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies XSBench, the contamination-control application that has publicly available ports in the target programming models."}],"review_version":1}