Pith. sign in

REVIEW 4 major objections 5 minor 7 cited by

MultiKernelBench: A Multi-Platform Benchmark for Kernel Generation

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MultiKernelBench is a 285-task benchmark spanning CUDA, AscendC, and Pallas, and its category-aware one-shot prompting measurably improves LLM kernel correctness on the less-covered platforms.

desk verdict A genuinely useful multi-platform kernel benchmark whose core Pass@k numbers rest on an unsupported N=5 correctness claim on AscendC and Pallas - worth engaging, but only after that claim is backed with data. read the letter →

arxiv 2507.17773 v2 pith:UXUR3YUK submitted 2025-07-20 cs.DC cs.LGcs.PFcs.SE

Zhongzhen Wen , Yinghui Zhang , Zhong Li , Zhongxin Liu , Linna Xie , Tian Zhang This is my paper · ORCID
classification cs.DCcs.LGcs.PFcs.SE
keywords deeplearningkernelskernelgenerationLLMcodemulti-platformbenchmarkCUDAAscendCPallasone-shotprompting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MultiKernelBench aims to make LLM-based deep-learning kernel generation evaluable across hardware vendors instead of only on Nvidia GPUs. It defines 285 kernel tasks in 14 functional categories, posed identically to LLMs for CUDA, AscendC, and Pallas, with a plugin-style backend that handles compilation and timing per platform. The paper's central empirical claim is that model performance is badly platform-dependent: Pass@1 on CUDA reaches 52.6% for the strongest model, while on AscendC it peaks at 2.5% and on Pallas at 8.4% when the one-shot example is a generic add kernel. Replacing that example with a kernel from the same category as the target raises pass rates substantially on AscendC and Pallas, in some categories from zero to over 20 points. A sympathetic reader would take away that cross-platform kernel generation is currently bottlenecked by training-data coverage, and that prompt-side exemplar selection is a cheap and effective partial remedy.

What carries the argument

The load-bearing mechanism is a uniform task frame plus a platform backend. Each task specifies a reference module and input tensors, so the same task can be posed to CUDA, AscendC, and Pallas; the backend interface isolates device setup, compilation, execution, correctness checking, and timing, so new platforms can be added by registering a subclass. The correctness criterion compares generated and reference outputs by $|\mathrm{output}_{\mathrm{llm}} - \mathrm{output}_{\mathrm{ref}}| < a_{\mathrm{tol}} + r_{\mathrm{tol}} \cdot |\mathrm{output}_{\mathrm{ref}}|$ with $N=5$ random tests. The experimental lever is category-aware one-shot selection: picking an exemplar kernel from the same category as the target, rather than always showing an add kernel, transfers category-specific conventions such as AscendC's separation of scalar, vector, and cube units, and that transfer is what drives the reported gains on under-represented platforms.

What would settle it

Run the 285-task suite on AscendC and Pallas with 100 random test inputs instead of $N=5$ under the same tolerances; if any kernel that passed the five-test check fails one of the hundred, the sufficiency claim in the correctness section is refuted and the published Pass@k rankings need re-baselining.

Watch

Extended reading notes

Core claim

The paper introduces MultiKernelBench and claims it is the first comprehensive multi-platform benchmark for LLM-generated deep-learning kernels. A task is a reference module plus input tensors; the model must emit platform-specific kernel code plus integration code, and the benchmark compiles, executes, and checks it against the reference. Correctness is judged by randomized testing with $N=5$ inputs and tolerances $a_{\mathrm{tol}}=r_{\mathrm{tol}}=10^{-2}$, and performance is reported through Compilation@k, Pass@k, and SpeedUp@k. On the default add-one-shot setting, the evaluated models solve only a minority of the 855 task-platform cases, and performance on AscendC and Pallas is much lower than on CUDA. The paper further shows that using a one-shot example drawn from the same functional category as the target kernel improves correctness on AscendC and Pallas by large relative margins, while revealing that task difficulty varies widely across its 14 categories.

Load-bearing premise

The load-bearing premise is that five random test inputs with $a_{\mathrm{tol}}$ and $r_{\mathrm{tol}}$ both $10^{-2}$ are enough to certify a kernel correct on every platform; the paper says it reproduced the $N=5$-versus-$N=100$ agreement on AscendC and Pallas but gives no supporting data for those platforms, so a failure of that premise would change the reported Pass@k numbers.

Editorial extensions

If this is right

  • LLM kernel-generation results on one accelerator cannot be extrapolated to others: all seven models show large Pass@1 drops on AscendC and Pallas relative to CUDA.
  • Category-aware one-shot prompting is a low-cost, prompt-only intervention that improves both compilation and correctness on platforms with little training-data coverage, with relative Pass@1 gains exceeding 100% on Pallas.
  • Fine-grained functional categories are necessary for meaningful evaluation: activation and convolution sit at the same coarse difficulty level yet differ by roughly 75 percentage points in average CUDA Pass@1.
  • Because the backend interface is plugin-based, the same evaluation pipeline can be extended to additional accelerators or DSLs; the paper estimates that supporting another DSL costs fewer than 20 lines of new code.
  • LLM-generated kernels occasionally beat the eager-mode baseline through sparsity exploitation or operation fusion, but such wins are rare and mostly confined to simple categories.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the same in-category-exemplar effect could be automated by retrieving a relevant kernel from a library at prompt time, turning a fixed one-shot into a dynamic selection that scales to all 14 categories; the paper demonstrates the principle on five categories only.
  • Beyond the paper: because correctness rests on five random tests, the reported Pass@k values are best treated as upper bounds until the $N=5$ rule is validated with larger test counts on AscendC and Pallas.
  • Beyond the paper: the observed shape-sensitivity in Reduce failures suggests that category-aware prompting could be extended to shape-aware prompting, selecting exemplars whose tensor ranks match the target, a variant the benchmark infrastructure could host without rework.
  • Beyond the paper: applying the same benchmark and prompting strategy to other under-represented accelerator platforms would test whether the training-coverage bottleneck and the prompt remedy generalize beyond the three platforms studied.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces MultiKernelBench, a benchmark for evaluating LLM-generated deep-learning kernels on three hardware platforms: NVIDIA GPUs (CUDA), Huawei NPUs (AscendC), and Google TPUs (Pallas). The benchmark extends KernelBench to 285 tasks in 14 functional categories, provides a modular backend abstraction layer for adding new platforms, and proposes a category-aware one-shot prompting strategy. The authors evaluate seven LLMs using Compilation@k, Pass@k, and SpeedUp@k, and report that LLMs perform poorly on AscendC and Pallas, that task difficulty varies strongly across categories, and that category-aware one-shot examples improve correctness on the less-represented platforms.

Significance. If validated, MultiKernelBench would fill a real gap: prior kernel-generation benchmarks target NVIDIA hardware only, while this paper adds AscendC and Pallas support, finer-grained categories, and an extensible backend interface. The empirical findings on poor cross-platform generalization and the benefit of category-specific exemplars are of interest to the code-generation and systems communities, and the public release of the benchmark is a concrete asset. The main strengths are the breadth of the task suite, the multi-platform compilation pipeline, and the clear decomposition of benchmarking logic from platform-specific code. However, the central empirical claims currently depend on an insufficiently validated correctness oracle and an underspecified timing method, so the results should be treated as conditional until those gaps are addressed.

major comments (4)
  1. [Section 3.5 (Correctness)] The claim that N=5 random tests are sufficient on AscendC and Pallas is asserted without any supporting data. The sentence 'We reproduced this phenomenon on AscendC and Pallas' provides no script, table, or citation, and reference [29] validates N=5 only for CUDA. Because every Pass@k result in Tables 4, 6, 7, 8, and 9 and the RQ3 improvement claim use this oracle, a wrong kernel that hard-codes outputs for the five sampled inputs, writes only a subset of output elements, or fails on shapes not exercised by chance would be labeled correct and would inflate the reported pass rates. Please add a validation study comparing N=5 against a larger test count (e.g., N=100) on a sample of tasks from each of the three platforms, specify exactly how random inputs are generated, and release the corresponding scripts.
  2. [Section 3.5 (Performance)] The Pallas timing methodology is described only as 'we use debugging tools to measure execution time.' This does not identify the tool, the measurement protocol, the number of repeats, or how synchronization is handled. Since SpeedUp@k values in Tables 4 and 9 and the speedup case studies in Section 4.3 depend on these measurements, the metric is not reproducible. Please specify the exact API or profiler used, the number of trials, and any variance-reduction procedure, and provide evidence that the timing is stable across runs.
  3. [Section 4.4 (RQ3)] Tables 8 and 9 report results only for the 'top three models with the highest pass rates under this setting,' which is a post-hoc selection. The relative improvements (e.g., GPT-4o's 160% Pass@1 improvement on AscendC) may not hold for the omitted models, so the general conclusion that category-aware one-shot prompting improves performance is not supported as stated. Additionally, the meaning of the 'Rel. Improve' row is not defined: it could be an average of per-category relative changes, a ratio of aggregate pass counts, or something else. Please report all seven models, or at least a pre-specified subset, and define the aggregate improvement measure explicitly.
  4. [Section 4.1 and Table 4] Reasoning models are exempted from stochastic sampling, so for DeepSeek-R1 and Qwen3-235B (think) the reported Pass@5 values are identical to Pass@1, while non-reasoning models benefit from 5 samples. This makes the Pass@5 comparisons in Table 4 unfair across model families and confounds the RQ1 conclusion that reasoning models perform better overall. Either run stochastic sampling for the reasoning models as well, or explicitly mark their Pass@5 entries as not applicable and restrict Pass@5 comparisons to models for which sampling was performed.
minor comments (5)
  1. [Section 3.5] The sentence 'The relatively large tolerance values also accommodate potential precision-reducing optimizations, such as using BF16 or FP8' is vague: if BF16 or FP8 outputs are allowed, the reference implementation should be run in a matching precision, otherwise a tolerance of atol=rtol=1e-2 can hide systematic errors. Please clarify how dtype mismatches between the generated kernel and the reference are handled.
  2. [Tables 6 and 7] The row 'Others All Pass@1 = 0.0' does not say which categories were merged. Please list the merged categories so the reader can assess per-category difficulty.
  3. [Tables 8 and 9] The notation '↑200.0', '↑133.3', and '↑300.0' in the 'Rel. Improve' row is ambiguous: it is not clear whether these are percentages, percentage points, or multiplicative factors. Please label the units clearly.
  4. [Section 5] The statement that 'supporting Triton requires fewer than 20 lines of additional code' is unverifiable without code. Please either provide the example implementation or remove the claim.
  5. [General] There are minor textual issues: 'Pytorch' should be 'PyTorch' in several places, and the figure captions in Figures 3, 4, and 5 spell 'Implementation' as 'Implmentation'. A final proofreading pass would fix these.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: benchmark construction, prompting interventions, and evaluation metrics are externally grounded; the unsupported N=5 claim is a validity concern, not a circular derivation.

full rationale

MultiKernelBench is an empirical benchmark paper rather than a derivation chain. The 285 tasks are constructed by extending KernelBench [29] and reviewing the PyTorch ATen CUDA codebase, with new categories and tasks manually verified; this is external grounding, not self-definition. The category-aware one-shot prompting study uses exemplars collected from official AscendC/Pallas documentation and open-source repositories, which are independent of the target tasks, so the reported improvement is not forced by construction. The correctness oracle (N=5, atol=rtol=1e-2) is adopted from prior work, and although Section 3.5 asserts that the authors 'reproduced this phenomenon on AscendC and Pallas' without providing supporting data, that assertion is a robustness/validity threat rather than a circular step: Pass@k values are not algebraically equal to any fitted parameter or to the oracle itself, and the benchmark's conclusions are not obtained by renaming its inputs. There are no self-citations to the present authors' prior work, no imported uniqueness theorems, and no ansatz smuggled in via citation. The skeptic's concern that N=5 may be insufficient on platforms with silent runtime misbehavior is legitimate but belongs under correctness risk, not circularity. Therefore the paper is self-contained against external benchmarks and receives a circularity score of 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or mathematical entities. The backend abstraction layer is a software design pattern, not a postulated entity. The main assumptions are about the validity of the correctness check and the representativeness of the taxonomy.

free parameters (4)
  • Number of random correctness tests N = 5
    Chosen following KernelBench [29]; the authors assert it is sufficient on all three platforms but only cite evidence for CUDA.
  • Correctness tolerances atol/rtol = 1e-2 / 1e-2
    Inherited from prior work; loose enough to admit kernels with small numerical errors as correct.
  • Sampling temperature and top-p for k=5 = 0.2 / 0.95
    Standard code-generation sampling setting, following prior benchmarks.
  • Speedup threshold alpha = 1.0
    SpeedUp_1@k reports the fraction of tasks with at least 1x speedup; no sensitivity analysis is provided.
assumptions (4)
  • domain assumption Five random input tests are sufficient to evaluate kernel correctness on AscendC and Pallas.
    Stated in Section 3.5; for CUDA it cites KernelBench [29], but no supporting data is shown for the new platforms.
  • domain assumption Kernel correctness can be checked with atol=1e-2 and rtol=1e-2.
    Section 3.5; tolerances are loose enough to mask wrong kernels with small numerical differences.
  • domain assumption Performance timing is stable across runs on all three platforms.
    Section 3.5 asserts stability 'within a small range' without reporting variance or measurement methodology for Pallas.
  • domain assumption The 14-category taxonomy is complete and representative of DL kernels.
    Section 3.3; categories were defined by consensus of three developers and pruned by judgment, not by an external standard.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MultiKernelBench: A Multi-Platform Benchmark for Kernel Generation." pith.science (2026). https://pith.science/paper/UXUR3YUK

@misc{pith2026250717773,
  author       = {Pith},
  title        = {Pith review of: MultiKernelBench: A Multi-Platform Benchmark for Kernel Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UXUR3YUK}},
  note         = {Machine review of arXiv:2507.17773}
}
read the original abstract

The automatic generation of deep learning (DL) kernels using large language models (LLMs) has emerged as a promising approach to reduce the manual effort and hardware-specific expertise required for writing high-performance operator implementations. However, existing benchmarks for evaluating LLMs in this domain suffer from limited hardware support, coarse-grained kernel categorization, and imbalanced task coverage. To address these limitations, we introduce MultiKernelBench, the first comprehensive, multi-platform benchmark for LLM-based DL kernel generation. MultiKernelBench spans 285 tasks across 14 well-defined kernel categories and supports three major hardware platforms: Nvidia GPUs, Huawei NPUs, and Google TPUs. To enable future extensibility, we design a modular backend abstraction layer that decouples platform-specific logic from the core benchmarking infrastructure, allowing easy integration of new hardware platforms. We further propose a simple yet effective category-aware one-shot prompting method that improves generation quality by providing in-category exemplars. Through systematic evaluations of seven state-of-the-art LLMs, we reveal significant variation in task difficulty, poor generalization to platforms with less training exposure, and the effectiveness of targeted prompting strategies. MultiKernelBench is publicly available at https://github.com/wzzll123/MultiKernelBench.

Figures

Figures reproduced from arXiv: 2507.17773 by the authors.

Figure 2
Figure 2. Framework of MultikernelBench. as activation functions and normalization layers). (2) Gap Iden￾tification via Kernel Library Review: We then reviewed Pytorch native CUDA kernel library [7], assessing whether each kernel could be mapped to an existing category. Kernels that did not fit were collected as candidates for new categories. (3) Integration and Refinement: Candidate kernels were systematically categorized, a… view at source ↗
Figure 3
Figure 3. AscendC Compilation Error Example. of DeepSeek-R1’s relatively high hallucination rate [3] and limited exposure to AscendC and Pallas code in its training corpus. Findings: (1) MultiKernelBench is a challenging benchmark for testing LLMs. (2) Reasoning-augmented models tend to perform better overall, while smaller models like Qwen-32B underperform, showing the impact of both reasoning ability and model scale. Platfo… view at source ↗
Figure 4
Figure 4. Diagonal Matrix Multiplication. # Pytorch forward x = self.gemm(x) x = torch.max(x, dim=self.max_dim, keepdim=True).values x = x - x.mean(dim=1, keepdim=True) x = torch.nn.functional.gelu(x) # Claude Pallas Implmentation def fused_max_submean_gelu_kernel(x_ref, out_ref): x = x_ref[...] x_max = jnp.max(x, axis=1, keepdims=True) x_mean = jnp.mean(x_max, axis=1, keepdims=True) x_centered = x_max - x_mean sqrt_2_over_pi… view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Kernel Fusion Example. tailored to specific tasks. For instance, as shown in [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. KernelBench-Verified: Do LLM-Generated Kernels Actually Beat PyTorch?

    cs.LG 2026-06 conditional novelty 7.0 of 10

    Under a TF32-enabled baseline and multi-distribution hidden tests, the best LLM kernel generator (GPT-5.5) achieves 0.88x geometric mean speedup versus PyTorch, not the reported 1.43x.

  2. Don't Regenerate, Debug: A Domain-Specific Agent for Repairing Near-Miss Hardware Operators

    cs.SE 2026-08 conditional novelty 6.0 of 10

    Debugging near-miss AscendC kernels with a domain-specific agent recovered 18/27 operators versus 11/27 from three regeneration trials, while using fewer tokens per success.

  3. KernelGenBench: A Multi-Source and Multi-Chip Benchmark for LLM-based Kernel Generation

    cs.AI 2026-07 conditional novelty 6.0 of 10

    A multi-source (ATen/vLLM/cuBLAS, 210 ops) and multi-chip (6 platforms) benchmark shows agentic kernel generation beats sampling but degrades sharply off NVIDIA and costs millions of tokens per success.

  4. CANN Bench: Benchmarking Agent Generated Kernels against Real NPU and Algorithmic Limits

    cs.AI 2026-07 conditional novelty 6.0 of 10

    CANN Bench is an Ascend-NPU benchmark with 53 operators and 1,060 released test cases that scores agent-generated kernels on compilation, correctness, and hardware-anchored performance.

  5. MKEvolve: A Modular Multi-Agent Framework for Kernel Code Generation

    cs.AI 2026-06 conditional novelty 6.0 of 10

    MKEvolve iteratively co-evolves a modular decomposition of PyTorch modules and per-subkernel Triton implementations, improving KernelBench L2/L3 correctness and speedup while cutting LLM tokens by up to 35%.

  6. Kernel Forge: An Agent Harness for LLM-based Generation and Optimization of CUDA Kernels

    cs.AI 2026-06 conditional novelty 6.0 of 10

    A new open-source harness uses an LLM and MCTS to generate CUDA kernels for unmodified PyTorch models, achieving operator-level speedups up to 2.83x over eager mode with guarded fallback.

  7. Compiler-Grounded Hierarchical Diagnosis for LLM-Based Triton Kernel Optimization

    cs.AI 2026-07 conditional novelty 5.0 of 10

    A hierarchical, compiler-grounded LLM optimization loop reports 4.35x geometric-mean Triton kernel speedups on 37 Ascend NPU benchmark kernels, but with no causal ablation yet.

Reference graph

Works this paper leans on

47 extracted references · 22 canonical work pages · cited by 7 Pith papers

  1. [29]

    Anne Ouyang, Simon Guo, Simran Arora, Alex L Zhang, William Hu, Christopher Ré, and Azalia Mirhoseini. 2025. Kernelbench: Can llms write efficient gpu kernels? arXiv preprint arXiv:2502.10517 (2025)

    arXiv 2025

  2. [1]

    Introduction to AscendC

    2025. Introduction to AscendC . https://www.hiascend.com/document/detail/ zh/canncommercial/81RC1/developmentguide/opdevg/Ascendcopdevg/atlas_ ascendc_10_0001.html

    work page 2025

  3. [2]

    Introduction to AscendC

    2025. Introduction to AscendC. https://gitee.com/ascend/samples

    work page 2025

  4. [3]

    Leaderboard Comparing LLM Performance at Producing Hallucinations

    2025. Leaderboard Comparing LLM Performance at Producing Hallucinations . https://github.com/vectara/hallucination-leaderboard

    work page 2025

  5. [4]

    LLM Rankings

    2025. LLM Rankings. https://openrouter.ai/rankings/programming

    work page 2025

  6. [5]

    Pallas: a JAX kernel language

    2025. Pallas: a JAX kernel language . https://docs.jax.dev/en/latest/pallas/index. html

    work page 2025

  7. [6]

    Python Built-in Function exec

    2025. Python Built-in Function exec . https://docs.python.org/3/library/functions. html#exec

    work page 2025

  8. [7]

    PyTorch ATen CUDA Codebase

    2025. PyTorch ATen CUDA Codebase. https://github.com/pytorch/pytorch/tree/ main/aten/src/ATen/native/cuda

    work page 2025

Show all 47 references
  1. [8]

    Pytorch Cpp Extension

    2025. Pytorch Cpp Extension. https://docs.pytorch.org/docs/stable/cpp_extension. html

    2025

  2. [9]

    Tensor Cores

    2025. Tensor Cores. https://www.nvidia.com/en-us/data-center/tensor-cores/

    2025

  3. [10]

    torch npu APIs

    2025. torch npu APIs . https://www.hiascend.com/doc_center/source/zh/ canncommercial/63RC2/modeldevpt/ptmigr/ptmigr_0193.html

    2025

  4. [11]

    torch.cuda.Event

    2025. torch.cuda.Event. https://docs.pytorch.org/docs/stable/generated/torch. cuda.Event.html

    2025

  5. [12]

    TPU architecture

    2025. TPU architecture. https://cloud.google.com/tpu/docs/system-architecture- tpu-vm

    2025

  6. [13]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    2023 arXiv

  7. [14]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021. Program Synthesis with Large Language Models. arXiv:2108.07732 [cs.PL] https://arxiv.org/abs/2108.07732

    2021 arXiv

  8. [15]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey W...

    2020

  9. [16]

    Jialun Cao, Zhiyong Chen, Jiarong Wu, Shing-Chi Cheung, and Chang Xu

  10. [17]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott G...

    2021 arXiv

  11. [18]

    DeepSeek-AI, Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y. Wu, Yukun Li, Huazuo Gao, Shirong Ma, Wangding Zeng, Xiao Bi, Zihui Gu, Hanwei Xu, Damai Dai, Kai Dong, Liyue Zhang, Yishi Piao, Zhibin Gou, Zhenda Xie, Zhewen Hao, Bingxuan Wang, Junxiao So...

    2024 arXiv

  12. [19]

    Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. 2024. Evaluating Large Language Models in Class-Level Code Generation. InProceedings of the IEEE/ACM 46th International Conference on Software Engine...

    2024

  13. [20]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)

    2024 arXiv

  14. [21]

    Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy S Liang. 2019. SPoC: Search-based Pseudocode to Code. In Advances in Neural Information Processing Systems , H. Wallach, H. Larochelle, A. Beygelzimer, F. d 'Alché-Buc, E. Fox, and R. G...

    2019

  15. [22]

    Jia Li, Ge Li, Yunfei Zhao, Yongmin Li, Huanyu Liu, Hao Zhu, Lecheng Wang, Kaibo Liu, Zheng Fang, Lanshen Wang, Jiazheng Ding, Xuanming Zhang, Yuqi Zhu, Yihong Dong, Zhi Jin, Binhua Li, Fei Huang, Yongbin Li, Bin Gu, and Mengfei Yang. 2024. DevEval: A Manually-Annotated Code G...

    2024

  16. [23]

    Jianling Li, Shangzhan Li, Zhenye Gao, Qi Shi, Yuxuan Li, Zefan Wang, Jiacheng Huang, Haojie Wang, Jianrong Wang, Xu Han, Zhiyuan Liu, and Maosong Sun

  17. [24]

    Zongjie Li, Daoyuan Wu, Shuai Wang, and Zhendong Su. 2025. API-Guided Dataset Synthesis to Finetune Large Code Models. Proc. ACM Program. Lang. 9, Conference’17, July 2017, Washington, DC, USA Zhongzhen Wen, Yinghui Zhang, Zhong Li, Zhongxin Liu, Linna Xie, and Tian Zhang OOPS...

    2025 doi

  18. [25]

    Heng Liao, Jiajin Tu, Jing Xia, Hu Liu, Xiping Zhou, Honghui Yuan, and Yuxing Hu. 2021. Ascend: a Scalable and Unified Architecture for Ubiquitous Deep Neural Network Computing : Industry Track Paper. In 2021 IEEE International Symposium on High-Performance Computer Architectu...

    2021

  19. [26]

    Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al . 2024. Deepseek- v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434 (2024)

    2024 arXiv

  20. [27]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Cheng- gang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 (2024)

    2024 arXiv

  21. [28]

    Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2024. WizardCoder: Empowering Code Large Language Models with Evol-Instruct. In The Twelfth International Conference on Learning Representations . https://...

    2024

  22. [30]

    Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xi- aoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton-Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défo...

    2023 doi arXiv

  23. [31]

    Disha Shrivastava, Hugo Larochelle, and Daniel Tarlow. 2023. Repository-Level Prompt Generation for Large Language Models of Code. InProceedings of the 40th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 202), Andreas Krause, Emma ...

    2023

  24. [32]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guil- laume Lample. 2023. LLaMA: Open and Efficient Foundation ...

    2023 arXiv

  25. [33]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yas- mine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhos- ale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucu- rull, David Esiobu, Jude Fernandes, Jeremy...

    2023 arXiv

  26. [34]

    Zhiruo Wang, Shuyan Zhou, Daniel Fried, and Graham Neubig. 2023. Execution- Based Evaluation for Open-Domain Code Generation. In Findings of the Asso- ciation for Computational Linguistics: EMNLP 2023 , Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computat...

    2023 doi

  27. [35]

    Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. 2024. Magicoder: Empowering Code Generation with OSS-Instruct. In Proceedings of the 41st International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 235), Ruslan Salakhutdinov...

    2024

  28. [36]

    Mengdi Wu, Xinhao Cheng, Oded Padon, and Zhihao Jia. 2024. A Multi-Level Su- peroptimizer for Tensor Programs. CoRR abs/2405.05751 (2024). arXiv:2405.05751 doi:10.48550/ARXIV.2405.05751

    2024 doi arXiv

  29. [37]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jian Yang, Jiaxi ...

    2025 doi arXiv

  30. [38]

    Daoguang Zan, Bei Chen, Zeqi Lin, Bei Guan, Wang Yongji, and Jian-Guang Lou

  31. [39]

    Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. 2023. RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural L...

    2023

  32. [40]

    Xu, Zhengbao Jiang, and Graham Neubig

    Shuyan Zhou, Uri Alon, Frank F. Xu, Zhengbao Jiang, and Graham Neubig

  33. [41]

    Yuhang Zhou, Zhibin Wang, Guyue Liu, Shipeng Li, Xi Lin, Zibo Wang, Yongzhong Wang, Fuchun Wei, Jingyi Zhang, Zhiheng Hu, Yanlin Liu, Chun- sheng Li, Ziyang Zhang, Yaoyuan Wang, Bin Zhou, Wanchun Dou, Guihai Chen, and Chen Tian. 2025. Squeezing Operator Performance Potential f...

    2025

  34. [42]

    Qiming Zhu, Jialun Cao, Yaojie Lu, Hongyu Lin, Xianpei Han, Le Sun, and Shing-Chi Cheung. 2025. DOMAINEVAL: An Auto-Constructed Benchmark for Multi-Domain Code Generation. In AAAI-25, Sponsored by the Association for the Advancement of Artificial Intelligence, February 25 - Ma...

    2025 doi

  35. [43]

    Terry Yue Zhuo, Vu Minh Chien, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen GONG, James Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen- Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Y...

    2025

  36. [2022]

    In Findings of the Association for Computational Linguistics: EMNLP 2022 , Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (Eds.)

    When Language Model Meets Private Library. In Findings of the Association for Computational Linguistics: EMNLP 2022 , Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (Eds.). Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 277–288. doi:10.18653/v1/20...

    2022 doi

  37. [2023]

    In The Eleventh International Conference on Learning Representations

    DocPrompting: Generating Code by Retrieving the Docs. In The Eleventh International Conference on Learning Representations . https://openreview.net/ forum?id=ZTCxT2t2Ru

  38. [2024]

    In Proceedings of the 39th IEEE/ACM Interna- tional Conference on Automated Software Engineering (Sacramento, CA, USA) (ASE ’24)

    JavaBench: A Benchmark of Object-Oriented Code Generation for Eval- uating Large Language Models. In Proceedings of the 39th IEEE/ACM Interna- tional Conference on Automated Software Engineering (Sacramento, CA, USA) (ASE ’24). Association for Computing Machinery, New York, NY...

  39. [2025]

    arXiv:2502.14752 [cs.CL] https://arxiv.org/abs/2502.14752

    TritonBench: Benchmarking Large Language Model Capabilities for Gener- ating Triton Operators. arXiv:2502.14752 [cs.CL] https://arxiv.org/abs/2502.14752

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.