REVIEW 3 major objections 7 minor
CUDA-L2: Surpassing cuBLAS Performance for Matrix Multiplication through Reinforcement Learning
T0 review · 3 major / 7 minor · reviewed 2026-08-03 · deepseek-v4-flash
Pith's one-line read An LLM guided by reinforcement learning writes FP16 matrix-multiplication kernels for A100 GPUs that outperform NVIDIA's exhaustively autotuned cuBLASLt by 11.4% on average.
desk verdict First LLM/RL-discovered HGEMM kernels reported to beat cuBLASLt-AutoTuning on A100, but the correctness gate in §2.3.2 is too permissive to rule out an accuracy tradeoff, and no artifacts are provided to check the claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the contrastive reinforcement-learning loop: an LLM proposes CUDA kernels, the kernels are compiled and executed on an A100, and the reward is the measured speedup over a reference kernel, with penalties for numerical deviation and code length. The training is staged—continued pretraining on diverse CUDA code, general-kernel RL, then HGEMM-specific RL—and the model sees NCU profiling metrics (memory throughput, occupancy, cache efficiency) plus retrieval-augmented context. The correctness gate is the filter that determines what counts as a successful kernel: exact agreement with an FP32 CPU reference on binary inputs where the reference entry is below 2048, and
What would settle it
Take CUDA-L2's best kernel for a given shape and compare it against an FP64 CPU reference on random half-precision inputs with large entry magnitudes; if many kernels deviate more than the NVIDIA baselines on high-magnitude outputs, the speedup claim is partly an accuracy tradeoff. A second check: recompile the kernels with forced FP32 accumulation—if the average speedup over cuBLASLt-AutoTuning vanishes or reverses, reduced-precision accumulation is the source of the gain.
Extended reading notes
Core claim
The central discovery is that a staged LLM+RL loop, trained first on general CUDA code and then specifically on HGEMM with measured speed as reward, can generate kernels that beat the strongest autotuning baseline NVIDIA provides. On all 1,000 (M,N,K) configurations drawn from dimensions 64 through 16384, the generated kernels are reported to be faster than cuBLASLt-AutoTuning—which benchmarks up to 100 heuristic candidates and picks the best—by 11.4% on average offline and 15.9% when kernels run at random intervals, with win rates above 79%. The authors attribute the gains to a combination of abstraction selection (raw WMMA for small matrices, CuTe for large ones), zero-padding to enable ti
Load-bearing premise
A kernel is judged correct if it exactly matches an FP32 CPU reference on binary inputs only for output entries below 2048, and otherwise if its biggest deviation is no larger than the biggest disagreement among the NVIDIA baselines; if faster kernels pass this gate by using less accurate arithmetic on high-magnitude values, the reported speedups would partly reflect reduced accuracy, not better kernels.
Editorial extensions
If this is right
- NVIDIA's autotuned cuBLASLt is not the per-shape optimum for FP16 matmul on A100; a learned search finds kernels faster than the best of up to 100 heuristic candidates.
- The specific moves the search discovers—zero-padding to non-dividing tile sizes, double-buffered register fragments, staggered A/B prefetch, direct wide epilogue copies—give kernel developers a concrete menu of techniques to try.
- Offline and server execution select different kernels; performance evaluations should report both, since thermal and clock dynamics change which optimization wins.
- The same RL pipeline, if the claim generalizes, transfers to other kernel families and GPU architectures without manual re-derivation of per-shape strategies.
- Combining CUDA-L2 kernels with library kernels (taking the faster of the two per shape) beats either alone, so even partial successes complement existing libraries.
Reading between the lines
- The speedups are measured on A100 only; I would expect them to shrink on newer architectures, where tensor-core shapes and memory hierarchies differ, and the paper's own analysis ties several gains to A100-specific behavior.
- The correctness gate is silent on entries with magnitude above 2048 for binary inputs; a natural stress test is to run the accepted kernels on realistic FP16 values in that range and check whether error grows faster than the NVIDIA baselines' error.
- Because server mode rewards kernels that win from a cooler GPU state, an autotuner could specialize per deployment mode, and the reported server gains suggest clock-boost dynamics are a first-class optimization target.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes CUDA-L2, a system that combines continued pretraining of a large language model with multi-stage reinforcement learning to automatically generate HGEMM CUDA kernels for A100 GPUs. The system is evaluated on 1,000 (M,N,K) configurations drawn from a grid of ten sizes per dimension, comparing against torch.matmul, cuBLAS, cuBLASLt-heuristic, and cuBLASLt-AutoTuning in two scenarios: offline (back-to-back execution) and server (random intervals). The headline results are average speedups of +22.0% over torch.matmul, +19.2% over cuBLAS-max, +16.8% over cuBLASLt-heuristic-max, and +11.4% over cuBLASLt-AutoTuning-max in offline mode, with larger gains in server mode. The paper also analyzes the optimization techniques discovered by the system, such as non-divisible tile padding, double-buffered register fragments, aggressive prefetching, and staggered A/B prefetch scheduling.
Significance. If the results hold, this is a notable empirical demonstration that LLM-guided RL can generate FP16 GEMM kernels for a flagship GPU that beat NVIDIA's aggressively autotuned cuBLASLt kernels on a wide range of shapes. The evaluation design is thoughtful in several respects: it includes a realistic strong baseline (cuBLASLt-AutoTuning with up to 100 candidates per shape), uses a 10-second warmup and 30-second timing runs with randomized order, and reports win rates in addition to mean speedups. The paper's analysis of discovered techniques (e.g., padding to non-divisible tile sizes, ping-pong register buffers, multi-step prefetching) is informative and goes beyond a simple 'it works' claim. The availability of code and the detailed appendix listings also support reproducibility. However, the central claim of outperforming cuBLASLt-AutoTuning 'at equal correctness' rests on a correctness gate that is not adequately validated, and the aggregate average speedup is reported without variance or per-size breakdown, which makes the systematic nature of the improvement hard to assess.
major comments (3)
- [§2.3.2, Listing 6] The correctness gate is load-bearing for the speedup claim, and it is too permissive. The 'Exact Match with binary Inputs' test only enforces exactness for output entries with reference value <2048; the paper notes that larger entries are ignored because they are not exactly representable in FP16. The second criterion, 'Baseline-Bounded Deviation', accepts a kernel if its max deviation from the FP32 CPU reference is no larger than the maximum disagreement among the NVIDIA baselines. However, the baselines are invoked with CUBLAS_COMPUTE_16F (Listing 6), which permits reduced-precision accumulation. Thus a kernel using FP16 accumulation—a design choice the paper itself identifies as speed-relevant in §1—could pass the gate even if its accuracy is worse than a strict FP32-accumulation reference. The paper does not state how often the baseline-bounded fallback is used, nor does it compare a
- [§4.1, Table 1] The reported average speedups lack measures of variance or statistical significance. Given that §4.3 shows speedups decline with problem size and approach 1.0× for large matrices, the overall mean is sensitive to the composition of the 1,000-configuration grid. The paper should report per-size-bucket means and standard deviations, and ideally perform a paired test (e.g., Wilcoxon signed-rank) between CUDA-L2 and cuBLASLt-AutoTuning to support the claim that the improvement is 'systematic rather than driven by outliers.' As written, the win rates (79.3-95.7%) are a useful but incomplete substitute.
- [Eq. (3)] The RL reward in Eq. (3) includes penalty coefficients α and β, but their numerical values are never reported. This is an important missing experimental detail: the trade-off between speed and numerical deviation in the reward depends critically on α, and the code-length penalty on β. Without these values, the training procedure is not reproducible and it is impossible to assess how strongly the RL objective penalizes incorrect or low-accuracy kernels. Please provide the values used and, if possible, the observed ranges of diff_i and L(custom) during training.
minor comments (7)
- [Abstract] The abstract states 'all 103 combinations'; the intended meaning is likely '10^3' (i.e., 1,000 combinations). Please correct the typesetting.
- [§2.2.3] The sentence 'for each dimension , we benchmark...' is missing a word or phrase; recommend 'for each dimension configuration'.
- [§4.1] Typo: 'The explains the larger variance' should be 'This explains the larger variance'.
- [Figure 3] The y-axis is labeled 'Relative Speedup' but the text describes values in terms of multipliers (1.4×, 1.0×). Clarify whether the quantity is s = t_ref/t_custom - 1 or t_ref/t_custom, and make the axis label consistent.
- [§5.2] The padding example reports a +15.2% speedup over 'cublaslt-AutoTuning-TN' only. For consistency with Table 1, also report the speedup against the max(NN,TN) baseline used elsewhere.
- [§6.1] The correlation coefficients (ρ) are reported without confidence intervals or significance tests. Since the grid is deterministic, the coefficients are descriptive; please state this explicitly to avoid overinterpretation.
- [§2.4.2] The 'server mode' uses 'random intervals' but no distribution or mean interval is specified. Provide details (e.g., Poisson process with what rate) so the scenario is reproducible.
Circularity Check
No significant circularity: HGEMM speedups are measured results from empirical kernel search, not predictions derived from their own inputs.
full rationale
The paper's central claim is an empirical benchmark result: CUDA-L2 generates HGEMM kernels through LLM-guided RL and measures execution times against independent NVIDIA baselines (torch.matmul, cuBLAS, cuBLASLt-heuristic, cuBLASLt-AutoTuning). There is no derivation chain in which an output is defined in terms of the claimed result. Equation (3) defines the RL reward with the same speedup score later reported, but that is standard objective optimization, not circular prediction: reported speedups are direct measurements on generated kernels, not fitted predictions. The paper cites its own CUDA-L1 for the contrastive RL strategy; this self-citation is methodological reuse and is not load-bearing for the HGEMM speedup claim, which is validated against independent closed-source NVIDIA libraries. Evaluation on the same 1,000 configurations used during optimization is an in-sample generalization concern, not circularity, because the paper claims achieved performance on those configurations. The correctness gate in Section 2.3.2 is the most substantive weakness: it allows kernels whose max deviation is no worse than the max disagreement among NVIDIA baselines, and the binary-input exact-match test ignores reference entries greater than or equal to 2048. This is a benchmark fairness / accuracy tradeoff concern, not a self-referential reduction, because speedup is not defined in terms of this gate and a stricter test could falsify the headline. No circular step is exhibited, so the appropriate finding is no significant circularity.
Assumptions & free parameters
free parameters (3)
- RL reward penalty coefficients α and β
- Per-config kernel search parameters (tile sizes BM/BN/BK, pipeline stages, swizzle stride, prefetch distance) =
learned per config, not listed
- Correctness acceptance threshold 'baseline max deviation' =
varies per input/config
assumptions (4)
- domain assumption CUDA event timing of a raw kernel launch is a stable proxy for true end-to-end performance and reward.
- domain assumption NVIDIA's cuBLAS/cuBLASLt baselines configured with CUBLAS_COMPUTE_16F are fair and representative strong references.
- domain assumption The 1,000-configuration benchmark is an adequate proxy for production LLM matmul workloads and for the generalization claim.
- standard math FP16 exactly represents all integers in [0,2048) due to 10+1 significand bits.
Cite this review
Pith. "Pith review of CUDA-L2: Surpassing cuBLAS Performance for Matrix Multiplication through Reinforcement Learning." pith.science (2026). https://pith.science/paper/ATFTBZLX
@misc{pith2026251202551,
author = {Pith},
title = {Pith review of: CUDA-L2: Surpassing cuBLAS Performance for Matrix Multiplication through Reinforcement Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/ATFTBZLX}},
note = {Machine review of arXiv:2512.02551}
}
read the original abstract
In this paper, we propose CUDA-L2, a system that combines large language models (LLMs) and reinforcement learning (RL) to automatically optimize Half-precision General Matrix Multiply (HGEMM) CUDA kernels. Using CUDA execution speed as the RL reward, CUDA-L2 automatically optimizes HGEMM kernels across 1,000 configurations. CUDA-L2 systematically outperforms major matmul baselines to date, from the widely-used torch.matmul to state-of-the-art Nvidia's closed-source libraries, i.e., cuBLAS, cuBLASLt. In offline mode, where kernels are executed consecutively without time intervals, CUDA-L2 yields +22.0% over torch.matmul on average; +19.2% over cuBLAS using the optimal layout configuration (normal-normal NN and transposed-normal TN); +16.8% over cuBLASLt-heuristic, which queries cuBLASLt library and selects the algorithm based on the heuristic's suggestion; and +11.4% over the most competitive cuBLASLt-AutoTuning model, which selects the fastest algorithm from up to 100 candidates from cuBLASLt's suggestions. In server mode, where kernels are executed at random intervals simulating real-time inference, the speedups further increase to +28.7%, +26.0%, +22.4%, and +15.9% for torch.matmul, cuBLAS, cuBLASLt-heuristic, and cuBLASLt-AutoTuning respectively. CUDA-L2 shows that even the most performance-critical, heavily-optimized kernels like HGEMM can be improved through LLM-guided RL automation by systematically exploring configuration spaces at scales impractical for humans. Project and code can be found at github.com/ornith-ai/CUDA-L2
Figures
Reviewed August 3, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.