Pith. sign in

REVIEW 2 major objections 5 minor 24 references

Performance Evaluation of General Purpose Large Language Models for Basic Linear Algebra Subprograms Code Generation

T0 review · 2 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Two general-purpose LLMs generate correct, often faster C implementations of BLAS routines from routine names alone.

desk verdict Useful but overstated: correctness only measured for positive strides, so the central claim needs qualification. read the letter →

arxiv 2507.04697 v1 pith:UFE7BIXY submitted 2025-07-07 cs.LG cs.DCcs.MS

Daichi Mukunoki , Shun-ichiro Hayashi , Tetsuya Hoshino , Takahiro Katagiri This is my paper · ORCID
classification cs.LGcs.DCcs.MS
keywords BLASLLMcodegenerationGPT-4.1o4-miniOpenMPSIMDvectorizationcacheblockingnumericalkernels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Two general-purpose large language models, GPT-4.1 and o4-mini, were asked to write C implementations of 20 double-precision BLAS routines for CPUs. The authors tried three prompt conditions: the routine name alone, the name plus instructions to add OpenMP, SIMD vectorization, and cache blocking, and the Fortran reference code attached. They report that correct code was produced in many cases from the routine name alone, that o4-mini was generally more reliable, and that some optimized generated kernels ran several times faster than the unoptimized reference. The paper's central claim is that prompt-based LLM code generation for numerical codes of BLAS-level complexity is already practical when the routine specifications are stated accurately.

What carries the argument

The carrying mechanism is the pairing of a deliberately minimal prompt with an exact correctness oracle. The oracle is the run_tests.py test program (reference [23]) that embeds each generated C routine, checks it against reference BLAS over many parameter combinations and problem sizes, and measures throughput; this is what converts plausible code into correct code. The prompts vary the information given to the model: routine name only, routine name plus optimization instructions, or routine name plus the attached Fortran reference containing its specification comments. The performance artifacts in the generated code are OpenMP thread parallelization, SIMD pragmas or AVX intrinsics, and cache blocking with block sizes such as 64.

What would settle it

Re-run the same generated codes with the full parameter space, especially negative increments and conjugate-transpose variants; if most of the routines that 'passed' now produce wrong results, the claim that correct code is generated in many cases does not generalize beyond the tested subset.

Watch

Extended reading notes

Core claim

The central discovery is that the routine name alone is often enough for a general LLM to reconstruct a correct BLAS routine. Across 10 samples per prompt, o4-mini passed the correctness checks for all but one of the level-1 and level-2 routines in the name-only setting, and GPT-4.1 passed most; level-3 routines were harder, with dtrmm and dtrsm almost never passing. The authors observe that the generated code usually does not mirror the Fortran reference structure and generally omits argument-validation code unless the Fortran source is attached, which they read as evidence that the models learned the public specifications rather than memorizing the reference implementation. With optimization prompts, the models produced OpenMP-parallelized, SIMD-vectorized, cache-blocked code, and on large problems the best generated kernels exceeded the reference performance by roughly 3-13x for level-1 routines and larger margins for several level-3 parameter combinations. The paper concludes that code generation for numerical codes at BLAS-level complexity is already in the practical stage when specifications are written accurately.

Load-bearing premise

The load-bearing premise is that passing the reduced test set counts as correctness, but the tests omitted negative vector increments and conjugate-transpose modes, and the paper concedes that many generated codes failed when negative increments were supplied.

Editorial extensions

If this is right

  • If the central claim is right, general-purpose LLMs can serve as first-pass generators for numerical kernels, with a test harness used to filter the correct outputs from the failed ones.
  • Since o4-mini outperformed GPT-4.1 on correctness, reasoning-oriented models appear better suited than fast general models for this kind of numerical code synthesis.
  • Because the models appear to draw on public specifications rather than on the reference implementation, the accuracy of the prompt's description of the routine matters more than attaching example code.
  • Optimization prompts alone can yield working parallel and vectorized kernels, suggesting that basic performance engineering is accessible without coding-specialized models.
  • The persistent failures on routines such as dtrmm and dtrsm mark the current boundary of name-only generation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Extrapolating beyond the paper, the reported pass rates likely overstate general correctness: the harness was run without negative vector increments and without conjugate-transpose cases, and the paper's own footnote reports that many generated codes failed when negative increments were tried.
  • The performance comparison should be read against the unoptimized reference BLAS; a comparison with an optimized vendor BLAS library would test whether these speedups matter in practice, not just against a scalar baseline.
  • A testable extension is to prompt with explicit processor details, such as AVX-512 availability and cache sizes, which the authors suggest may set better block sizes and SIMD choices.
  • The practical-stage conclusion may not transfer outside the well-documented BLAS canon; generating routines with invented or proprietary specifications would separate genuine code-generation ability from retrieval of memorized specifications.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper evaluates two general-purpose OpenAI LLMs, GPT-4.1 and o4-mini, on the task of generating C implementations of 20 double-precision BLAS routines (levels 1-3). Three prompt conditions are compared: name-to-plain-C, name-to-optimized-C, and Fortran-reference-to-optimized-C. Correctness is checked with the BLAS++ test harness against reference LAPACK BLAS, and performance is measured on a 40-core Xeon using problem sizes exceeding cache capacity. The reported results indicate that o4-mini, and to a lesser extent GPT-4.1, can often generate code that passes the restricted test set from routine names alone, and that some generated codes with OpenMP/SIMD/cache-blocking optimizations outperform the unoptimized reference BLAS.

Significance. The study provides a well-structured, reproducible evaluation protocol using an external third-party harness (BLAS++) and reports explicit pass counts (out of 10) plus per-parameter performance tables. This is a useful baseline for the emerging area of LLM-based numerical code generation. The authors are transparent about several test-set restrictions, including the omission of negative increments and conjugate-transpose cases. If the correctness claim could be supported over the full BLAS parameter space, the results would constitute a meaningful positive data point for general LLMs in HPC code generation. The current restrictions, however, materially weaken the central claim that 'correct code can be generated in many cases.'

major comments (2)
  1. [§3.3, Table 3] The correctness evaluation is restricted to incx, incy in {1, 2} and omits conjugate-transpose cases; footnote 5 states that 'Many generated codes did not work correctly when negative values were given.' Negative increments are part of the BLAS specification and are standard test values in the BLAS++ harness (the text itself notes that run_tests.py normally uses 1, 2, -1, and -2). Therefore the pass rates in Table 3 and the abstract's claim that 'correct code can be generated in many cases' are not established for the full BLAS contract. This issue is load-bearing because correctness is the paper's primary evaluation axis; the authors should either run the full parameter set or explicitly restrict and rephrase the claim.
  2. [§4.2, Tables 4-6] Performance is reported per parameter combination, and the text concedes that 'this evaluation is conducted on a per-parameter combination basis, which does not necessarily mean that a single code obtained the performance for all parameter cases.' Since the correctness test is likewise restricted to positive increments and unit strides, the speedups in Tables 4-6 may be achieved by codes that fail on negative increments or other parameter combinations. No evidence is given for a single generated code that is both correct across the full parameter space and fast. The abstract's statement that 'the code is faster than the reference code' therefore overstates what the experimental design demonstrates; the authors should either report the performance of the best code that passes the full test set or add an explicit qualification in the abstract and conclusion.
minor comments (5)
  1. [§3.3, Abstract, Table 2] There are several presentation errors: 'LAPACK 3.12.167' should be 'LAPACK 3.12.1', the sentence ends with a doubled period, the abstract says 'routine name are given' instead of 'routine names are given', and the drotm equation in Table 2 is garbled.
  2. [Tables 5-6] The captions of Tables 5 and 6 should state explicitly that blank cells mean all generated codes failed; the text explains this, but the captions should be self-contained for readability.
  3. [§3.4] The three prompts are clearly specified, but the paper does not provide the generated code or a repository link; for a reproducibility-focused case study, making the prompts and at least representative generated codes available would be valuable.
  4. [§4.1] The observation that generated code structure often differs from the Fortran reference is interesting, but it is presented anecdotally; consider reporting how often structurally divergent codes pass the restricted correctness test, since the structural analysis is otherwise difficult to weigh.
  5. [§3.3, §5] The baseline is the unoptimized LAPACK reference BLAS; the paper should note more prominently that the reported speedups do not imply competitiveness with optimized BLAS libraries such as OpenBLAS or MKL.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found; the evaluation is an empirical comparison against external BLAS++/LAPACK reference checks.

full rationale

The paper's claim chain is: generate code from prompts, embed it into BLAS++, then check correctness against reference BLAS called via CBLAS and measure performance against the LAPACK reference implementation. These are external, third-party oracles, not derived from the LLM outputs or from any parameter fitted to the results. There are no fitted inputs, no self-citations by the present authors, and no equation-level reduction of a predicted quantity to an input. The restricted test settings described in Section 3.3, such as omitting negative increments and conjugate-transpose cases, are disclosed limitations of the external check; they weaken the generality of the correctness claim but do not make the evaluation circular. Similarly, the per-parameter performance numbers in Tables 5 and 6 are explicitly presented as not necessarily corresponding to a single code, which reduces the strength of the speedup claim but remains an empirical measurement against an external baseline.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claims depend on the faithfulness of the BLAS++ test harness as a correctness oracle, on the representativeness of the reduced parameter set (which the authors explicitly shrink), and on using unoptimized reference BLAS as both the correctness reference and performance baseline. No new entities or fitted parameters are introduced.

assumptions (3)
  • domain assumption BLAS++ run_tests.py correctly verifies BLAS routine behavior for the parameter combinations tested.
    Correctness of generated code is judged solely by this external harness, which the authors do not own or formally verify.
  • ad hoc to paper The reduced parameter set (positive incx/incy only, no conjugate-transpose cases for real routines) is sufficient to judge correctness.
    The paper omits negative increments and conjugate cases; footnote 5 in §3.3 admits 'Many generated codes did not work correctly when negative values were given,' meaning pass rates overstate general correctness.
  • domain assumption Reference LAPACK BLAS is correct and is an appropriate performance baseline.
    The reference BLAS is used both as the correctness oracle and as the speed baseline, so conclusions are only as strong as this simple, single-threaded implementation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Performance Evaluation of General Purpose Large Language Models for Basic Linear Algebra Subprograms Code Generation." pith.science (2026). https://pith.science/paper/UFE7BIXY

@misc{pith2026250704697,
  author       = {Pith},
  title        = {Pith review of: Performance Evaluation of General Purpose Large Language Models for Basic Linear Algebra Subprograms Code Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UFE7BIXY}},
  note         = {Machine review of arXiv:2507.04697}
}
read the original abstract

Generative AI technology based on Large Language Models (LLM) has been developed and applied to assist or automatically generate program codes. In this paper, we evaluate the capability of existing general LLMs for Basic Linear Algebra Subprograms (BLAS) code generation for CPUs. We use two LLMs provided by OpenAI: GPT-4.1, a Generative Pre-trained Transformer (GPT) model, and o4-mini, one of the o-series of Reasoning models. Both have been released in April 2025. For the routines from level-1 to 3 BLAS, we tried to generate (1) C code without optimization from routine name only, (2) C code with basic performance optimizations (thread parallelization, SIMD vectorization, and cache blocking) from routine name only, and (3) C code with basic performance optimizations based on Fortran reference code. As a result, we found that correct code can be generated in many cases even when only routine name are given. We also confirmed that thread parallelization with OpenMP, SIMD vectorization, and cache blocking can be implemented to some extent, and that the code is faster than the reference code.

Figures

Figures reproduced from arXiv: 2507.04697 by the authors.

Figure 1
Figure 1. The description of dsymm routine (partially. Au￾thors deleted empty lines). reference code uses if statement to separate four patterns of transposition mode (for transa and transb) combina￾tions, and a triple loop is written for each of them. However, none of the 10 generated codes has such a structure. In many cases, the length of the code is shortened. Furthermore, although the reference code has a code that outpu… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 11 canonical work pages

  1. [1]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in neural information processing systems , vol. 30. Curran Associates, Inc., 2017

    work page 2017

  2. [2]

    GPT-4 Technical Report,

    OpenAI, “GPT-4 Technical Report,” 2024. [Online]. Available: https://arxiv.org/abs/2303.08774

    arXiv 2024

  3. [3]

    Basic Linear Algebra Subprograms for Fortran Usage,

    C. L. Lawson, R. J. Hanson, D. R. Kincaid, and F. T. Krogh, “Basic Linear Algebra Subprograms for Fortran Usage,” ACM Trans. Math. Softw., vol. 5, no. 3, p. 308–323, Sep. 1979. [Online]. Available: https://doi.org/10.1145/355841.355847

    arXiv 1979

  4. [4]

    Evaluating Large Language Models Trained on Code,

    Chen, M. et al., “Evaluating Large Language Models Trained on Code,” 2021. [Online]. Available: https://arxiv.org/abs/2107.03374

    arXiv 2021

  5. [5]

    CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis,

    E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y . Zhou, S. Savarese, and C. Xiong, “CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis,” 2023. [Online]. Available: https://arxiv.org/abs/2203.13474

    arXiv 2023

  6. [6]

    StarCoder: may the source be with you!

    Li, R. et al., “StarCoder: may the source be with you!” 2023. [Online]. Available: https://arxiv.org/abs/2305.06161

    arXiv 2023

  7. [7]

    CodeRAG: Supportive Code Retrieval on Bigraph for Real-World Code Generation,

    J. Li, X. Shi, K. Zhang, L. Li, G. Li, Z. Tao, J. Li, F. Liu, C. Tao, and Z. Jin, “CodeRAG: Supportive Code Retrieval on Bigraph for Real-World Code Generation,” 2025. [Online]. Available: https://arxiv.org/abs/2504.10046

    arXiv 2025

  8. [8]

    LLM-Assisted Translation of Legacy FORTRAN Codes to C++: A Cross-Platform Study,

    N. R. Ranasinghe, S. M. Jones, M. Kucer, A. Biswas, D. O’Malley, A. Most, S. L. Wanna, and A. Sreekumar, “LLM-Assisted Translation of Legacy FORTRAN Codes to C++: A Cross-Platform Study,” in Proceedings of the 1st Workshop on AI and Scientific Discovery: Directions and Opportunities . Albuquerque, New Mexico, USA: TABLE 5: Performance of Level-2 routines ...

    work page

Show all 24 references
  1. [9]

    Enhancing Code Translation in Language Models with Few-Shot Learning via Retrieval-Augmented Generation,

    M. Bhattarai, J. E. Santos, S. Jones, A. Biswas, B. Alexandrov, and D. O’Malley, “Enhancing Code Translation in Language Models with Few-Shot Learning via Retrieval-Augmented Generation,” 2024. [Online]. Available: https://arxiv.org/abs/2407.19619

    2024 arXiv

  2. [10]

    MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoning,

    K. Wang, H. Ren, A. Zhou, Z. Lu, S. Luo, W. Shi, R. Zhang, L. Song, M. Zhan, and H. Li, “MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoning,” in The Twelfth International Conference on Learning Representations, 2024. [Online]. Available: https://o...

    2024

  3. [11]

    HPC-GPT: Integrating Large Language Model for High-Performance Computing,

    X. Ding, L. Chen, M. Emani, C. Liao, P.-H. Lin, T. Vanderbruggen, Z. Xie, A. Cerpa, and W. Du, “HPC-GPT: Integrating Large Language Model for High-Performance Computing,” in Proceedings of the SC ’23 Workshops of the International Conference on High Performance Computing, Netw...

    2023

  4. [12]

    HPC-Coder: Modeling Parallel Programs using Large Language Models,

    D. Nichols, A. Marathe, H. Menon, T. Gamblin, and A. Bhatele, “HPC-Coder: Modeling Parallel Programs using Large Language Models,” in ISC High Performance 2024 Research Paper Proceedings (39th International Conference). IEEE, May 2024, p. 1–12. [Online]. Available: http://dx.d...

    2024

  5. [13]

    Can Large Language Models Write Parallel Code?

    D. Nichols, J. H. Davis, Z. Xie, A. Rajaram, and A. Bhatele, “Can Large Language Models Write Parallel Code?” in Proceedings of the 33rd International Symposium on High-Performance Parallel and Distributed Computing, ser. HPDC ’24. Association for Computing Machinery, 2024, p. 281–294

    2024

  6. [14]

    LM4HPC: Towards Effective Language Model Ap- plication in High-Performance Computing,

    L. Chen, P.-H. Lin, T. Vanderbruggen, C. Liao, M. Emani, and B. de Supinski, “LM4HPC: Towards Effective Language Model Ap- plication in High-Performance Computing,” in OpenMP: Advanced Task-Based, Device and Compiler Programming: 19th International Workshop on OpenMP , IWOMP 2...

    2023

  7. [15]

    MARCO: A Multi-Agent System for Optimizing HPC Code Generation Using Large Language Models,

    A. Rahman, V . Cvetkovic, K. Reece, A. Walters, Y . Hassan, A. Tummeti, B. Torres, D. Cooney, M. Ellis, and D. S. Nikolopoulos, “MARCO: A Multi-Agent System for Optimizing HPC Code Generation Using Large Language Models,” 2025. [Online]. Available: https://arxiv.org/abs/2505.03906

    2025 arXiv

  8. [16]

    ARCS: Agentic Retrieval-Augmented Code Synthesis with Iterative Refine- ment,

    M. Bhattarai, M. Cordova, J. Santos, and D. O’Malley, “ARCS: Agentic Retrieval-Augmented Code Synthesis with Iterative Refine- ment,” 2025. [Online]. Available: https://arxiv.org/abs/2504.20434

    2025

  9. [17]

    MonoCoder: Domain-Specific Code Language Model for HPC Codes and Tasks,

    T. Kadosh, N. Hasabnis, V . A. V o, N. Schneider, N. Krien, M. Capota, A. Wasay, N. Ahmed, T. Willke, G. Tamir, Y . Pinter, T. Mattson, and G. Oren, “MonoCoder: Domain-Specific Code Language Model for HPC Codes and Tasks,” 2024. [Online]. Available: https://arxiv.org/abs/2312.13322

    2024 arXiv

  10. [18]

    QiMeng-GEMM: Automatically Generating High-Performance Matrix Multiplication Code by Exploiting Large Language Models,

    Q. Zhou, Y . Wen, R. Chen, K. Gao, W. Xiong, L. Li, Q. Guo, Y . Wu, and Y . Chen, “QiMeng-GEMM: Automatically Generating High-Performance Matrix Multiplication Code by Exploiting Large Language Models,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no...

    2025

  11. [19]

    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning,

    DeepSeek-AI, “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning,” 2025. [Online]. Available: https://arxiv.org/abs/2501.12948

    2025 arXiv

  12. [20]

    ChatBLAS: The First AI-Generated and Portable BLAS Library,

    P. Valero-Lara, W. F. Godoy, K. Teranishi, P. Balaprakash, and J. S. Vetter, “ChatBLAS: The First AI-Generated and Portable BLAS Library,” in SC24-W: Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis , 2024, pp. 19–24

    2024

  13. [21]

    Comparing Llama-2 and GPT-3 LLMs for HPC kernels generation,

    P. Valero-Lara, A. Huante, M. A. Lail, W. F. Godoy, K. Teranishi, P. Balaprakash, and J. S. Vetter, “Comparing Llama-2 and GPT-3 LLMs for HPC kernels generation,” 2023. [Online]. Available: https://arxiv.org/abs/2309.07103

    2023 arXiv

  14. [22]

    Evaluation of OpenAI Codex for HPC Parallel Programming Models Kernel Generation,

    W. Godoy, P. Valero-Lara, K. Teranishi, P. Balaprakash, and J. Vetter, “Evaluation of OpenAI Codex for HPC Parallel Programming Models Kernel Generation,” in Proceedings of the 52nd International Con- ference on Parallel Processing Workshops, ser. ICPP Workshops ’23. New York,...

    2023

  15. [23]

    BLAS++: C++ API for the Basic Linear Algebra Subroutines,

    Innovative Computing Laboratory, University of Tennessee, “BLAS++: C++ API for the Basic Linear Algebra Subroutines,”

  16. [2023]

    Available: https://github.com/icl-utk-edu/blaspp/ TABLE 6: Performance of Level-3 routines (in GFlops/s, problem size: n = 2048)

    [Online]. Available: https://github.com/icl-utk-edu/blaspp/ TABLE 6: Performance of Level-3 routines (in GFlops/s, problem size: n = 2048). Performance ratio to Ref is shown in parentheses. Blank indicates that all generated codes have failed. NameToOptCcode FrtcodeToOptCcode ...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.