Pith. sign in

REVIEW 3 major objections 5 minor 29 references

QiMeng-Attention: SOTA Attention Operator is generated by SOTA Attention Algorithm

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLM-generated FlashAttention kernels outrun hand-tuned libraries

desk verdict A genuinely new IR for LLM kernel generation, but the missing correctness checks and artifacts make the central performance claim currently unsupportable. read the letter →

arxiv 2506.12355 v1 pith:EJZZ76SZ submitted 2025-06-14 cs.LG cs.CL

Qirui Zhou , Shaohui Peng , Weiqiang Xiong , Haixin Chen , Yuanbo Wen , Haochen Li , Ling Li , Qi Guo
show 6 more authors
Yongwei Zhao Ke Gao Ruizhi Chen Yanjun Wu Chen Zhao Yunji Chen
This is my paper · ORCID
classification cs.LGcs.CL
keywords LLMcodegenerationFlashAttentionGPUkerneloptimizationTensorCoresCuTeoperatorfusionoperatorsLLM-TL
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that a large language model, guided by a small hand-designed 'thinking language,' can write GPU attention kernels of the FlashAttention type—tiled, fused, memory-efficient attention—that compete with or beat kernels hand-coded by human experts. It claims that the key obstacle is not the LLM's coding ability but its lack of a way to express data movement and computation at the right abstraction level. The proposed solution, LLM-TL, reduces attention kernels to Copy and Compute statements, then translates that sketch into low-level CuTe code. The reported result is automatic FlashAttention implementations across A100, RTX8000, T4, and L40S GPUs, with up to 35.16x speedup over vanilla LLM-generated code, performance at or above cuDNN and FlashAttention in most scenarios, and development time cut from months to minutes.

What carries the argument

The load-bearing mechanism is LLM-TL, an abstraction language with exactly two statement types: Copy describes moving a tensor between global memory, shared memory, and registers with a block size and coordinate; Compute describes a GPU operation such as GEMM, softmax, or scaling. A Reshape statement allows the output of one tensor-core GEMM to be reused as input to a second fused GEMM, which is the step LLMs omit when generating directly. The two-stage workflow first generates a coarse TL Sketch, then a parameterized TL Code, and finally adaptively translates it to CuTe primitives (copy atoms and MMA atoms) for the target GPU. CuTe is the high-level template library that encapsulates tensor-core PTX instructions, so translation becomes pattern-matching rather than index arithmetic.

What would settle it

Rerun the MHA-with-causal-mask benchmark on an A100 at sequence length 16k with the prompts from Appendix D and compare the generated kernel's output against a reference fp32 attention implementation; if its maximum error exceeds typical tolerance (say 1e-2) or its TFLOPS is substantially below 184.3, the central performance claim is refuted.

Watch

Extended reading notes

Core claim

The central claim is that current LLMs fail to generate high-performance attention kernels because they must simultaneously reason about algorithm-level optimization logic and hardware-level implementation details. The paper's discovery is that decoupling these two levels via LLM-TL makes the problem tractable for an off-the-shelf LLM: the model first produces a TL Sketch describing the fused data flow (Copy Q/K/V between memory levels, Compute GEMM/softmax/GEMM), then fills in block sizes, offsets, and reshape statements, and finally translates the TL code into CuTe. On the tested GPUs and attention variants (MHA, GQA, MQA, MLA), the generated kernels match or exceed hand-optimized libraries in most configurations, including workloads the official libraries do not support, such as RTX8000 and FP8.

Load-bearing premise

The central claim assumes the generated kernels are numerically correct and that the reported TFLOPS numbers are measured under fair, reproducible conditions; the paper gives no accuracy checks, no released code, and no benchmark scripts.

Editorial extensions

If this is right

  • If the central claim holds, LLMs can produce production-quality attention kernels for GPUs that official libraries skip, such as RTX8000 and FP8 data types.
  • Attention operator development cost drops from months to roughly ten minutes per kernel, since the same TL code translates across architectures.
  • The generated kernels exceed cuDNN on MLA by 2.15x and match or outperform FlashAttention and cuDNN on MHA, GQA, and MQA in most tested configurations.
  • The two-stage design is essential: when LLMs are asked to emit TL code directly, they omit the required reshape between fused GEMMs and mis-handle transposed layouts.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same Copy/Compute abstraction should transfer to other fused operators, such as QKV projection, MoE routing, or KV-cache paged attention, because the language is not attention-specific.
  • The 35.16x speedup over vanilla LLMs mostly reflects how weak the vanilla PyTorch baseline is; the more informative comparison is against cuDNN and FlashAttention, where the generated kernels are comparable or slightly better in most cases.
  • A natural testable extension would be applying the workflow to a new operator family and checking whether the translation stage remains error-free without hand-written prompts for each architecture.
  • Because the paper does not report numerical accuracy checks or release the generated kernels and benchmark scripts, an independent replication would need to re-generate the kernels from the stated prompts and compare outputs against a reference attention implementation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes LLM-TL, a two-statement abstract language (Copy and Compute) intended to help LLMs generate high-performance FlashAttention kernels. The workflow first asks an LLM to produce a TL Sketch describing the operator's data flow, then to fill in parameters (block sizes, memory coordinates, reshape operations) to obtain TL Code, and finally to translate the TL Code into CuTe-based CUDA kernels. The authors evaluate the approach with four LLMs (GPT-4o, Claude 3.5, DeepSeek-V3, DeepSeek-R1) on A100, RTX8000, T4, and L40S GPUs across MHA, GQA, MQA, and MLA variants, reporting TFLOPS speedups up to 35.16× over vanilla LLM-generated PyTorch code and claiming that the generated kernels match or exceed cuDNN, flash-attn, and FlexAttention in most scenarios. The paper also reports a development-time comparison with a human expert and ablation results on prompt design and LLM backends.

Significance. If the claims hold, the idea of an intermediate, LLM-friendly abstract language that decouples high-level optimization logic from low-level GPU implementation is a useful contribution to automated high-performance kernel generation, with potential generalization beyond attention. The evaluation is broad in terms of GPU generations, LLMs, and attention variants, and the reported TFLOPS tables are extensive. However, the central claim that the generated kernels are correct and fast is currently unverified: the manuscript contains no numerical accuracy checks, releases no code or benchmark scripts, and the performance comparison lacks a fully specified measurement protocol. The stress-test concern lands: without a correctness check, fast-but-wrong kernels would make all speedup numbers uninterpretable. The paper does not ship machine-checked proofs, reproducible code, or parameter-free derivations; it relies entirely on empirical performance claims that need stronger evidence.

major comments (3)
  1. [§4, Tables 1–2, 5–6] The evaluation reports only TFLOPS and provides no correctness validation of the generated kernels against a reference attention output. Appendix B itself documents that the same pipeline produces kernels with 'reshape omission' and 'GEMM layout error' that lead to 'computation errors', and Table 3 states that GPT-4o fails to translate correct CuTe code. Because the central claim is that LLM-TL generates kernels that are both correct and fast, the absence of per-configuration numerical checks (e.g., max absolute error or cosine similarity against a PyTorch reference with tolerances appropriate for FP16/FP8) is load-bearing: the reported speedups may be for kernels that compute an incorrect function. A correctness appendix and, preferably, a released validation script are required before the performance claims can be interpreted.
  2. [§4.1 Benchmark setting] The benchmark section does not specify the measurement protocol: number of runs, warmup iterations, kernel synchronization, GPU clock/power management, or the compiler and driver versions. It also does not state that baseline libraries (cuDNN, flash-attn, FlexAttention) were invoked with the same shapes, dtypes, masks, and scale factors, or that the baselines were run on the same machines by the authors. The 'vanilla LLM' baseline in Table 1 is a PyTorch implementation, so the headline 35.16× speedup is against an unoptimized implementation rather than against a state-of-the-art kernel. For the claim of matching or exceeding hand-optimized libraries, the comparison must be made with a common, described benchmarking harness; the current text and the absence of released benchmark scripts make the TFLOPS tables non-reproducible.
  3. [§4.3, Table 4] Table 4 reports 'Human Expert ∼months, 162.7 TFLOPS' versus 'LLM-TL 10 mins, 175.6 TFLOPS' as evidence that development time is reduced from months to minutes. This is an anecdotal comparison: the task definition, the expert's background, the optimization effort spent, and the measurement conditions are not specified, and the '∼months' figure is not tied to any recorded process. As presented, this comparison cannot support the quantitative reduction claim in the abstract and should either be removed or substantiated with a protocol-based user study.
minor comments (5)
  1. [Table 1, RTX8000 block] The no-causal-mask block for RTX8000 contains a row labeled 'flash-attn v2', which contradicts §4.1's statement that FlashAttention v2 is not available on Turing architecture and that v1.0.9 is used; the label should be 'flash-attn v1'.
  2. [Abstract and §1] The abstract and introduction say verification is on A100, RTX8000, and T4, but Table 6 reports results on L40S; please update the claimed hardware coverage.
  3. [§1 and Appendix A] There are unresolved citation placeholders ('?') in the code-generation related references in Section 1 and in 'NSA(?)' in Appendix A; these should be completed or removed.
  4. [Throughout] There are several typographical issues, including 'librariy' in §2.1.2, 'Programme' in §3.2.1, 'casual' instead of 'causal' in table headings, and 'Ackowledgement' in §6.
  5. [Table 2] The row labeled 'DeepSeek-V3' should be clearly identified as vanilla LLM-generated PyTorch code, as in Table 1, to avoid confusion with the official DeepSeek-V3 implementation of MLA.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claim is an empirical benchmark result against external libraries, not a derivation from its own assumptions.

full rationale

The paper makes no mathematical derivation that could reduce to its own inputs. Its central claim is that an LLM, guided by the proposed LLM-TL abstraction and a two-stage workflow, can generate FlashAttention implementations whose measured TFLOPS match or beat external libraries. That claim is evaluated against independent baselines (cuDNN, flash-attn, FlexAttention, and CoT-based LLM code) on external hardware, so the outcome is not fixed by construction. LLM-TL is a prompt-language and workflow design, not a fitted model: no parameter is calibrated to the benchmark results, and no reported quantity is defined in terms of the method's own outputs. The paper explicitly acknowledges limitations, including the absence of H100 testing and the restriction to attention operators, and Appendix B documents concrete failure modes that the workflow is designed to mitigate; these are correctness and scope concerns, not circular reasoning. The absence of numerical correctness checks and released artifacts is a serious validity risk for the speedup claims, but it is an empirical verification gap rather than a case of a prediction being equivalent to its input by definition. Because no load-bearing step relies on an unverified self-citation, a uniqueness theorem imported from the authors' prior work, or a fitted input renamed as a prediction, the appropriate circularity score is 0.

Assumptions & free parameters 1 free parameters · 2 assumptions · 1 invented entities

The central claim rests on the effectiveness of the LLM-TL abstraction and the correctness of the benchmark methodology. The only free parameters are the per-run block sizes and layout choices made by the LLM during code generation, which are part of the method rather than fitted constants.

free parameters (1)
  • Generated kernel tile sizes and layout parameters (BM, BN, reshape decisions) = Chosen by LLM during TL code generation
    The LLM selects block sizes and shapes while filling in TL code; performance depends on these choices, but the paper treats them as part of the method's reasoning output rather than fitting them to the benchmark data.
assumptions (2)
  • domain assumption CuTe provides a sufficiently high-level, correct abstraction over Tensor Core PTX instructions so that generated CuTe code can achieve near-optimal performance without manual tuning.
    The translation stage assumes CuTe APIs map correctly to hardware on every target GPU (A100, RTX8000, T4, L40S); if this fails, the translation step cannot deliver performance.
  • domain assumption The benchmark methodology (FLOPs = 4*seqlen^2*head_dim*n_heads, sequence length configurations, batch adjustments) is a fair basis for comparing TFLOPS across baselines.
    Performance claims depend on all baselines using the same FLOPs definition and workload settings; any inconsistency would distort speedup numbers.
invented entities (1)
  • LLM-TL (Thinking Language) independent evidence
    purpose: Intermediate representation with Copy and Compute statements to let LLMs reason about GPU data movement and computation.
    The language is defined in the paper and validated via benchmarks, providing a falsifiable performance handle, though no separate artifact is released.

how reviews work

0 comments
Cite this review

Pith. "Pith review of QiMeng-Attention: SOTA Attention Operator is generated by SOTA Attention Algorithm." pith.science (2026). https://pith.science/paper/EJZZ76SZ

@misc{pith2026250612355,
  author       = {Pith},
  title        = {Pith review of: QiMeng-Attention: SOTA Attention Operator is generated by SOTA Attention Algorithm},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EJZZ76SZ}},
  note         = {Machine review of arXiv:2506.12355}
}
read the original abstract

The attention operator remains a critical performance bottleneck in large language models (LLMs), particularly for long-context scenarios. While FlashAttention is the most widely used and effective GPU-aware acceleration algorithm, it must require time-consuming and hardware-specific manual implementation, limiting adaptability across GPU architectures. Existing LLMs have shown a lot of promise in code generation tasks, but struggle to generate high-performance attention code. The key challenge is it cannot comprehend the complex data flow and computation process of the attention operator and utilize low-level primitive to exploit GPU performance. To address the above challenge, we propose an LLM-friendly Thinking Language (LLM-TL) to help LLMs decouple the generation of high-level optimization logic and low-level implementation on GPU, and enhance LLMs' understanding of attention operator. Along with a 2-stage reasoning workflow, TL-Code generation and translation, the LLMs can automatically generate FlashAttention implementation on diverse GPUs, establishing a self-optimizing paradigm for generating high-performance attention operators in attention-centric algorithms. Verified on A100, RTX8000, and T4 GPUs, the performance of our methods significantly outshines that of vanilla LLMs, achieving a speed-up of up to 35.16x. Besides, our method not only surpasses human-optimized libraries (cuDNN and official library) in most scenarios but also extends support to unsupported hardware and data types, reducing development time from months to minutes compared with human experts.

Figures

Figures reproduced from arXiv: 2506.12355 by the authors.

Figure 1
Figure 1. Our LLM-TL enables LLMs to generate high [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Demonstration of GPU architecture. mation for effective feature extraction and integra￾tion. The input consists of sequences Q, K, V of di￾mension d and number of tokens N. The attention mechanism computes the output using the formula Attention(Q, K, V ) = Softmax(Mask( QKT √ d ))V . In addition to the early Multi-Head Attention (MHA) (Vaswani et al., 2017), recent advance￾ments have introduced Multi-Query Attention… view at source ↗
Figure 3
Figure 3. LLM-TL overview. We design the Thinking Language (TL) to help LLMs describe attention execution workflows and parameters on GPUs. Our approach consists of two stages, TL Code Generation and Translation. The TL Sketch, representing abstract semantic execution flow, is initially generated from user requirements. Subsequently, the LLM infers the parameter details of the statements within the Sketch. Finally, TL code wi… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: The content within the dashed box illustrates [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 12 canonical work pages

  1. [1]

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. 2023. https://openreview.net/forum?id=hmOwOZWzYE GQA : Training generalized multi-query transformer models from multi-head checkpoints . In The 2023 Conference on Empirical Methods in Natural Language Processing

    work page 2023

  2. [2]

    Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, Daniele Mazzotta, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. 2023. https://arxiv.org/abs/2311.16867 The falcon series of open language models . Preprint, arXiv:2...

    arXiv 2023

  3. [3]

    Ramakrishna Bairi, Atharv Sonwane, Aditya Kanade, et al. 2024. https://doi.org/10.1145/3643757 Codeplan: Repository-level coding using llms and planning . Proc. ACM Softw. Eng., 1(FSE)

    doi:10.1145/3643757 2024

  4. [4]

    Tianqi Chen, Thierry Moreau, Ziheng Jiang, et al. 2018. Tvm: An automated end-to-end optimizing compiler for deep learning. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18), pages 578--594

    work page 2018

  5. [5]

    Sharan Chetlur, Cliff Woolley, Philippe Vandermersch, Jonathan Cohen, John Tran, Bryan Catanzaro, and Evan Shelhamer. 2014. https://arxiv.org/abs/1410.0759 cudnn: Efficient primitives for deep learning . Preprint, arXiv:1410.0759

    arXiv 2014

  6. [6]

    Claude3.5. 2024. Claude 3.5 sonnet. https://www.anthropic.com/news/claude-3-5-sonnet

    work page 2024

  7. [7]

    Tri Dao. 2024. https://openreview.net/forum?id=mZn2Xyh9Ec Flashattention-2: Faster attention with better parallelism and work partitioning . In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net

    2024

  8. [8]

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher R\' e . 2022. https://proceedings.neurips.cc/paper_files/paper/2022/file/67d57c32e20fd0a7a302cb81d36e40d5-Paper-Conference.pdf Flashattention: Fast and memory-efficient exact attention with io-awareness . In Advances in Neural Information Processing Systems, volume 35, pages 16344--16359. Curran ...

    2022

Show all 29 references
  1. [9]

    DeepSeek-AI, Daya Guo, Dejian Yang, et al. 2025. https://arxiv.org/abs/2501.12948 Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning . Preprint, arXiv:2501.12948

    2025 arXiv

  2. [10]

    DeepSeek-AI, Aixin Liu, Bei Feng, et al. 2024 a . https://arxiv.org/abs/2405.04434 Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model . Preprint, arXiv:2405.04434

    2024 arXiv

  3. [11]

    DeepSeek-AI, Aixin Liu, Bei Feng, et al. 2024 b . https://arxiv.org/abs/2412.19437 Deepseek-v3 technical report . Preprint, arXiv:2412.19437

    2024 arXiv

  4. [12]

    Juechu Dong, Boyuan Feng, Driss Guessous, Yanbo Liang, and Horace He. 2024. https://arxiv.org/abs/2412.05496 Flex attention: A programming model for generating optimized attention kernels . Preprint, arXiv:2412.05496

    2024 arXiv

  5. [13]

    Samuel Holt et al. 2024. L2mac: Large language model automatic computer for extensive code generation. In The Twelfth International Conference on Learning Representations

    2024

  6. [14]

    Ke Hong, Guohao Dai, Jiaming Xu, Qiuli Mao, Xiuhong Li, Jun Liu, Kangdi Chen, Yuhan Dong, and Yu Wang. 2024. https://arxiv.org/abs/2311.01282 Flashdecoding++: Faster large language model inference on gpus . Preprint, arXiv:2311.01282

    2024 arXiv

  7. [15]

    Raymond Li, Loubna Ben Allal, Yangtian Zi, et al. 2023. https://arxiv.org/abs/2305.06161 Starcoder: may the source be with you! Preprint, arXiv:2305.06161

    2023 arXiv

  8. [16]

    Rui Li, Liyang He, Qi Liu, et al. 2024. https://api.semanticscholar.org/CorpusID:268692878 Consider: Commonalities and specialties driven multilingual code retrieval framework . In AAAI Conference on Artificial Intelligence

    2024

  9. [17]

    Llama3.1. 2024. Introducing llama 3.1: Our most capable models to date. https://ai.meta.com/blog/meta-llama-3-1/

    2024

  10. [18]

    Stefano Markidis, Der Chien, and Steven Wei et al. 2018. Nvidia tensor core programmability, performance & precision. In 2018 IEEE international parallel and distributed processing symposium workshops (IPDPSW), pages 522--531. IEEE

    2018

  11. [19]

    NVIDIA/CUTLASS. 2023. Getting started with cute. https://github.com/NVIDIA/cutlass/blob/main/media/docs/cute/00_quickstart.md

    2023

  12. [20]

    OpenAI. 2024. Gpt-4o. https://openai.com/index/hello-gpt-4o/

    2024

  13. [21]

    Adam Paszke and Gross et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32

    2019

  14. [22]

    Noam Shazeer. 2019. https://arxiv.org/abs/1911.02150 Fast transformer decoding: One write-head is all you need . Preprint, arXiv:1911.02150

    2019 arXiv

  15. [23]

    Gomez, ukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS'17, page 6000–6010. ...

    2017

  16. [24]

    Yue Wang, Hung Le, Akhilesh Deepak Gotmare, et al. 2023. https://arxiv.org/abs/2305.07922 Codet5+: Open code large language models for code understanding and generation . Preprint, arXiv:2305.07922

    2023 arXiv

  17. [25]

    An Yang, Baosong Yang, Beichen Zhang, et al. 2025. https://arxiv.org/abs/2412.15115 Qwen2.5 technical report . Preprint, arXiv:2412.15115

    2025 arXiv

  18. [26]

    Gonzalez, and Ion Stoica

    Lianmin Zheng, Chengfan Jia, Minmin Sun, Zhao Wu, Cody Hao Yu, Ameer Haj-Ali, Yida Wang, Jun Yang, Danyang Zhuo, Koushik Sen, Joseph E. Gonzalez, and Ion Stoica. 2020. https://www.usenix.org/conference/osdi20/presentation/zheng Ansor: Generating High-Performance tensor program...

    2020

  19. [27]

    Li Zhong et al. 2024. Can llm replace stack overflow? a study on robustness and reliability of large language model code generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 21841--21849

    2024

  20. [28]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  21. [29]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.