{"id":"9a79d7a3-a3a2-4b29-a3d8-d9b2c7f2f1ac","arxiv_id":"2505.06302","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A system that prompts LLMs to generate and autotune hardware-primitive-level GEMM and convolution kernels, outperforming OpenBLAS and cuBLAS on several platforms.","lead":"QiMeng-TensorOp is a framework that uses large language models to automatically write assembly-level tensor operators from a one-line prompt, then tunes them on the target hardware. In tests, its generated GEMM kernels beat vendor libraries such as OpenBLAS by up to 2.5x and cuBLAS by up to 1.24x on several CPUs and GPUs.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark comparability is unestablished: if generated kernels ran at reduced precision or without numerical validation while vendor baselines were configured differently, the 2.51x/1.24x speedup claims do not hold.","rationale":"The reader's weakest_assumption correctly identifies missing measurement conditions as the load-bearing issue. My stress-test agrees with that target but sharpens it in two ways. First, the A100 throughput numbers make clear that reducedprecision Tensor Core execution is necessarily involved, so the missing precision statement is not a minor omission but a prerequisite for interpreting every GPU ratio. Second, the paper never states a numerical correctness criterion for the generated assembly/CUDA kernels; 'tested on the target hardware' could mean only that the code compiles and runs. Without correctness verification, a fast-but-wrong kernel would produce meaningless speedups. The development-cost and 1291x claims are also problematic, but they are subordinate to the central claim of beating OpenBLAS and cuBLAS. The fix is straightforward and reproducible: rerun the headline comparisons with identical precision, configuration, and a tolerance-based correctness check. Because the paper's conditional acceptance already reflects these concerns, I do not move the verdict; the conditional stance remains appropriate, and the concrete test would determine whether acceptance is warranted.","tokens_in":14819,"tokens_out":4624,"duration_ms":48583,"concrete_test":"Reproduce Table 1's K1 GEMM (1024x1024x1024) and A100 (16384,1024,8192) cases. Fix and report arithmetic precision (FP32 on K1; FP16 and FP32 separately on A100), thread counts, compiler flags, CPU/GPU clock settings, library versions, and OpenBLAS/cuBLAS build configuration. Also verify each generated kernel's output against a reference GEMM with a stated tolerance (e.g., max relative error below 1e-4 for FP32 and below 1e-2 for FP16). If the 2.51x and 1.24x ratios persist under same-precision, same-configuration, correctness-verified runs, the concern is resolved; if the ratios shrink or vanish, the headline performance claims are invalid as stated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that QiMeng-TensorOp generates hardware-primitive tensor operators that beat expert-tuned libraries. This requires that the generated kernels compute the same tensor operation at the same precision as the baseline, and that both sides are measured under equivalent conditions. The paper reports neither. On the A100, the reported 260-293 TFLOPS is impossible for strict FP32 GEMM (A100 FP32 peak is roughly 19.5 TFLOPS), so the generated kernels must be using Tensor Cores in FP16/BF16/TF32 mode; however, the cuBLAS baseline precision is never stated. If cuBLAS ran FP32 or a different math mode, the '124% of cuBLAS' ratio is not an apples-to-apples comparison. On CPUs, thread counts, compiler flags, CPU frequency scaling, and the OpenBLAS build configuration (e.g., whether RVV extensions were enabled, single-threaded vs multithreaded) are all missing. Additionally, Section 3.2 says kernels are 'jointly compiled and tested on the target hardware to obtain feedback,' but no numerical correctness criterion is specified; a kernel that terminates and runs fast but returns incorrect results would pass this test. The 1291x and 200x cost claims are secondary; the load-bearing claim is speedup over vendor libraries, and without precision and configuration disclosure it is not verifiable from the paper alone.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes QiMeng-TensorOp, an LLM-based framework that generates hardware-primitive-level tensor operators (CPU assembly, CUDA Tensor Core kernels via CuTe/PTX) from a one-line user prompt. The framework has three stages: hardware architecture comprehension (optimization hints plus hardware factor extraction), tensor operator generation (C/CUDA sketch plus Python-script-generated assembly/CuTe kernels), and LLM-assisted MCTS auto-tuning. The authors evaluate on RISC-V, ARM, and NVIDIA GPUs with GPT-4o, DeepSeek-V3, Claude 3.5 Sonnet, and Llama-3.1-405B, comparing against vanilla prompts, CoT prompts, OpenBLAS, ACL, cuBLAS, cuDNN, and TVM. Headline claims are up to 1291× over vanilla prompting, 2.51× over OpenBLAS on RISC-V, 1.24× over cuBLAS on A100, and a 200× development-cost reduction relative to human engineers.","tokens_in":15074,"tokens_out":2516,"duration_ms":26139,"significance":"If the performance claims are correct, this would be a notable advance: it would demonstrate that an LLM with structured hints, few-shot examples, and MCTS-guided tuning can produce assembly-level or Tensor-Core-level kernels that match or exceed expert-tuned vendor libraries across diverse hardware, while dramatically reducing development cost. The framework is well-motivated, the system design is coherent, and the ablation studies (prompt types, component removal, search history) are a genuine strength. The paper also evaluates on multiple hardware platforms and multiple LLMs, which supports generality. However, the significance cannot be fully assessed because the central evaluation lacks the measurement precision, baseline configuration, and correctness criteria needed to verify the speedup ratios. The reported A100 numbers imply Tensor-Core reduced-precision execution, yet the precision and math mode of both the generated kernels and the cuBLAS baselines are never stated; this is load-bearing for the paper's core claim of outperforming vendor libraries.","major_comments":[{"comment":"The A100 rows report 260–293 TFLOPS for QiMeng-TensorOp. The A100 FP32 non-tensor peak is roughly 19.5 TFLOPS, so these kernels must be executing in a Tensor-Core reduced-precision mode (FP16, BF16, or TF32). The paper never states the arithmetic precision or CUDA math mode used by either the generated kernels or the cuBLAS baseline. Without this information, the claim of '124% of cuBLAS' is not an apples-to-apples comparison, and the central speedup claim over cuBLAS is unverified.","section":"Section 4.2, Table 1 (A100 rows)"},{"comment":"For the CPU evaluations, the paper does not report thread counts, compiler flags, CPU frequency scaling or governor settings, or the exact build/configuration of OpenBLAS, ACL, and TVM (e.g., whether RVV/NEON extensions were enabled, single-threaded versus multithreaded BLAS builds). Since the claimed 2.51× over OpenBLAS on K1 and 1.21× over ACL on A72 are head-to-head ratios, the absence of these configuration details makes the CPU speedups impossible to reproduce or fully validate from the manuscript.","section":"Section 4.1 and Table 1 (CPU comparisons)"},{"comment":"The method section states that generated sketches and kernels are 'jointly compiled and tested on the target hardware to obtain feedback' and that the MCTS simulation 'tests the tensor operator ... to obtain an initial performance value,' but no numerical correctness criterion is specified anywhere. A kernel that compiles, terminates, and runs quickly while returning incorrect results would pass the described test. The paper should state the reference implementation and error tolerance (e.g., maximum relative error against OpenBLAS/cuBLAS) used to accept or reject each generated operator, since the performance claims presume the generated kernels compute the target operation correctly.","section":"Sections 3.2 and 3.3 (correctness criterion)"}],"minor_comments":[{"comment":"The Conv results are presented only as a figure with no numeric table, and the figure content is not described in the text; adding a table or explicit numeric values would make the Conv claims checkable.","section":"Section 4.2, Figure 3"},{"comment":"The text repeatedly refers to Appendices A, B, and C (additional results, MCTS pseudo-code, and a detailed showcase), but these appendices are not included in the submitted manuscript, so the reader cannot inspect the tuning algorithm or the claimed example generation.","section":"Appendices"},{"comment":"There are several typos and wording issues, e.g., 'a detaled' in the Section 3 overview caption, 'tensor operates' in the ablation discussion, and inconsistent use of 'vanilla prompt' versus 'vanilla LLMs' in the abstract; these should be cleaned up during revision.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The core idea is promising and the framework is reasonably designed, but the evaluation section does not currently establish the central claim of beating vendor libraries because precision, baseline configuration, thread/count and compiler settings, and correctness tolerance are all undisclosed. This is a fixable but load-bearing gap: the authors should be asked to provide a full experimental appendix with these details and to rerun or clarify the precision-matched comparisons. I would not recommend acceptance until that evidence is available."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is a genuine engineering contribution: it's the first end-to-end pipeline I've seen that uses general LLMs to produce assembly-level tensor kernels, by having the LLM write Python scripts that emit assembly, guided by extracted hardware factors and an LLM-in-the-loop MCTS tuner. Second, the headline speedups over cuBLAS and OpenBLAS are not yet trustworthy, because the evaluation never reports the math precision, thread counts, compiler flags, or library versions. On the A100 the numbers (260–293 TFLOPS) are only physically reachable with Tensor Cores at reduced precision, so the cuBLAS baseline must be run in the same mode for the 1.24x claim to mean anything. The paper doesn't say it was.\n\nWhat's good: the component design is sensible, the ablations show each piece (sketch generation, primitive kernels, MCTS) contributes, and the evaluations span RISC-V, ARM, and NVIDIA with multiple LLMs. The development-cost reduction (200x) is anecdotal but plausible in direction. The citation pattern looks fine, with prior work like TLP, Exo, and Ansor acknowledged.\n\nSoft spots, in proportion: the missing measurement detail is load-bearing, not cosmetic. Also, Section 3.2 says kernels are 'jointly compiled and tested' but no numerical correctness criterion is given; a kernel that runs fast but returns wrong results would pass. The 1291x versus vanilla prompt is a real number but the baseline is deliberately naive, so it doesn't belong in the abstract as a headline. And the hardware hints are hand-written by the authors, so the framework still depends on expert knowledge at bootstrap, though presumably that's a one-time cost per architecture.\n\nBottom line: the idea deserves referee time and could easily be a solid systems paper after revision. The authors need to disclose precision/configuration, add a correctness check, and ideally release code and data. As is, I'd treat the results as promising but not verified. Bring it to the reading group if you work on ML systems or codegen; otherwise it's a maybe.","headline":"A promising LLM-based pipeline for assembly-level tensor operators, with benchmark claims that currently rest on unstated precision and configuration details.","tokens_in":15690,"tokens_out":2498,"would_cite":true,"duration_ms":24017,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that an LLM steered by hardware hints and LLM-guided tree search can generate assembly-level tensor operators from a one-line prompt, outperforming expert-tuned vendor libraries on several platforms.","keywords":["tensor operator generation","hardware primitives","LLM code generation","Monte Carlo tree search auto-tuning","RISC-V vector extension","CUDA Tensor Core","GEMM","convolution"],"falsifier":"Inspect the generated A100 kernel and the cuBLAS baseline configuration: reported throughput near 293 TFLOPS implies Tensor Core reduced-precision arithmetic, so if the cuBLAS row was measured at a different precision or with different tensor-core settings, the 1.24× ratio is not a like-for-like comparison. Re-running both at the same precision and recording the configuration would settle the claim.","tokens_in":14592,"feed_emoji":"⚡","tokens_out":7419,"duration_ms":67287,"temperature":0.7,"pith_summary":"QiMeng-TensorOp is a framework for turning a single user sentence—\"generate a 1024×1024×1024 GEMM for the RISC-V C910\"—into a kernel built from hardware primitives such as RISC-V vector instructions, ARM NEON, or CUDA Tensor Core intrinsics. The paper argues that generic optimization hints plus hardware factors extracted from manuals are enough to make an LLM write the low-level code it cannot normally produce, and that an LLM-guided Monte Carlo tree search then tunes the kernel to or beyond the performance of expert-tuned libraries. On the tested platforms the generated kernels reach up to 251% of OpenBLAS on RISC-V CPUs and 124% of cuBLAS on NVIDIA GPUs, while beating TVM and cutting development time by up to 200×. The load-bearing comparison is raw throughput against vendor baselines, so the speedup claims stand or fall on whether the measurements were made under identical precision and configuration.","feed_headline":"One-line prompts beat vendor tensor libraries","feed_subtitle":"LLM-guided framework writes assembly-level kernels, hitting 251% of OpenBLAS and 124% of cuBLAS.","key_machinery":"The machinery is a three-stage pipeline. First, Hardware Architecture Comprehending injects a generic description of five optimization techniques (tiling, reordering, vectorization, layout, pipeline) and their relation to four hardware factors (memory hierarchy, instructions, registers, streaming processors), then retrieves the target chip's factors from its manual. Second, Tensor Operator Generation has the LLM write a C or CUDA C++ sketch that handles tiling, reordering, and layout, and then—because LLMs write Python more reliably than assembly—write a Python script that emits the PACK and COMPUTE microkernels as assembly or PTX-level code. Third, Auto-Tuning runs an LLM-assisted Monte Carlo tree search over sketch parameters and instruction order, using the search history as in-context context for choosing the next tuning action. The load-bearing object is the Python-script bridge: it converts the LLM's strength at high-level code into correct low-level kernels.","core_discovery":"The paper's central claim is that a modern LLM, given only a one-line prompt describing an operator and a target processor, can be prompted into producing kernels written in the machine's own hardware primitives—RISC-V vector assembly, ARM NEON, or CUDA Tensor Core code—and that those kernels can be tuned automatically to match or beat hand-optimized vendor libraries. The reported evidence is kernel throughput on GEMM and convolution across RISC-V, ARM, and NVIDIA platforms: up to 2.51× OpenBLAS on RISC-V, up to 1.21× ACL on ARM, up to 1.24× cuBLAS and 3.89× cuDNN on NVIDIA GPUs, and up to 1.38× (GEMM) and 2.43× (Conv) over TVM. The paper also reports that the same framework outperforms a vanilla LLM prompt by up to 1291× and cuts the development time of a senior engineer from several days to under twenty minutes. The reason this matters is that manual or compiler-based optimization of tensor operators is the bottleneck to deploying neural workloads on new hardware, and the framework proposes to replace months of expert tuning with a prompt and a search loop.","pith_inferences":["Beyond the paper: the technique of asking the LLM to write a Python script that emits assembly, rather than writing assembly directly, is a general pattern—LLMs may be more reliable as code-generator generators than as low-level coders, and the same bridge could apply to other instruction sets.","Beyond the paper: if the speedups survive controlled benchmarking, the main practical consequence is that compiler backends and handwritten microkernels become optional: a prompt plus a manual and a search loop could substitute for much of the engineering in deployment on a new processor.","Beyond the paper: a direct test of generality would be applying the three-step pipeline to non-GEMM kernels such as attention, softmax, or layer norm under the same one-line prompts; the paper states this as future work but does not report results."],"forward_implications":["The pipeline replaces months of manual assembly tuning: a senior engineer's several-day effort for one GEMM is compressed to under twenty minutes per platform, with up to 200× cost reduction reported.","The benefit is largest where vendor libraries are weakest: RISC-V GEMMs reach 2.51× OpenBLAS, while mature ARM and NVIDIA libraries are merely matched or slightly exceeded (1.02–1.21× ACL, 0.98–1.24× cuBLAS).","Because the same prompts and pipeline cover GEMM and Conv on CPUs and GPUs, the method is operator- and platform-generic rather than a single handcrafted kernel.","The auto-tuning component improves even hand-written OpenBLAS C code, indicating that the LLM-guided MCTS search is a separable optimization layer, not just a code-generation trick."],"supporting_citations":[{"why":"Provides the proprietary LLM whose in-context abilities the framework steers.","marker":"[OpenAI, 2025]"},{"why":"Provides the open-source LLM used to show the pipeline works across model families.","marker":"[DeepSeek-AI et al., 2024]"},{"why":"OpenBLAS, the manually optimized CPU library that serves as the main RISC-V/ARM baseline to beat.","marker":"[Xianyi et al., 2012]"},{"why":"cuBLAS, the vendor GPU library that serves as the main NVIDIA baseline.","marker":"[NVIDIA, 2023]"},{"why":"TVM, the auto-compiler baseline compared on ARM and NVIDIA, and the dominant alternative paradigm.","marker":"[Chen et al., 2018a]"},{"why":"Documents NVIDIA Tensor Core programmability, the hardware primitive the GPU kernels target.","marker":"[Markidis et al., 2018]"},{"why":"Describes the RISC-V C910 processor whose vector extension and cache hierarchy are extracted as hardware factors.","marker":"[Chen et al., 2020a]"},{"why":"Describes A100 SM and Tensor Core organization used to shape CUDA grid and block allocation.","marker":"[Choquette et al., 2021]"}],"fun_headline_variants":["One-line prompt writes kernels that beat cuBLAS and OpenBLAS","LLM auto-generates assembly kernels, 251% of OpenBLAS","AI kernel generator beats vendor libs, 1291x faster than vanilla","Prompt-to-kernel: auto-tuning beats hand-optimized libraries","One prompt, any hardware: LLM kernels exceed expert hand-tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The speedup claims assume the generated kernels and the vendor libraries were benchmarked under identical precision, thread counts, compiler flags, and clock settings, since Table 1 reports only raw GFLOPS/TFLOPS.","fun_headline_variants_meta":{"raw":{"variants":["One-line prompt writes kernels that beat cuBLAS and OpenBLAS","LLM auto-generates assembly kernels, 251% of OpenBLAS","AI kernel generator beats vendor libs, 1291x faster than vanilla","Prompt-to-kernel: auto-tuning beats hand-optimized libraries","One prompt, any hardware: LLM kernels exceed expert hand-tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000293,"raw_usage":{"total_tokens":1762,"prompt_tokens":1056,"completion_tokens":706,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":672,"completion_tokens_details":{"reasoning_tokens":610}},"tokens_in":672,"tokens_out":706,"duration_ms":6187,"temperature":1.0,"reasoning_tokens":610,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:18:10.358975+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Inspect the generated A100 kernel and the cuBLAS baseline configuration: reported throughput near 293 TFLOPS implies Tensor Core reduced-precision arithmetic, so if the cuBLAS row was measured at a different precision or with different tensor-core settings, the 1.24× ratio is not a like-for-like comparison. Re-running both at the same precision and recording the configuration would settle the claim.","supporting_citations":[{"cited_title":"DeepSeek-V3 Technical Report","cited_arxiv_id":null,"evidence_quote":"Provides the open-source LLM used to show the pipeline works across model families."},{"cited_title":"Model-driven level 3 BLAS performance opti- mization on Loongson 3A processor","cited_arxiv_id":null,"evidence_quote":"OpenBLAS, the manually optimized CPU library that serves as the main RISC-V/ARM baseline to beat."},{"cited_title":"CUBLAS LIBRARY user guide v12.1","cited_arxiv_id":null,"evidence_quote":"cuBLAS, the vendor GPU library that serves as the main NVIDIA baseline."},{"cited_title":"NVIDIA Tensor Core Programmabil- ity, Performance & Precision","cited_arxiv_id":null,"evidence_quote":"Documents NVIDIA Tensor Core programmability, the hardware primitive the GPU kernels target."},{"cited_title":"Nvidia A100 Tensor Core GPU: Performance and innovation","cited_arxiv_id":null,"evidence_quote":"Describes A100 SM and Tensor Core organization used to shape CUDA grid and block allocation."}],"review_version":1}