Pith. sign in

REVIEW 3 major objections 5 minor 19 references

Portable High-Performance Kernel Generation for a Computational Fluid Dynamics Code with DaCe

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read From one kernel description, DaCe generates GPU code that matches hand-tuned CUDA/HIP kernels on Nvidia hardware.

desk verdict Solid engineering case study whose cross-platform portability claim is confounded by DaCe version skew. read the letter →

arxiv 2506.20994 v1 pith:C5AH3KZE submitted 2025-06-26 cs.DC cs.PF

M{aa}ns I. Andersson , Martin Karp , Niclas Jansson , Stefano Markidis This is my paper · ORCID
classification cs.DCcs.PF
keywords PerformanceportabilitySpectralelementmethodComputationalfluiddynamicsDaCeStatefuldataflowmultigraphGPUcodegenerationNekoMatrix-freekernel
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that a single kernel description, written once in DaCe's data-centric form, can replace the hand-written CUDA and HIP kernels that CFD solvers maintain for each GPU vendor. It applies this to Neko, a spectral-element CFD solver, focusing on the small tensor operations at the heart of its matrix-free Poisson evaluation. The authors express the kernel as a Stateful Dataflow Multigraph (SDFG), apply loop-fusion, map-collapse, and shared-memory transformations, and link the generated code into Neko through a C-to-Fortran interface. They report that the generated kernel is competitive with Neko's hand-tuned kernels on Nvidia GH200 and A100 GPUs, while trailing on AMD MI250X. If the approach holds up, CFD developers could stop rewriting kernels per accelerator and instead tune one graph representation.

What carries the argument

The central object is the Stateful Dataflow Multigraph (SDFG), DaCe's intermediate representation of a program as a dataflow graph of states, maps, memlets, and tasklets. The SDFG decouples what is computed from how it is executed: the same graph feeds code generation for CUDA, HIP, and other backends. The argument is carried by a small set of graph transformations—MapFusion to merge the kernel's two element loops, MapCollapse to flatten spatial loops into one GPU thread block, MapExpansion to expose parallel hierarchy, and LocalStorage to place temporary arrays in shared memory—together with constant propagation of the polynomial order lx. Because these transforms operate on the graph rather than on source code, the same optimisation recipe can be replayed when targeting different accelerators.

What would settle it

Compile the same ax.sdfg with the same DaCe release (e.g. 1.0.1) on a GH200 and an MI250X, with identical mesh sizes and polynomial orders, and compare Gflops/s; if the GH200 result falls to the MI250X level, the reported Nvidia performance is a property of DaCe 0.15.1, not of the kernel description. A simpler check already present in the paper's own table: running the Nvidia benchmarks under DaCe 1.0.1 should reproduce the degradation the authors report.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that DaCe's code generator, applied to the ax matrix-evaluation kernel of the Neko spectral-element solver, produces GPU code whose performance is comparable with Neko's hand-optimised CUDA and HIP kernels, and does so from a single source. The kernel is formulated as a naive two-map Python program over all elements; the SDFG is then optimised by fusing the two element maps, collapsing the spatial maps, and promoting temporary arrays into shared memory. Compiled to CUDA on the two Nvidia systems and to HIP on the AMD system, the generated code matches or overtakes the 1D strategy and approaches the KSTEP strategy on GH200 and A100, especially at higher polynomial orders and large mesh sizes. On MI250X the generated code is significantly slower than the hand-tuned implementations. The paper's portability result is therefore explicitly qualified: competitive performance is demonstrated on Nvidia, while AMD support is functional but leaves performance on the table.

Load-bearing premise

The load-bearing premise is that the performance difference between architectures is explained by the SDFG description and its transformations rather than by which DaCe version happened to be installed; the paper pairs DaCe 0.15.1 with Nvidia GPUs and DaCe 1.0.1 with the AMD GPU, so if one version consistently generates worse code, the portability comparison is not apples-to-apples.

Editorial extensions

If this is right

  • A single SDFG kernel can target Nvidia and AMD GPUs from the same Python-level description, so adding a new accelerator backend no longer requires writing a new kernel by hand.
  • The C-to-Fortran interface used for Neko provides a template for embedding DaCe-generated code in other large Fortran/object-oriented scientific codes.
  • If the optimisation recipe (map fusion, collapse, shared-memory promotion) carries over, other small-tensor kernels in spectral-element solvers could be ported the same way.
  • Performance tuning shifts from hand-writing device code to choosing and ordering SDFG transformations, which can be replayed automatically for each new mesh size or polynomial order.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because the paper used DaCe 0.15.1 on the Nvidia systems and DaCe 1.0.1 on MI250X, the cross-platform comparison conflates hardware differences with code-generator version differences; a same-version test would be needed to know whether the AMD gap is inherent to DaCe's HIP back-end or an artifact of the newer release.
  • Editorial inference: the paper's own version table suggests that portable performance is currently a property of a pinned toolchain, not of the SDFG alone; if DaCe 1.0.1 degrades Nvidia performance as reported, then the single-source promise depends on generator stability across releases.
  • Editorial inference: the approach is tested only at small tensor orders lx ≤ 8, where SDFG transformations are easiest; a natural extension would be to benchmark lx > 8 or the gather-scatter operation, where hand-tuned kernels may retain a larger edge.
  • Editorial inference: the optimisation recipe described here could be tested as a reusable library of SDFG transforms for other matrix-free finite-element codes, independent of Neko.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents a workflow for expressing the main matrix-vector evaluation kernel (Ax) of the spectral-element CFD code Neko as a DaCe Stateful Dataflow Multigraph (SDFG), applying a sequence of optimization transformations, compiling the SDFG into a library, and linking it into Neko through a C-Fortran interface. The authors benchmark the DaCe-generated kernel against Neko's hand-tuned CUDA/HIP implementations (KSTEP and 1D variants) on three GPU systems: Nvidia GH200, Nvidia A100, and AMD MI250X. They report that the DaCe kernel is competitive with hand-tuned kernels on the Nvidia systems for several polynomial orders and mesh sizes, while on MI250X it is 'significantly lacking' relative to the HIP baseline. The paper claims portability and competitive performance across all three platforms.

Significance. If the central claim held, the work would be a useful demonstration that a single SDFG kernel expression can replace per-architecture rewrites for a core CFD computational kernel, and the detailed integration workflow (SDFG generation, transform recipe, and linking into Neko) would be a practical contribution for the performance-portability community. The comparison against Neko's hand-tuned CUDA/HIP kernels is a meaningful external baseline, and the paper does not appear to derive its conclusions from curve fitting or circular reasoning. However, the significance is substantially limited by two issues: the abstract overstates the results given the explicitly reported AMD shortfall, and the cross-platform comparison is confounded by different DaCe versions on different platforms.

major comments (3)
  1. [Table 2 and Section 7] The cross-platform portability claim is not supported by an apples-to-apples comparison because different DaCe versions are used on different platforms: the Nvidia systems use DaCe 0.15.1 while the MI250X system uses DaCe 1.0.1, and Section 7 states that the changes between 0.15.1 and 0.16 'degraded performance for the Nvidia version compared with the Neko kernels to a similar degree as we see on the AMD system.' This means the observed AMD shortfall cannot be attributed solely to the SDFG formulation or the target hardware; it may be an artifact of code-generation regressions in a particular toolchain version. The authors should either rerun the Nvidia benchmarks with DaCe 1.0.1 and the AMD benchmark with an older version that works, or explicitly qualify all conclusions as version-specific and provide a sensitivity analysis across versions.
  2. [Abstract vs Section 5, MI250X paragraph] The abstract's claim of 'competitive performance results' across 'Nvidia GH200, Nvidia A100, and AMD MI250X GPUs' is contradicted by the paper's own results. Section 5 states that 'the DaCe performance on the MI250X is significantly lacking compared with the base implementation,' and Fig. 6 shows DaCe below the hand-tuned HIP kernels across most configurations. The claim should be qualified to the Nvidia systems, or the AMD gap should be addressed with additional optimization before making an unqualified portability-and-performance assertion.
  3. [Figures 4-6] The performance figures report no error bars, number of repetitions, or statistical treatment, although Section 5 notes that the A100 results have a larger standard deviation due to background processes on a shared node. For configurations where DaCe and KSTEP are close (e.g., A100 for lx < 7), the reported rankings are not meaningful without variance information. The authors should report mean and standard deviation over multiple runs, or at least state the number of repetitions and justify why error bars are omitted.
minor comments (5)
  1. [Listing 1.2] The printed Python code contains syntax errors that make it not directly compilable as shown, including mismatched parentheses on lines 9-10 and a capital 'I' used in place of 'i' on line 34. The authors should provide a corrected, self-contained listing.
  2. [Listing 1.3] The optimization script is partially redacted or truncated, e.g., `for arr in [ 'ud', ' ', ...]` and `for arr in [ 'dxtd', ' ', ..]`, which prevents reproduction of the exact transform sequence. Either expand the array lists or describe them in text.
  3. [Section 2] The sentence 'we discuss the background material and previous related put in context our work' is grammatically incomplete and should be rewritten.
  4. [Figure 3 caption] The caption contains typos such as 'eplise-shaped' instead of 'ellipse-shaped' and should be proofread.
  5. [Section 4.4] The text states 'only one GCD is considered' for the MI250X measurements without explaining whether this is a deliberate choice to match the single-GPU Nvidia systems; this rationale should be stated explicitly.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper reports an empirical benchmark of a DaCe-generated kernel against Neko's hand-tuned backends, with no fitted parameters or self-referential predictions.

full rationale

This paper is an empirical integration and benchmarking study. The central claim is that a DaCe SDFG formulation of Neko's matrix-evaluation kernel can be automatically compiled and achieve competitive performance on NVIDIA GH200, NVIDIA A100, and AMD MI250X. No quantity is fitted to data and then reported as a prediction; no theoretical result is derived from an ansatz that presupposes the conclusion. The performance numbers are measured against the independent, existing hand-written CUDA and HIP kernels in Neko v0.9.99, which constitute an external baseline rather than an output of the paper's own construction. The DaCe formulation itself is derived from Neko's standard kernel and is stated openly as an adapted/simplified version, so there is no disguised renaming of a known result. The paper does contain self-citations, but they are contextual references to prior workflow and Neko development, not load-bearing justifications for the benchmark outcome. The main caveat identified in the paper is that DaCe 0.15.1 was used on the NVIDIA systems while DaCe 1.0.1 was used on the AMD system, and the authors themselves attribute the AMD performance shortfall to DaCe version changes. That is a correctness or internal-validity concern about whether the headline portability claim generalizes, but it is not circularity: the benchmark still compares a generated kernel against an independent hand-tuned baseline, and the reported measurements are not forced by construction. Accordingly, no circular step meets the evidentiary bar required by the analysis.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new entities, forces, or conserved quantities. Its contribution is an implementation and a benchmark. The listed axioms are the background assumptions the central claim rests on, none of which are proven inside the paper.

assumptions (4)
  • domain assumption The DaCe SDFG framework and its code generators for CUDA and HIP produce functionally correct code for the expressed kernel.
    The paper relies on the framework's correctness without formal verification or a correctness test description; correctness is implied by matching Neko's baseline outputs.
  • domain assumption Listing 1.2's simplified kernel is numerically equivalent to Neko's Ax-Helm operator used in the comparison.
    Section 4.2 says the kernel is 'adapted from the standard kernel but simplified' and introduces temporary arrays, but no equivalence proof or validation experiment is shown.
  • domain assumption The two hand-tuned Neko backends (1D and KSTEP) represent a strong performance baseline.
    Section 4 states the baselines are from Neko v0.9.99 with auto-tuning; the paper assumes these are the best available hand-optimized implementations for the comparison.
  • domain assumption Performance on the single Ax test kernel is a meaningful proxy for the performance impact in the full Neko solver.
    Section 1 calls it 'a critical computational kernel', and Section 7 concludes about large-scale application sustainability, but no full-solver run is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Portable High-Performance Kernel Generation for a Computational Fluid Dynamics Code with DaCe." pith.science (2026). https://pith.science/paper/C5AH3KZE

@misc{pith2026250620994,
  author       = {Pith},
  title        = {Pith review of: Portable High-Performance Kernel Generation for a Computational Fluid Dynamics Code with DaCe},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/C5AH3KZE}},
  note         = {Machine review of arXiv:2506.20994}
}
read the original abstract

With the emergence of new high-performance computing (HPC) accelerators, such as Nvidia and AMD GPUs, efficiently targeting diverse hardware architectures has become a major challenge for HPC application developers. The increasing hardware diversity in HPC systems often necessitates the development of architecture-specific code, hindering the sustainability of large-scale scientific applications. In this work, we leverage DaCe, a data-centric parallel programming framework, to automate the generation of high-performance kernels. DaCe enables automatic code generation for multicore processors and various accelerators, reducing the burden on developers who would otherwise need to rewrite code for each new architecture. Our study demonstrates DaCe's capabilities by applying its automatic code generation to a critical computational kernel used in Computational Fluid Dynamics (CFD). Specifically, we focus on Neko, a Fortran-based solver that employs the spectral-element method, which relies on small tensor operations. We detail the formulation of this computational kernel using DaCe's Stateful Dataflow Multigraph (SDFG) representation and discuss how this approach facilitates high-performance code generation. Additionally, we outline the workflow for seamlessly integrating DaCe's generated code into the Neko solver. Our results highlight the portability and performance of the generated code across multiple platforms, including Nvidia GH200, Nvidia A100, and AMD MI250X GPUs, with competitive performance results. By demonstrating the potential of automatic code generation, we emphasise the feasibility of using portable solutions to ensure the long-term sustainability of large-scale scientific applications.

Figures

Figures reproduced from arXiv: 2506.20994 by the authors.

Figure 1
Figure 1. The SDFG for the Ax kernel (ax.sdfg) is generated from a restricted Python formulation. In general, the SDFGs can be generated and compiled Just-in-Time. But in this case, we generate the SDFG apply optimization transforms, and store the transformed SDFGs in a compressed format. The Data-Centric Programming Framework, DaCe [5] is used to generate optimized code to be integrated into the CFD code. DaCe programming fr… view at source ↗
Figure 2
Figure 2. The library is shipped with several ax.sdfg (one for each polynomial order) files. Additional optimization passes can be applied to ax.sdfg before compiling with sdfgcc. Configuration of the compilation is best controlled with DaCe environment variables. implementations for different back ends. Neko is based on the spectral element method. In this method, the simulation is divided into several elements that are disc… view at source ↗
Figure 3
Figure 3. An example of the human-readable IR transformed from the naive SDFG implementation to the 3D parallelization. A data container is eplise-shaped. The dark cyan represents shared memory data containers and dark pink represents global GPU data containers. The promotions of memory are allowed since all data movement occurs within the same parallel map [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: The performance (Gflops/s) for different mesh sizes and 3 ≤ lx ≤ 8 with the different implementations on a GH200 node. for lx < 7. For the higher lx: KSTEP performs better. The 1D implementation has consistently the lowest performance for large mesh sizes but also the …
Figure 5
Figure 5. Figure 5: The performance (Gflops/s) for different mesh sizes and 3 ≤ lx ≤ 8 with the different implementations on an A100 node. 6 Related works Performance portability frameworks As HPC systems continue to evolve with diverse hardware architectures, achieving performance portab…
Figure 6
Figure 6. Figure 6: The performance (Gflops/s) for different mesh sizes and 3 ≤ lx ≤ 8 with the different implementations on an MI250X node using one GCD. 7 Discussion and Conclusion We have shown an implementation of the main computational kernel of Neko’s matrix evaluation in DaCe with …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 15 canonical work pages

  1. [3]

    In: Extended Abstract In The 53rd International Conference on Parallel Processing Workshops (ICPP Workshops ’24) (2024)

    Andersson, M.I., Karp, M., Markidis, S.: Towards performance portable kernels for computational fluid dynamics using dace. In: Extended Abstract In The 53rd International Conference on Parallel Processing Workshops (ICPP Workshops ’24) (2024)

    work page 2024

  2. [1]

    Parallel Computing108, 102841 (2021)

    Abdelfattah, A., Barra, V., Beams, N., Bleile, R., Brown, J., Camier, J.S., Carson, R., Chalmers, N., Dobrev, V., Dudouit, Y., et al.: Gpu algorithms for efficient exascale discretizations. Parallel Computing108, 102841 (2021)

    work page 2021

  3. [2]

    In: Proceedings of the International Conference on High Performance Computing in Asia-Pacific Region

    Andersson, M.I., Markidis, S.: A case study on dace portability & performance for batched discrete fourier transforms. In: Proceedings of the International Conference on High Performance Computing in Asia-Pacific Region. pp. 55–63 (2023)

    work page 2023

  4. [4]

    In: 2019 ieee/acm international workshop on performance, portability and productivity in hpc (p3hpc)

    Beckingsale, D.A.e.A.: RAJA: Portable performance for large-scale scientific appli- cations. In: 2019 ieee/acm international workshop on performance, portability and productivity in hpc (p3hpc). pp. 71–81. IEEE (2019)

    work page 2019

  5. [5]

    In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

    Ben-Nun, T., de Fine Licht, J., Ziogas, A.N., Schneider, T., Hoefler, T.: Stateful dataflow multigraphs: A data-centric model for performance portability on hetero- geneous architectures. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. SC ’19 (2019)

    work page 2019

  6. [6]

    https://doi.org/10.5281/zenodo.4004744, https://github.com/ paranumal/libparanumal, release 0.5.0

    Chalmers, N., Karakus, A., Austin, A.P., Swirydowicz, K., Warburton, T.: libParanumal: a performance portable high-order finite element li- brary (2022). https://doi.org/10.5281/zenodo.4004744, https://github.com/ paranumal/libparanumal, release 0.5.0

    work page doi:10.5281/zenodo.4004744 2022

  7. [7]

    The International Journal of High Performance Computing Applications 34(5), 562–586 (2020)

    Fischer, P., Min, M., Rathnayake, T., Dutta, S., Kolev, T., Dobrev, V., Camier, J.S., Kronbichler, M., Warburton, T., Świrydowicz, K., et al.: Scalability of high- performance pde solvers. The International Journal of High Performance Computing Applications 34(5), 562–586 (2020)

    work page 2020

  8. [8]

    In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

    Jansson, N., Karp, M., Perez, A., Mukha, T., Ju, Y., Liu, J., Páll, S., Laure, E., Weinkauf, T., Schumacher, J., Schlatter, P., Markidis, S.: Exploring the ultimate regime of turbulent rayleigh–bénard convection through unprecedented spectral- element simulations. In: Proceedings of the International Conference for High Performance Computing, Networking, ...

    arXiv 2023

Show all 19 references
  1. [9]

    Computers & Fluids p

    Jansson, N., Karp, M., Podobas, A., Markidis, S., Schlatter, P.: Neko: A modern, portable, and scalable framework for high-fidelity computational fluid dynamics. Computers & Fluids p. 106243 (2024)

    2024

  2. [10]

    The International Journal of High Performance Computing Applications p

    Karp, M., Massaro, D., Jansson, N., Hart, A., Wahlgren, J., Schlatter, P., Markidis, S.: Large-scale direct numerical simulations of turbulence using gpus and modern fortran. The International Journal of High Performance Computing Applications p. 10943420231158616 (2023)

    2023

  3. [11]

    In: 2021 IEEE International Parallel and Distributed Processing Symposium (IPDPS)

    Karp, M., Podobas, A., Jansson, N., Kenter, T., Plessl, C., Schlatter, P., Markidis, S.: High-performance spectral element methods on field-programmable gate arrays : Implementation, evaluation, and future projection. In: 2021 IEEE International Parallel and Distributed Proces...

    2021

  4. [12]

    The international journal of high performance computing applications p

    Karp, M., Suarez, E., Meinke, J.H., Andersson, M.I., Schlatter, P., Markidis, S., Jansson, N.: Experience and analysis of scalable high-fidelity computational fluid dynamics on modular supercomputing architectures. The international journal of high performance computing applic...

    2025

  5. [13]

    Pennati, L., Andersson, M.I., Steiniger, K., Widera, R., Narwal, T., Bussmann, M., Markidis, S.: A parallel and highly-portable hpc poisson solver: Preconditioned bi-cgstab with alpaka (2025),https://arxiv.org/abs/2503.08935

    2025 arXiv

  6. [14]

    In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

    Schaad, P., Schneider, T., Ben-Nun, T., Calotoiu, A., Ziogas, A.N., Hoefler, T.: Fuzzyflow: Leveraging dataflow to find and squash program optimization bugs. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. SC ’2...

    2023

  7. [15]

    In: Proceedings of the 53rd International Conference on Parallel Processing

    Schieffer, G., Wahlgren, J., Ren, J., Faj, J., Peng, I.: Harnessing integrated cpu- gpu system memory for hpc: a first look into grace hopper. In: Proceedings of the 53rd International Conference on Parallel Processing. p. 199–209. ICPP ’24, Association for Computing Machinery...

    2024

  8. [16]

    In: Proceedings of the 11th International Symposium on Highly Effi- cient Accelerators and Reconfigurable Technologies

    Svedin, M., Chien, S.W.D., Chikafa, G., Jansson, N., Podobas, A.: Benchmarking the nvidia gpu lineage: From early k80 to modern a100 with asynchronous memory transfers. In: Proceedings of the 11th International Symposium on Highly Effi- cient Accelerators and Reconfigurable Te...

    2021

  9. [17]

    IEEE Transactions on Parallel and Distributed Systems33(4), 805–817 (2021)

    Trott, C.R., et Al.: Kokkos 3: Programming model extensions for the exascale era. IEEE Transactions on Parallel and Distributed Systems33(4), 805–817 (2021)

    2021

  10. [18]

    In: International Conference on High Performance Computing in Asia-Pacific Region

    Vincent, J., Gong, J., Karp, M., Peplinski, A., Jansson, N., Podobas, A., Jocksch, A., Yao, J., Hussain, F., Markidis, S., Karlsson, M., Pleiter, D., Laure, E., Schlatter, P.: Strong scaling of openacc enabled nek5000 on several gpu based hpc systems. In: International Confere...

    2022

  11. [19]

    IEEE Computer Society (May 2016)

    Zenker, E., Worpitz, B., Widera, R., Huebl, A., Juckeland, G., Knüpfer, A., Nagel, W.E., Bussmann, M.: Alpaka - an abstraction library for parallel kernel acceleration. IEEE Computer Society (May 2016)

    2016

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.