Pith. sign in

REVIEW 3 major objections 5 minor 15 references

A GPU port of the collapse-time kernel in PINOCCHIO, written only with OpenMP directives, runs 4–8x faster on NVIDIA and AMD hardware while matching CPU scientific output.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 12:37 UTC pith:5DOOIWOE

load-bearing objection Solid, honest engineering paper on GPU-porting PINOCCHIO's collapse-time kernel; speedups are real but single-run benchmarks and unequal per-CU hardware make headline numbers softer than they appear. the 3 major comments →

arxiv 2510.02873 v1 pith:5DOOIWOE submitted 2025-10-03 astro-ph.IM

Accelerating cosmological simulations on GPUs: a portable approach using OpenMP

M. D. Lepinzan , G. Lacopo , D. Goz , G. Taffoni , P. Monaco , P. J. Elahi , U. Varetto , M. Cytowski This is my paper
classification astro-ph.IM
keywords GPU offloadingOpenMP target directivescosmological simulationsdark matter halo catalogsinterpolationPINOCCHIOportabilityroofline analysis
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper reports a GPU port of the collapse-time calculation in the PINOCCHIO cosmological simulation code, using only OpenMP target directives so the same source runs on NVIDIA and AMD accelerators. The authors show that this embarrassingly parallel kernel, which the CPU version spends about 108 seconds on in a typical production run, executes 4x faster on an NVIDIA A100 and up to 8x faster on an AMD MI250X when offloaded to the GPU. To make the port possible, they wrote custom GPU-native cubic-spline and bilinear interpolation routines that reproduce the GNU Scientific Library results to about 0.003% residual. They further show that the ported kernel reaches over 80% of the FP64 roofline peak on NVIDIA and that the collapsed-particle distributions and halo mass functions from GPU and CPU runs agree to within about 1%, so the speedup does not cost scientific fidelity. If true, this is evidence that a directive-based programming model can be a portable, maintainable route to GPU acceleration for legacy cosmological codes.

Core claim

The central discovery is that the collapse-time computation in PINOCCHIO, which assigns each grid point an independent ellipsoidal-collapse calculation, is not only highly parallel but also cleanly expressible with OpenMP target offload constructs. By replacing the GSL-based interpolation calls with fully GPU-resident cubic spline and bilinear routines, eliminating branch divergence through masked (predicated) execution, and laying out data in struct-of-arrays form for coalesced memory access, the authors achieve a 4x speedup on an NVIDIA platform and up to 8x on an AMD platform relative to the CPU baseline at equal computational-unit counts. The port preserves the physics: the number of col

What carries the argument

The load-bearing device is the OpenMP target offload model applied to an embarrassingly parallel kernel, reinforced by two custom GPU-native interpolation routines — a cubic spline and a bilinear interpolator — that replace GSL calls and run entirely on the device. To keep the GPU busy, the code uses a masked/predicated control-flow transformation to remove conditional branches (avoiding thread divergence) and a struct-of-arrays data layout for coalesced global memory access. The roofline analysis on the NVIDIA platform shows the kernel sits in the compute-bound regime at more than 80% of the FP64 peak, confirming that these transformations succeed in feeding the arithmetic units.

Load-bearing premise

All performance figures come from a single run of each benchmark, and the 10–20 smoothing-radius repetitions used to sample variability all happen inside that one run, so the 4x/8x speedups and the ~160,000 Standard-hour saving carry no measured statistical uncertainty.

What would settle it

Run the same single-node strong-scaling benchmark (Classic kernel, 256^3 grid, on the same NVIDIA and AMD systems) at least 10 times as independent processes, recording wall-clock time each time; if the CPU-to-GPU speedup ratio has a coefficient of variation larger than 10–20%, or if the mean speedup falls below 4x on NVIDIA or 8x on AMD, the central performance claim is undermined. A complementary check is to replace the custom GPU interpolation with a higher-order reference (or GSL on CPU) and see whether the z≲0.5 collapsed-particle underestimate persists; if it vanishes, the residual is an

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • If the speedup holds in production, each PINOCCHIO run saves roughly 100 seconds of wall time; across the thousands of runs needed for survey covariance matrices, this accumulates to over 160,000 Standard-core-hours (about 28 hours wall time) per large campaign.
  • The ported kernel produces collapsed-particle counts agreeing with the CPU reference within about 1% over most redshifts and a halo mass function within 1% across all masses, indicating the GPU version is scientifically interchangeable with the CPU version.
  • Because the implementation uses only OpenMP directives and a single source, the same code runs on both NVIDIA and AMD GPUs without vendor-specific languages, lowering the maintenance cost of porting other legacy simulation kernels.
  • The custom GPU interpolation routines alone reach up to 12x speedup over GSL for large evaluation sets and keep a residual near 0.003%, so they are reusable components beyond PINOCCHIO.
  • The roofline result (over 80% FP64 peak) implies the offloaded kernel is near the hardware ceiling, so further gains will require algorithmic changes or faster hardware rather than more tuning of the kernel itself.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same OpenMP-target porting pattern (masked control flow plus device-resident interpolation) could be applied to the Tabulated collapse-time kernel used in modified-gravity runs; the paper notes its speedup is inflated by the unparallelized CPU baseline, so a fair test would first add OpenMP to that CPU version.
  • The small systematic underestimate of collapsed particles at z ≲ 0.5 (2.5–3%) suggests that interpolation artifacts near the rapid/slow growth transition in D(a) could matter for applications needing percent-level accuracy in the low-redshift galaxy population; the paper argues this is absorbed by fragmentation and does not affect HMFs, but a dedicated test on clustering statistics would settle it
  • The roughly 100 s saving is small relative to the total runtime (about 3140 s), which raises the question of whether the same OpenMP strategy would yield proportionally larger gains when applied to the Fragmentation module, which dominates runtime at roughly 76%.
  • A companion work is promised to quantify energy savings; until then, the claimed resource savings are in compute time, not measured energy, so green-HPC conclusions should be treated as an expectation rather than an established result.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper reports an OpenMP-target-based GPU porting of the collapse-time calculation in the cosmological simulation code PINOCCHIO. The authors replace the GSL-based cubic-spline and bilinear interpolation routines with GPU-native implementations, validate these against GSL on synthetic smooth functions, and benchmark the ported kernels on the LEONARDO (NVIDIA A100) and SETONIX (AMD MI250X) systems. They report speedups of 4x on NVIDIA and up to 8x on AMD in single-node tests, about 6x in a production run on NVIDIA, a cumulative saving of about 160000 Standard-h, and a roofline analysis showing more than 80% of FP64 peak performance. Physical validation is provided by comparing collapsed-particle distributions and halo mass functions between the CPU and GPU versions.

Significance. If the quantitative claims hold, the paper is a useful demonstration that OpenMP target directives can be used to port a production cosmology kernel to both NVIDIA and AMD GPUs with modest code changes, and it provides a practical template for modernizing legacy CPU codes. The main strengths are the end-to-end physical validation (collapsed-particle distribution and HMF comparison), the roofline analysis, and the explicit portability test on two vendor architectures. The paper is also commendably transparent about the single-run nature of its benchmarks, although that transparency does not remove the statistical weakness. The results would be more convincing with repeated timing measurements and a cleaner cross-platform comparison.

major comments (3)
  1. [Section 5, first paragraph; Sections 5.3 and 5.5] The statement that 'performance results are based on a single run' is not rescued by the following sentence: the 10-20 smoothing radii are executed sequentially inside the same process on the same node, so they do not sample the run-to-run distribution of wall-clock times (thermal state, clock boosting, node contention, OS jitter, compiler/runtime variability). Consequently, the quantitative claims that depend on these timings - 4x on NVIDIA, up to 8x on AMD (Figures 6-8), about 6x and about 100 s saving per production run, and the 160000 Standard-h cumulative saving (Section 5.5, abstract) - carry no stated uncertainty. Please repeat the single-node benchmarks on fresh allocations a statistically meaningful number of times and report mean +/- std or min-max, and also report raw wall-clock times rather than only normalized speedups. If this is not feasible, the claims should be softened
  2. [Sections 4.4 and 5.3; Section 2.3.2] The cross-platform comparison is confounded by the definition of a 'computational unit.' One CU is 1/4 of a node: 8 CPU cores + 1 A100 on LEONARDO, but 16 CPU cores + 2 MI250X GCDs on SETONIX. The statement that the AMD speedup is 'roughly twice' the NVIDIA speedup therefore conflates the GPU architecture with a different number of accelerators per CU (2 GCDs vs 1 A100) and different CPU baselines (16 vs 8 cores). The explanation in Section 2.3.2 based on 'approximately 5x' FP64 peak is also incomplete: the per-CU FP64 ratio is roughly (2*47.9)/9.7 ~ 10, not 5. As written, the paper does not support the claim that AMD 'exploits its theoretical advantages more effectively' or that the AMD speedup is intrinsically larger. Please present per-accelerator absolute timings or matched-GPU comparisons, or restrict the cross-platform discussion to 'speedup relative to the local CPU configuration.
  3. [Sections 4.3, 5.1, 5.2 and 5.5] The custom interpolation routines are validated only on smooth synthetic functions (x^2 and x^2+y^2). These are nearly ideal inputs for spline interpolation, so the 0.003% residual quoted in the abstract is not evidence of accuracy on the actual collapse-time table. The production comparison in Section 5.5 (Figure 10) is the relevant test, but it shows local deviations of 2.5-3% in the collapsed-particle distribution at z<0.5, two orders of magnitude larger than the reported interpolation residual. The text attributes this to a 'subtle interpolation artifact' but does not quantify or directly connect it to the 0.003% validation. Please either validate the interpolation routines on realistic tables extracted from PINOCCHIO, or rephrase the abstract and conclusions so that 0.003% is explicitly the toy-problem residual rather than the production accuracy.
minor comments (5)
  1. [Section 3] Typo: 'the eigenvalues λ1, λ3, λ3' should presumably read 'λ1, λ2, λ3'.
  2. [Sections 5.1 and 5.2] The methodology (Section 4.3) says evaluation points range from 100 to 10^7, but the residual histograms in Figures 2 and 4 are said to be 'aggregated over the 35 evaluation points' / '50 evaluation points.' Please clarify whether the residual statistics are computed on the small displayed subset or on the full set of evaluation points; 35 points is too few to support a 0.003% average residual.
  3. [Section 5.5] The text refers to 'Appendix A.11'; the appendix is labeled 'Appendix A' and the figure is 'Figure A.11.' Please correct the cross-reference.
  4. [Section 5.4] The roofline analysis reports 'over 80% of theoretical FP64 peak' but gives no measured FLOP count, arithmetic intensity value, or details on how the metric was extracted from Nsight Compute. Including these numbers would allow readers to reproduce or sanity-check the result.
  5. [Section 5.5] The unit 'Standard-h' is not defined. If it means core-hours, please state so explicitly and show the conversion to the quoted '~28 hours wall time,' including the assumed number of cores and runs.

Circularity Check

0 steps flagged

No circular reasoning: the paper's claims are based on direct measurements and external benchmarks (GSL, CPU reference, roofline profiling), not on fitted parameters or self-referential definitions.

full rationale

The paper's central claims are: (i) custom GPU interpolation routines agree with GSL-based CPU implementations to ~0.003%; (ii) offloading the collapse-time kernels yields 4x/8x single-node speedups and ~6x production speedup; (iii) the Classic kernel operates above 80% of FP64 peak; (iv) GPU collapsed-particle distributions and HMFs match CPU outputs. None of these claims is derived from the quantity it is supposed to establish. Interpolation accuracy is checked against the independent GNU Scientific Library ('We then compare the interpolated values produced by the GPU-native routines against those obtained using the original GSL-based CPU implementation') on synthetic analytic functions, so the accuracy target is not the GPU routine itself. Speedups are defined as measured ratios of wall-clock times ('labels above GPU bars indicate the speedup of the GPU implementation relative to the CPU implementation at the same number of CUs'), not as predictions from fitted parameters. The roofline efficiency is obtained from NVIDIA Nsight Compute, an external profiler, and reports measured arithmetic intensity and throughput relative to hardware peak. Scientific correctness is validated by direct comparison of a downstream physical output (collapsed-particle counts and HMFs) between GPU and CPU runs, with residuals at the 1% level. The only significant caveat is a statistical one, not a circular one: 'performance results are based on a single run' and the paper attempts to mitigate this by saying the repeated evaluation over 10–20 smoothing radii 'effectively samples variability', but those repetitions occur inside the same process and do not sample independent wall-clock draws. That limitation affects the error bars on the speedup numbers, but it does not make any derivation reduce to its own input. Citations to prior PINOCCHIO work (e.g., Monaco et al. 2002, 2013; Munari et al. 2017) supply the physical model and context; they are not used to define or prove the GPU porting result. Therefore no circular step is present.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 0 invented entities

Central claims rest on external CPU/GSL reference, the embarrassingly parallel/compute-bound nature of the kernel, and the representativeness of single-run benchmarks; no free parameters or invented entities are introduced.

axioms (3)
  • domain assumption The collapse-time calculation is embarrassingly parallel and compute-bound, with no hidden data dependencies that would change results when offloaded.
    Section 3 describes each grid point as independent; Section 5.4 roofline supports compute-bound. If thread-safety or order-dependent behavior existed, the GPU speedup and physical equivalence claims would fail.
  • domain assumption The custom GPU spline/bilinear routines reproduce GSL sufficiently well on the actual collapse-time data, not just on the synthetic x^2 test functions.
    Section 4.3 validates against synthetic y=x^2 and z=x^2+y^2; Section 5.5 checks final collapsed-particle statistics instead of directly validating interpolation on the production table.
  • ad hoc to paper The wall-clock time of one run is representative of the kernel's steady-state performance.
    Section 5 states 'performance results are based on a single run' and uses the 10-20 smoothing radii as pseudo-repetitions; this is an assumption about timing stability that is not directly evidenced.

pith-pipeline@v1.3.0-alltime-deepseek · 13959 in / 11699 out tokens · 742864 ms · 2026-08-04T12:37:55.716274+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Accelerating cosmological simulations on GPUs: a portable approach using OpenMP." pith.science (2026). https://pith.science/paper/5DOOIWOE

@misc{pith2026251002873,
  author       = {Pith},
  title        = {Pith review of: Accelerating cosmological simulations on GPUs: a portable approach using OpenMP},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5DOOIWOE}},
  note         = {Machine review of arXiv:2510.02873}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

In this work we present the porting to Graphics Processing Units (GPUs, using OpenMP target directives) and optimization of a key module within the cosmological {\pinocchio} code, a Lagrangian Perturbation Theory (LPT)-based framework widely used for generating dark matter (DM) halo catalogs. Our optimization focuses on a specific segment of the code responsible for calculating the collapse time of each particle involved in the simulation. Due to the embarrassingly parallel nature of this computation, it represents an ideal candidate for GPU offloading. As part of the porting process, we developed fully GPU-native implementations of both cubic spline and bilinear interpolation routines, required for evaluating collapse times. Since GNU Scientific Library (GSL) does not support GPU offloading, these custom implementations run entirely on the GPU and achieve residuals of only $\sim0.003\%$ when compared to the CPU-based implementation of GSL. Comparative benchmarking on the LEONARDO (NVIDIA-based) and SETONIX (AMD-based) supercomputers reveals notable portability and performance, with speedups of~\textit{4x} and up to~\textit{8x}, respectively. While collapse time calculation is not a primary bottleneck in the overall workflow, the acceleration reduces full production runs by $\sim 100$ seconds each leading to a cumulative saving of $\sim 160000$ Standard-h ($\sim28$ hours wall time) across thousands of simulations. Roofline analysis confirms that our GPU porting achieves over 80\% of the theoretical FP64 peak performance, confirming efficient compute-bound execution. This work demonstrates that OpenMP directives offer a portable, effective strategy for accelerating large-scale cosmological simulations on heterogeneous hardware.

Figures

Figures reproduced from arXiv: 2510.02873 by D. Goz, G. Lacopo, G. Taffoni, M. Cytowski, M. D. Lepinzan, P. J. Elahi, P. Monaco, U. Varetto.

Figure 1
Figure 1. Figure 1: Overview of the GPU offloading strategy used in the [PITH_FULL_IMAGE:figures/full_fig_p011_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Left panel displays the comparison between the GPU-custom cubic spline in [PITH_FULL_IMAGE:figures/full_fig_p016_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Comparison of interpolation wall-time between the GPU-native cubic spline [PITH_FULL_IMAGE:figures/full_fig_p017_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Same as Figure [PITH_FULL_IMAGE:figures/full_fig_p018_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Same as Figure [PITH_FULL_IMAGE:figures/full_fig_p019_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Speedup in strong-scaling tests for NVIDIA platform for different grid sizes, [PITH_FULL_IMAGE:figures/full_fig_p020_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Same as Figure [PITH_FULL_IMAGE:figures/full_fig_p021_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Same as Figure [PITH_FULL_IMAGE:figures/full_fig_p022_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Roofline analysis of the GPU-accelerated [PITH_FULL_IMAGE:figures/full_fig_p023_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Comparison of the number of collapsed particles per redshift bin obtained using [PITH_FULL_IMAGE:figures/full_fig_p025_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

15 extracted references · 2 canonical work pages

  1. [6]

    Euclid. I. Overview of the Euclid mission arXiv:2405.13491. Monaco, P., Sefusatti, E., Borgani, S., Crocce, M., Fosalba, P., Sheth, R.K., Theuns, T.,

    arXiv

  2. [13]

    Eurohpc space coe: Redesigning scalable parallel astrophysical codes for exascale. invited pa- per, in: Proceedings of the 22nd ACM International Conference on Com- puting Frontiers: Workshops and Special Sessions, Association for Com- puting Machinery, New York, NY, USA. p. 177–184. URL:https: //doi.org/10.1145/3706594.3728892, doi:10.1145/3706594.372889...

    arXiv

  3. [15]

    (Eds.), Software and Cyberinfrastructure for Astronomy VIII, International Society for Optics and Photonics

    Redesign of astro- physical codes for exascale computing: the SPACE experience, in: Ibsen, J., Chiozzi, G. (Eds.), Software and Cyberinfrastructure for Astronomy VIII, International Society for Optics and Photonics. SPIE. p. 131010X. doi:10.1117/12.3020361. Appendix A. Halo Mass Function comparison: GPU vs CPU in a production run scenario The Halo Mass Fu...

    work page doi:10.1117/12.3020361 1974

  4. [46]

    Sheth, R.K., Tormen, G.,

    doi:10.1088/1475-7516/2024/08/046,arXiv:2403.08634. Sheth, R.K., Tormen, G.,

    Pith/arXiv arXiv 2024

  5. [119]

    1999.02692.x,arXiv:astro-ph/9901122

    doi:10.1046/j.1365-8711. 1999.02692.x,arXiv:astro-ph/9901122. Shukla, N., Romeo, A., Caravita, C., et al.,

    arXiv 1999

  6. [587]

    Munari, E., Monaco, P., Sefusatti, E., Castorina, E., Mohammad, F.G., Anselmi, S., Borgani, S.,

    doi:10.1046/j.1365-8711.2002.05162.x, arXiv:astro-ph/0109323. Munari, E., Monaco, P., Sefusatti, E., Castorina, E., Mohammad, F.G., Anselmi, S., Borgani, S.,

    arXiv 2002

  7. [1974]

    Astrophys

    Formation of galaxies and clusters of galax- ies by selfsimilar gravitational condensation. Astrophys. J. 187, 425–438. doi:10.1086/152650. 29 Salvalaggio, J., Castiblanco, L., Noreña, J., Sefusatti, E., Monaco, P.,

    doi:10.1086/152650

  8. [2013]

    An accurate tool for the fast generation of dark matter halo catalogs. Mon. Not. Roy. Astron. Soc. 433, 2389–2402. doi:10.1093/ mnras/stt907,arXiv:1305.1505. Monaco, P., Theuns, T., Taffoni, G.,

    Pith/arXiv arXiv

  9. [2016]

    Evaluating openmp 4.0’s effectiveness as a heterogeneous parallel programming model, in: 2016 IEEE International Parallel and Distributed Processing Symposium Work- shops (IPDPSW), pp. 338–347. doi:10.1109/IPDPSW.2016.70. Mellier, Y., et al. (Euclid),

    work page doi:10.1109/ipdpsw.2016.70 2016

  10. [2017]

    Improvingfastgenerationofhalocatalogues with higher order Lagrangian perturbation theory. Mon. Not. Roy. Astron. Soc. 465, 4658–4677. doi:10.1093/mnras/stw3085,arXiv:1605.04788. Press, W.H., Schechter, P.,

    Pith/arXiv arXiv

  11. [2021]

    Euclid : Effects of sample covariance on the number counts of galaxy clusters. Astron. Astrophys. 652, A21. doi:10.1051/0004-6361/202140592,arXiv:2102.08914. Fumagalli, A., et al. (Euclid),

    Pith/arXiv arXiv

  12. [2022]

    Numerical implementation of the Cubic Galileon model in PINOCCHIO. Mon. Not. Roy. Astron. Soc. 516, 5762–5774. doi:10.1093/mnras/stac2298,arXiv:2111.02240. Taffoni, G., Mignone, A., Tornatore, L., et al.,

    Pith/arXiv arXiv

  13. [2023]

    Euclid preparation. XXIV. Calibration of the halo mass function inΛ(ν)CDM cosmologies. Astron. Astrophys. 671, A100. doi:10.1051/0004-6361/202244674,arXiv:2208.02174. Euclid Collaboration: Monaco, P., Parimbelli, G., Elkhashab, Y., et al.,

    Pith/arXiv arXiv

  14. [2024]

    Covari- ance model validation for the two-point correlation function of galaxy clus- ters

    Euclid preparation - XXXV. Covari- ance model validation for the two-point correlation function of galaxy clus- ters. Astron. Astrophys. 683, A253. doi:10.1051/0004-6361/202245540, arXiv:2211.12965. Lacopo, G., et al., in prep. Accelerating cosmological simulations on GPUs: a step towards sustainability and green-awareness. In prep. Martineau, M., McIntos...

    arXiv

  15. [2025]

    simulating thousands of euclid spectroscopic skies

    Euclid preparation. simulating thousands of euclid spectroscopic skies. Astron. Astrophys.arXiv:2507.12116. Farber, R.,

    arXiv