Efficient high performance computing with the ALICE Event Processing Nodes GPU-based farm

Alexander Wilhelmi; Andreas Redelbach; David Rohr; Dirk Hutter; Edvard Andreassen; Federico Ronchetti; Felix Weiglhofer; Gautam Dange; Giada Erba; Grigory Kozlov

REVIEW 3 major objections 5 minor 13 references

Efficient high performance computing with the ALICE Event Processing Nodes GPU-based farm

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The ALICE EPN GPU farm sustains the 50 kHz Pb–Pb collision rate with zero rejected time frames.

desk verdict A useful, honest status report from a real GPU farm; the physics program is real, but the no-data-loss claim needs a defined counter before it can be taken at face value. read the letter →

arxiv 2412.13755 v1 pith:AKS4KJPQ submitted 2024-12-18 hep-ex

Federico Ronchetti , Valentina Akishina , Edvard Andreassen , Nora Bluhme , Gautam Dange , Jan de Cuveland , Giada Erba , Hari Gaur

show 11 more authors

Dirk Hutter Grigory Kozlov Luboš Krčál Sarah La Pointe Johannes Lehrbach Volker Lindenstruth Gvozden Neskovic Andreas Redelbach David Rohr Felix Weiglhofer Alexander Wilhelmi

This is my paper · ORCID

classification hep-ex

keywords scientificcomputingsustainableHTCHPCGPUALICEonlinedatacompressionrANSentropycoding

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper describes the ALICE Event Processing Nodes (EPN) farm, a 350-server, 2800-GPU system that performs online data reconstruction and compression while LHC collisions are happening. The authors' central result is that this farm keeps up with lead–lead collisions at the 50 kHz interaction rate: during sustained 2024 running, the synchronous processing chain carried about 800 GB/s of zero-suppressed TPC data and recorded zero rejected time frames. The data volume is tamed in two stages: FPGA zero suppression cuts the TPC's raw 3.3 TB/s output down to roughly 800 GB/s, and a custom entropy coder based on rANS compresses the data close to the theoretical entropy limit, a factor of 2–3. The same infrastructure, cooled by adiabatic air handling units, reaches a power usage effectiveness of 1.05 during heavy-ion fills.

What carries the argument

The central object is the time frame (TF), a fixed time window of continuous readout data from all detectors that is processed as a single unit. Each TF is built from sub-time frames on the EPN nodes, must fit in GPU memory, and is the unit on which the synchronous processing chain operates. The two mechanisms that carry the performance claim are the FPGA-level zero suppression on the detector front end, which cuts the TPC rate from 3.3 TB/s to about 800 GB/s, and the custom vectorized rANS entropy coder, which compresses each detector's integer arrays close to the entropy limit with a speed of up to 3200 MB/s and a reported twofold speedup over state-of-the-art CPU implementations.

What would settle it

Count rejected time frames in the TF scheduler logs during a full sustained 50 kHz Pb–Pb fill while measuring the actual post-zero-suppression TPC input rate; if any time frame is rejected at 800 GB/s, or if the resulting compressed time frame sizes exceed the 2–3x entropy-bound compression ratio over a complete fill, the central claim fails.

Watch

Extended reading notes

Core claim

The paper's claim is that a GPU-based online processing farm can absorb the full data rate of an upgraded large detector in continuous readout mode: at 50 kHz Pb–Pb collisions, the EPN farm reconstructs and compresses time frames in real time, with the rejected-time-frame rate measured at zero during sustained running in 2024. The TPC's raw 3.3 TB/s output is first reduced by FPGA zero suppression to about 800 GB/s, then processed on GPUs, which supply 90% of the farm's compute power in synchronous operation, and compressed by a custom rANS entropy coder into compressed time frames near the entropy limit, a factor of 2–3 smaller. The authors further claim that the same farm, with its adiabatic cooling infrastructure, achieves a PUE of 1.05, and that without GPUs roughly eight times as many CPU-only servers would be required.

Load-bearing premise

The performance claim depends on the assumed TPC data rates (3.3 TB/s raw, about 800 GB/s after zero suppression at 50 kHz Pb–Pb) and on the rANS compressor staying near the 2–3x entropy limit, both of which are taken from detector design and simulations rather than derived in this paper.

Editorial extensions

If this is right

If the reported performance holds, ALICE can record Pb–Pb data continuously at 50 kHz without losing a single time frame, enabling a data sample roughly ten times larger than the combined Run 1 and Run 2 samples.
The near-entropy 2–3x lossless compression factor, combined with retaining only 3–4.5% of compressed data on disk after event selection, keeps the central storage buffers from filling during high-rate running.
The adiabatic cooling design, with PUE 1.05 during Pb–Pb fills and up to 1.16 during summer proton running, makes the GPU farm economically and environmentally competitive compared with CPU-only alternatives.
During asynchronous reconstruction, about 60% of the workload is GPU-accelerated, yielding roughly a 2.5x speedup over CPU-only processing, and porting the remaining central-barrel detectors could raise GPU coverage to about 80%.
The measured headroom of 1.24 TB/s in 2022 proton stress tests, nearly double the nominal 600 GB/s design rate, suggests the farm can absorb further data-rate increases from detector upgrades.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A direct extension of the reported margins is to track the actual post-zero-suppression TPC rate during every 50 kHz fill and correlate it with buffer occupancy on the slower MI-50 nodes; this would quantify how much safety margin sits behind the zero-rejected-TF result.
The same pipeline pattern — FPGA zero suppression, lossless entropy coding, GPU reconstruction — transfers to any continuous-readout detector whose raw data have strongly skewed symbol distributions, since the rANS gain is tied to that skew.
The paper's implicit cost argument could be made explicit by comparing the total cost of ownership of the GPU farm with the eight-times-larger CPU-only alternative the authors cite, including energy costs over the full Run 3 and Run 4 period.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a solid operational status report from a working system, not a methods paper. The genuinely new material is the 2022–2024 operational record: the first sustained 50 kHz Pb–Pb running, the PUE of 1.05, the 180 PB of pp and 39 PB of Pb–Pb data in 2024, and the measured rANS throughput. That is worth having on the record.

What it does well: it is unusually candid about deviations from design. The paper openly reports the 30% TPC data-size increase, the mid-stream addition of 30 MI-50 and 70 MI-100 nodes, the change in time-frame length after tests, and the fact that the 2022 rate scans used an intermediate firmware. That kind of honesty makes the headline claims more credible.

Soft spots, in order of importance. The stress-test note is on target: the 'rejected TF rate is zero' in Sec. 4.3 is the only evidence for no data loss during synchronous processing, and the counter is never defined—observation window, histogram scope, or whether it counts only scheduler-level rejections as opposed to losses in the FLP buffers or on the RDMA path. The abstract's 'cope with 50 kHz' claim needs at least a sentence defining the metric and the interval (peak plateau vs. full fill). That is a fixable evidence gap, not a sign of fraud; a status report from a running experiment should be able to pull these counts.

Second, several load-bearing numbers are cited from collaboration reports rather than derived here: the 3.3 TB/s raw TPC rate, the 800 GB/s post-ZS at 50 kHz, and the factor-of-eight CPU comparison are all from [3] and the TDR. That is acceptable for a status report, but it means the paper is a secondary source for those numbers, not a primary one.

Third, no error bars or measurement methodology on the PUE or the throughput figures. For this kind of paper, I would call it a minor issue—operational readings are what they are—but a line on how PUE was averaged would be cheap.

The citation pattern is fine. The rANS work is referenced to the collaboration's own paper, which is the right citation. No invented entities or parameter fitting; the free parameters named in the report (Tset, hysteresis, TF length) are genuinely operational ones, not made up.

Who it is for: anyone working on large-scale scientific computing or HEP computing models, and people writing about sustainable data centers. It is not a general-interest physics result. It deserves a serious referee—a technical expert can check the claims in a day, and the requested clarifications are minor. I would send it to review but require the rejection-counter definition and a short measurement note before publication.

Referee Report

3 major / 5 minor

Summary. The paper describes the ALICE O2 Event Processing Nodes (EPN) farm for LHC Run 3, claiming that a 350-server/2800-GPU farm sustains 50 kHz Pb-Pb synchronous data taking by performing online reconstruction on GPUs and applying entropy-based lossless compression to produce compressed time frames (CTFs). It also reports an adiabatic cooling infrastructure with PUE values of 1.05 under Pb-Pb conditions and 1.16 for proton-proton running. Operational evidence includes 2022-2024 data volumes, a stress test sustaining 1.24 TB/s input, and a 2024 high-luminosity Pb-Pb run in which the rejected time-frame rate in the TF Scheduler is stated to be zero.

Significance. If the operational claims are backed by well-defined measurements, the paper would be a valuable reference for the HEP computing community: it demonstrates that a GPU-based synchronous reconstruction chain can handle 50 kHz Pb-Pb rates at the scale of a full experiment, and it documents a working adiabatic cooling system with a measured PUE near 1.05. The concrete description of the farm architecture, the 2022-2024 data volumes, and the discussion of time-frame length tuning are useful engineering information. A notable strength is that the paper reports numbers from a real deployed system rather than from simulation alone; however, the central performance claims currently rest on self-reported monitoring and on rates and compression limits taken from design studies and internal references, so the evidence base is narrower than the abstract suggests.

major comments (3)

[Section 4.3 and Fig. 4] The sentence 'the rejected TF rate is zero, indicating that the system handles raw input rates effectively, with no data loss during synchronous reconstruction' is the only evidence offered for the central 50 kHz no-data-loss claim, but the manuscript never defines the rejected-TF counter, the observation window, the number of fills or time frames covered, or whether the zero interval spans the full fill or only the 30-40 minute peak plateau mentioned earlier in the section. Since time frames can be lost before they reach the EPN TF Scheduler (for example in FLP buffering or on the RDMA/InfiniBand path), a zero count in this scheduler histogram is not by itself end-to-end evidence. Please define the metric, state the monitoring interval and fill statistics, and provide a reconciliation of TF counts between the FLP farm and the EPN farm.
[Sections 3.1 and 4.1] The input data rates of 3.3 TB/s raw TPC data and 800 GB/s post-zero-suppression data at 50 kHz Pb-Pb, as well as the '2x-3x' entropy-limited compression ratio, are taken from detector design studies and from internal references [3] and [10] rather than derived from measurements presented in this paper. The farm's compute and storage margin conclusions depend directly on these numbers. Please provide measured post-zero-suppression rates and achieved CTF compression ratios for the 2024 Pb-Pb run, or explicitly label each number as a design target, a simulation result, or an operational measurement.
[Section 3.3] The PUE values of 1.05 and 1.16 are presented as operational results, but no measurement methodology is given. To support the abstract's energy-efficiency claim, please state the averaging period, the scope of the power measurement (IT load, cooling, water purification, lighting, etc.), and the instrumentation used, or label these values as design estimates rather than measured PUE.

minor comments (5)

[Section 2.1] The statement that 'approximately eight times as many CPU-only servers' would be needed cites reference [3]; please clarify whether this is an extrapolated design estimate or a measured comparison, and give the basis for the factor of eight.
[Section 4.1] The phrase 'compression speeds of up to 109 symbols per second' should read '10^9 symbols per second' (or the intended unit should be stated); as printed, the number is implausible and will confuse readers.
[Section 4.1] The sentence 'the entropy limit suggests a maximum compression ratio of factor 2x-3x' conflates a theoretical lower bound on compressed size with an observed compression ratio; please clarify whether 2x-3x is the measured CTF compression range or a theoretical expectation.
[Section 3.3] The text contains several typos, including 'an PUE' instead of 'a PUE', 'higher then' instead of 'higher than', and 'other then' instead of 'other than' in Section 4.2; these should be corrected during revision.
[Author affiliations] The affiliation for the Frankfurt Institute reads 'Adv5anced Studies' in the header; this appears to be a transcription error and should be corrected to 'Advanced Studies'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity identified: the paper reports measurements of a deployed system rather than deriving predictions from fitted inputs.

full rationale

The paper is an operational and performance report from the ALICE O2 EPN farm. Its central claims are empirical: the 50 kHz Pb–Pb data-taking with zero rejected time frames (Section 4.3, Fig. 4), the PUE of 1.05 under Pb–Pb conditions (Section 3.3), and the rANS-based compression performance (Section 4.1). These are measurements of a working system, not quantities derived from the paper's own fitted parameters. The entropy-limit compression argument is grounded in information theory and in comparisons against Huffman and gzip/zlib, and the compression ratios are reported as observed CTF sizes. The factor-of-eight CPU comparison is attributed to reference [3], which has an overlapping author, but it is a motivating context claim rather than the load-bearing evidence for the paper's measured performance. The absence of a precise definition for the 'rejected TF rate' in Fig. 4 is an evidence-quality or verification concern, not a circularity: the claim is not true by construction, and the paper does not rename a fitted quantity as a prediction. No load-bearing step reduces to its own input by definition or by self-citation, so the circularity score is 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

No new physical entities are introduced. The central claims rest on standard information theory, on detector data-rate numbers inherited from the ALICE design and simulation, and on a self-cited CPU versus GPU comparison.

free parameters (3)

Cooling set-point temperature Tset = 27 C
Hand-set supply air temperature in the adiabatic cooling control loop (Fig. 2). It is a tuned engineering parameter, not a fitted scientific constant.
Cooling hysteresis H = 2.2 C
Pump activation hysteresis in the adiabatic cooling control loop (Fig. 2); chosen by the design team.
Time frame length = 32 LHC orbits (2.8 ms)
Reduced from 128 orbits during 2022 validation to gain 10-14% compute performance (Section 4); a configuration choice affecting throughput.

assumptions (3)

standard math Shannon entropy provides the fundamental lower bound for lossless compression, and rANS can approach it.
Used in Section 4.1 to justify near-entropy CTF compression and the 2x-3x maximum ratio claim.
domain assumption The TPC produces 3.3 TB/s raw data and about 800 GB/s after FPGA zero suppression at 50 kHz PbPb.
Stated in Sections 3.1 and 4 without derivation; it sets the required input bandwidth of the EPN farm.
domain assumption A CPU-only solution would need about 8 times as many servers.
Cited from [3] by the same group, not derived in this paper; underpins the energy and cost-saving claims.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Efficient high performance computing with the ALICE Event Processing Nodes GPU-based farm." pith.science (2026). https://pith.science/paper/AKS4KJPQ

@misc{pith2026241213755,
  author       = {Pith},
  title        = {Pith review of: Efficient high performance computing with the ALICE Event Processing Nodes GPU-based farm},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AKS4KJPQ}},
  note         = {Machine review of arXiv:2412.13755}
}

abstract

Due to the increase of data volumes expected for the LHC Run 3 and Run 4, the ALICE Collaboration designed and deployed a new, energy efficient, computing model to run Online and Offline O$^2$ data processing within a single software framework. The ALICE O$^2$ Event Processing Nodes (EPN) project performs online data reconstruction using GPUs (Graphic Processing Units) instead of CPUs and applies an efficient, entropy-based, online data compression to cope with PbPb collision data at a 50 kHz hadronic interaction rate. Also, the O$^2$ EPN farm infrastructure features an energy efficient, environmentally friendly, adiabatic cooling system which allows for operational and capital cost savings.

Figures

Figures reproduced from arXiv: 2412.13755 by the authors.

**Figure 1.** Schematic overview of the O2 data flow and the ALICE computing infrastructure. See Section 3.1. Additional information in [13] 9 [PITH_FULL_IMAGE:figures/full_fig_p009_1.png] view at source ↗

**Figure 2.** Correlation between the adiabatic mode of one Air Handling Unit (AHU) and the outside ambient temperature (red markers). Each time the supply air temperature (blue markers) exceeds the value Tstart the pumps are activated (green markers, expressing the pump speed in opening grade from 0 to 100) and the supply air to the racks is cooled down until the value Tstart − H is reached, where Tstart = Tset + ∆T with Tset = … view at source ↗

**Figure 3.** Block diagram of the O2 synchronous processing and online calibrations. Details are given in Section 4.2. 10 [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: O2 EPN synchronous processing performance during the 50 kHz (sustained) Pb–Pb 2024 running. Details are in Section 4.3. 11 [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 11 canonical work pages

[3]

The O2 software framework and GPU usage in ALICE online and offline reconstruction in Run 3

Giulio Eulisse and David Rohr. The O2 software framework and GPU usage in ALICE online and offline reconstruction in Run 3. In26th International Conference on Computing in High Energy and Nuclear Physics (CHEP 2023) , page 295. EPJ Web of Conferences, 2024. https://doi.org/10.1051/epjconf/202429505022

arXiv 2023
[10]

Fast Entropy Coding for ALICE Run 3

Michael Lettrich. Fast entropy coding for ALICE Run 3.Proceedings of Science, 2 2021. https://arxiv.org/abs/2102.09649

work page Pith review arXiv 2021
[1]

The ALICE experiment at the CERN LHC.Journal of Instrumenta- tion, 3:S08002, 3 2008

The ALICE Collaboration. The ALICE experiment at the CERN LHC.Journal of Instrumenta- tion, 3:S08002, 3 2008. https://inspirehep.net/literature/796251

work page 2008
[2]

ALICE upgrades during the LHC Long Shutdown 2

The ALICE Collaboration. ALICE upgrades during the LHC Long Shutdown 2. Journal of Instrumentation, 19:P05062, 5 2024. https://inspirehep.net/literature/2628978

work page arXiv 2024
[4]

Real-time data processing in the ALICE High Level Trigger at the LHC

The ALICE Collaboration. Real-time data processing in the ALICE High Level Trigger at the LHC. Computer Physics Communications , 242:25–48, 9 2019. https://www.sciencedirect.com/science/article/abs/pii/s0010465519301250

work page 2019
[5]

Alexander S. Gillis. Power usage effectiveness.TechTarget, 4 2021. https://www.techtarget.com/searchdatacenter/definition/power-usage-effectiveness-pue

work page 2021
[6]

Technical design report for the upgrade of the online-offline computing system

P Buncic, M Krzewicki, and P Vande Vyvre. Technical design report for the upgrade of the online-offline computing system. Technical report, CERN, 2015. https://cds.cern.ch/record/2011297

arXiv 2015
[7]

Evolution of the O2 system

The ALICE Collaboration. Evolution of the O2 system. edms.cern.ch, 2019. https://edms.cern.ch/document/2248772/1

work page arXiv 2019

Show all 13 references

[8]

Nvidia cable management guidelines and FAQ.docs.nvidia.com, 11 2023

NVIDIA Corporation. Nvidia cable management guidelines and FAQ.docs.nvidia.com, 11 2023. https://docs.nvidia.com/nvidia-cable-management-guidelines-and-faq.pdf

2023
[9]

Impact of cable losses.Analog Devices, 9 2008

Bernard Hyland. Impact of cable losses.Analog Devices, 9 2008. https://www.analog.com/en/resources/technical-articles/impact-of-cable-losses.html

2008
[11]

D.A. Huffman. Proceedings of the IRE 40. InOptics, Global Edition , page 1098. Proceedings of the IRE 40, 1952

1952
[12]

Intrinsics for Intel advanced vector extensions 2 (intel AVX2)

Intel Corporation. Intrinsics for Intel advanced vector extensions 2 (intel AVX2). www.intel.com/content/www/us/en/docs, 2013. https://www.intel.com/content/www/us/en/docs/cpp-compiler/developer-guide-reference/2021- 8/intrinsics-for-avx2.html

2013
[13]

ALICE ups its game for sustainable computing.CERN Courier, Volume 63, Number 5 , 9-10 2023

Volker Lindenstruth. ALICE ups its game for sustainable computing.CERN Courier, Volume 63, Number 5 , 9-10 2023. https://cerncourier.com/wp-content/uploads/2023/09/cerncourier2023sepoct-digitaledition.pdf. 12

2023

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [3]

The O2 software framework and GPU usage in ALICE online and offline reconstruction in Run 3

Giulio Eulisse and David Rohr. The O2 software framework and GPU usage in ALICE online and offline reconstruction in Run 3. In26th International Conference on Computing in High Energy and Nuclear Physics (CHEP 2023) , page 295. EPJ Web of Conferences, 2024. https://doi.org/10.1051/epjconf/202429505022

arXiv 2023

[2] [10]

Fast Entropy Coding for ALICE Run 3

Michael Lettrich. Fast entropy coding for ALICE Run 3.Proceedings of Science, 2 2021. https://arxiv.org/abs/2102.09649

work page Pith review arXiv 2021

[3] [1]

The ALICE experiment at the CERN LHC.Journal of Instrumenta- tion, 3:S08002, 3 2008

The ALICE Collaboration. The ALICE experiment at the CERN LHC.Journal of Instrumenta- tion, 3:S08002, 3 2008. https://inspirehep.net/literature/796251

work page 2008

[4] [2]

ALICE upgrades during the LHC Long Shutdown 2

The ALICE Collaboration. ALICE upgrades during the LHC Long Shutdown 2. Journal of Instrumentation, 19:P05062, 5 2024. https://inspirehep.net/literature/2628978

work page arXiv 2024

[5] [4]

Real-time data processing in the ALICE High Level Trigger at the LHC

The ALICE Collaboration. Real-time data processing in the ALICE High Level Trigger at the LHC. Computer Physics Communications , 242:25–48, 9 2019. https://www.sciencedirect.com/science/article/abs/pii/s0010465519301250

work page 2019

[6] [5]

Alexander S. Gillis. Power usage effectiveness.TechTarget, 4 2021. https://www.techtarget.com/searchdatacenter/definition/power-usage-effectiveness-pue

work page 2021

[7] [6]

Technical design report for the upgrade of the online-offline computing system

P Buncic, M Krzewicki, and P Vande Vyvre. Technical design report for the upgrade of the online-offline computing system. Technical report, CERN, 2015. https://cds.cern.ch/record/2011297

arXiv 2015

[8] [7]

Evolution of the O2 system

The ALICE Collaboration. Evolution of the O2 system. edms.cern.ch, 2019. https://edms.cern.ch/document/2248772/1

work page arXiv 2019