REVIEW 3 major objections 5 minor 32 references
Ultra High Energy Cosmic Rays versus Models of High Energy Hadronic Interactions
T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read This paper asks whether including the fourth and fifth central moments of the shower-depth-maximum (Xmax) distribution can discriminate among leading hadronic interaction models.
desk verdict A promising higher-moment sensitivity study for hadronic models, but the quoted rejection CLs rely on an uncalibrated chi-square that ignores the fitted composition. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is the comparison of the first $n$ central moments of the Xmax distribution, $z_1, \dots, z_n$ (mean, variance, skewness, kurtosis, and beyond), between simulated predictions and measured data. The mass composition is a 26-component vector $w$ of primary fractions (proton through iron), fitted by maximizing a log-likelihood built from bootstrap-estimated multivariate normal distributions of the moments; the rejection confidence level is then computed from a $\chi^2$ statistic comparing the best-fit moment vector to the measured one.
What would settle it
A Monte Carlo calibration: simulate many pseudo-datasets from the best-fit composition, refit the composition and evaluate the same test statistic each time; if its 95th percentile lies below the $\chi^2_n$ 95th percentile, the reported rejection confidence levels for EPOS and Sibyll are overstated.
Extended reading notes
Core claim
The paper claims that the first three central moments of the Xmax distribution are not enough to challenge the leading hadronic models, but the fourth and fifth moments make the predictions of EPOS and Sibyll statistically incompatible with Pierre Auger data once the statistics are scaled upward. Concretely, including five moments yields a rejection confidence level above 95% at a statistics multiplier of f = 5 for both models, and above 90% at f ≈ 10 with four moments. The paper stresses that these are projections based on fixed central values of the measured moments, so they demonstrate sensitivity rather than a definitive incompatibility, but they indicate that the full Auger dataset would have sufficient discriminating power.
Load-bearing premise
The paper assumes that its test statistic follows a $\chi^2$ distribution with $n$ degrees of freedom, even though the 26 composition fractions are fitted to the same measured moments that are then tested; with more fitted parameters than moments, the post-fit discrepancy is likely smaller than the assumed distribution, so the reported confidence levels are not calibrated.
Editorial extensions
If this is right
- With the first three Xmax moments, EPOS and Sibyll fit the Pierre Auger open data well even when the sample is extrapolated to twenty times its size.
- Adding the fourth moment makes the rejection confidence grow with statistics, passing roughly 90% near f = 10.
- Adding the fifth moment pushes the rejection confidence above 95% already at f = 5 for both EPOS and Sibyll.
- QGSJetII-04 and QGSJet01 are already disfavored using only the first one or two moments.
- If the measured moment values hold as statistics grow, the full Auger dataset would be enough to discriminate among the four models.
Reading between the lines
- A proper calibration of the test statistic, e.g. by parametric bootstrap, could change the claimed confidence levels; this is the most direct way to test whether the f = 5 rejection survives.
- The same moment-based comparison could be turned into a tuning target for next-generation hadronic models: matching the fourth and fifth moments of Xmax may be a stronger constraint than matching only the mean and variance.
- If higher moments of Xmax prove to be this sensitive, other shower observables with similarly non-Gaussian tails might offer additional discriminators for hadronic models.
- The scaling procedure assumes the central values of the measured moments stay fixed as f grows; any energy-dependent systematic that shifts those central values with statistics would change the projection.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper tests the compatibility of four hadronic interaction models (EPOS, Sibyll, QGSJetII-04, QGSJet01) with Pierre Auger Open Data by comparing the first n central moments of the Xmax distribution. For each model, a 26-component mass composition w* is fitted via a bootstrap-based likelihood, and a rejection confidence level is computed from the post-fit difference between simulated and measured moments using Eq. (4), which is asserted to follow a chi-square distribution with n degrees of freedom. The main quantitative result is that for EPOS and Sibyll, adding the fourth and fifth moments (n = 4, 5) makes the rejection CL grow with the statistical multiplier f, reaching CL >= 95% already at f = 5. The paper also studies energy-spectrum reweighting, concluding the effect is negligible in a narrow energy bin, and reports rejection results for the QGSJet models in an appendix.
Significance. If the statistical calibration were sound, this would be a valuable use of public Auger data to discriminate among current hadronic interaction models with a modest amount of new methodology. The paper has clear strengths: it uses public data and CORSIKA simulations, includes all 26 primaries up to iron, propagates systematic and statistical uncertainties through a bootstrap procedure, and makes an explicit projection to larger statistics. The energy-reweighting study in Sect. III is a useful systematic check. However, the central rejection CLs rest on an uncalibrated chi-square assumption, so the quantitative claims—in particular 'CL >= 95% already at f = 5'—are not yet supported. The qualitative tension visible in the pulls may survive recalibration, but the specific confidence levels and the projection to the full Auger dataset require a corrected or empirically calibrated test statistic.
major comments (3)
- [II.C, Eq. (4)] The statement that z_s - z_m follows N(mu_s - mu_m, Sigma_s + Sigma_m) and that Eq. (4) follows a chi-square distribution with n degrees of freedom is not justified, because mu_s = mu_s(w*) and Sigma_s = Sigma_s(w*) are evaluated at the composition w* obtained by maximizing the same log-likelihood against the measured moments z_m. The composition vector has 26 components (25 independent fractions), so for n = 3, 4, 5 the number of fitted composition parameters exceeds the number of moments. Fitting parameters to the same data generally reduces the effective number of degrees of freedom of a quadratic goodness-of-fit statistic and introduces correlation between mu_s and z_m, so the covariance of their difference is not simply Sigma_s + Sigma_m. The paper neither subtracts the number of fitted parameters nor calibrates the distribution of Eq. (4) with pseudo-experiments. Consequently, the right-tail probabilities reported in Figs. 3 and 4—including the headline 'CL >= 95% already at f = 5 for both models'—are uncalibrated and do not currently have the stated frequentist meaning. A calibration procedure that repeats the full composition fit on Monte Carlo pseudo-data is needed before these quantitative claims can be accepted.
- [III, Figs. 3 and 4] The projection to larger statistics by the factor f assumes that the central values of the measured moments remain fixed while only their covariance shrinks. Under the null hypothesis that a hadronic model is correct, a new realization of a larger dataset would have different central values, and the composition fit would absorb part of those fluctuations. The current procedure therefore computes an in-sample goodness-of-fit rather than an out-of-sample prediction. The paper's own caveat in Sec. IV ('this does not mean that these models will in fact be incompatible once more data is analyzed') is appropriate, but it is in tension with the abstract and Sect. III, where the f = 5 result is presented as evidence that the existing full PA dataset would suffice to reject these models. A proper treatment would either calibrate the post-fit statistic under the null or present the f-dependence only as a sensitivity projection, with the CL interpretation deferred to a calibrated analysis.
- [Appendix A, Figs. 6 and 8] The appendix's conclusions that QGSJetII-04 and QGSJet01 are already in strong tension with data for n = 1 or n = 2 moments rely on the same uncalibrated chi-square assumption as Eq. (4). In particular, the high CL values close to unity in Fig. 8 are computed without accounting for the fact that the composition w* is fitted to the same moments that enter the statistic. The df problem is even more pronounced for n = 1, where a single moment is compared after fitting 25 independent composition fractions. These appendix claims therefore inherit the calibration issue and should be either re-derived with a calibrated statistic or explicitly labeled as provisional.
minor comments (5)
- [II.A] The text uses 'P A Open Data' in several places; this should be 'PA Open Data' for consistency.
- [II.B, Eq. (2)] In Eq. (2), the denominator appears as a sum over u_i with a superscript 'l' in the typeset version; this should be simply the sum of u_i, matching the definition in the text.
- [III] In the sentence 'At f = 1, that is the actual PAOD, we have CL ~ 20%', the phrase 'the actual PAOD' is imprecise; use 'the current PAOD subsample' to avoid implying the full dataset.
- [Footnote 2] The statement that new model implementations (EPOS4, Sibyll*, QGSJetIII) 'are expected to be consistent' with the versions used here is an unsupported assertion; it would be helpful to cite a quantitative comparison or to soften the wording.
- [Fig. 5] The legend in Fig. 5 is hard to parse because the labels 'f = 1' and 'f = 20' are placed on a single line without separating the line styles for each panel; a per-panel legend would improve readability.
Circularity Check
The 'predicted' Xmax moments are post-fit values from a composition fitted to the same measured moments, so the Eq. (4) chi-square_n rejection CLs are in-sample and uncalibrated.
-
fitted input called prediction
[Sec. II.C, Eq. (4); also Abstract]
"The most probable composition, w∗, can then be computed by maximizing the log-likelihood in eq. (1) ... Finally, we compute the confidence level of rejecting a given hadronic model ... by comparing the distribution of moments for the best fit composition, p(z|w∗), to the measured one, P (z). ... χ2 n = (µs − µm)T (Σs + Σm)−1(µs − µm), (4) which follows a χ2 distribution with n degrees of freedom."
The moments µ_s(w*) in Eq. (4) are called 'predicted by the best-fit inferred compositions' in the Abstract, but w* is obtained by maximizing the likelihood in Eq. (1), which is built from the same measured moment distribution P(z) that Eq. (4) then tests. Thus µ_s(w*) is a post-fit quantity, not an independent prediction. With 26 composition fractions (25 independent) fitted to only n = 3, 4, or 5 moments, the fit absorbs part of the measured discrepancy, so z_s − z_m is not distributed as N(0, Σ_s + Σ_m) and Eq. (4) does not follow a chi-square distribution with n degrees of freedom. The reported rejection CLs, including 'CL ≳ 95% already at f = 5', are therefore uncalibrated in-sample goodness-of-fit measures rather than calibrated model-rejection probabilities.
full rationale
The central quantitative claim of the paper—that EPOS and Sibyll reach CL ≥ 95% at f = 5 when n = 5—rests on Eq. (4), which compares the best-fit simulated moments µ_s(w*) to the measured moments µ_m. But w* is fitted to those same measured moments via the likelihood in Eq. (1), so the 'predicted' moments are in-sample. The test statistic's assumed chi-square_n distribution ignores the reduction in effective degrees of freedom from fitting 25 independent composition fractions to n = 3–5 moments. This is a fitted-input-called-prediction circularity: the comparison is not an independent model prediction, and the stated CLs are not calibrated. The tension visible in the higher-moment pulls of Fig. 5 may survive a correct calibration, but the specific CL values in Figs. 3, 4, and 8 are not supported as stated. The paper's self-citations to [12–14] are methodological continuity and do not by themselves carry the conclusion; the circular step is the in-sample 'prediction' and the mis-specified test statistic. Score 6 reflects that the headline rejection probabilities reduce to a post-fit comparison with an unjustified distributional assumption.
Assumptions & free parameters
free parameters (2)
- Mass composition fractions w_p,...,w_Fe (26 components) =
Not tabulated; shown as cumulative fractions in Fig. 2
- Number of energy bins for P(E) estimation =
30 bins across [0.65, 1] EeV
assumptions (5)
- domain assumption The UHECR flux in the bin can be represented as a mixture of 26 pure primaries from proton to iron.
- domain assumption All primaries have identical energy spectra within the chosen energy interval.
- ad hoc to paper The post-fit discrepancy in Eq. (4) follows a chi-square distribution with n degrees of freedom.
- domain assumption New hadronic model implementations (EPOS4, Sibyll*, QGSJetIII) predict EAS observables consistent with the older models tested here.
- domain assumption Detector resolution smearing parameters fitted to current PA data remain valid for future larger datasets.
Cite this review
Pith. "Pith review of Ultra High Energy Cosmic Rays versus Models of High Energy Hadronic Interactions." pith.science (2026). https://pith.science/paper/LT7MMLJK
@misc{pith2026241110223,
author = {Pith},
title = {Pith review of: Ultra High Energy Cosmic Rays versus Models of High Energy Hadronic Interactions},
year = {2026},
howpublished = {\url{https://pith.science/paper/LT7MMLJK}},
note = {Machine review of arXiv:2411.10223}
}
read the original abstract
We evaluate the consistency of hadronic interaction models in the CORSIKA simulation package with publicly available fluorescence telescope data from the Pierre Auger Observatory. By comparing the first few central moments of the extended air shower depth maximum distributions, as extracted from measured events, to those predicted by the best-fit inferred compositions, we derive a statistical measure of the consistency of a given hadronic model with data. To mitigate possible systematic biases, we include all primaries up to iron, compensate for the differences between the measured and simulated energy spectra of cosmic rays and account for other known systematic effects. Additionally, we study the effects of including higher central moments in the fit and project our results to larger statistics.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
We denote theselectedeventsas {( ˜Ei, ˜Xmax,i, δ˜Xmax,i)}ℓ, with i = 1,
Randomly choose N events, with uniform prob- ability and allowing for repetitions. We denote theselectedeventsas {( ˜Ei, ˜Xmax,i, δ˜Xmax,i)}ℓ, with i = 1, . . . , N. Here ℓ is the current bootstrap step, while δ ˜Xmax,i is the total systematic uncertainty of ˜Xmax,i
-
[2]
Estimate the energy density,P ( ˜Ei), and compute the weight for each event,{ui} = 1/P ( ˜Ei)
-
[3]
Compute the moments: zℓ 1 = PN i=1 ˜Xmax,i + δ ˜Xmax,i ϵi ui PN i=1 ul i , (2) zℓ n = PN i=1 ˜Xmax,i + δ ˜Xmax,i ϵi − zl 1 n ui PN i=1 ui , (3) where ϵi are samples from a normal distribution with mean 0 and standard deviation1
-
[4]
Repeat the previous steps forℓ = 1, . . . , B. The procedure to compute central moments from data remains the same as described in Ref. [12]. For simulated events, this procedure needs to be car- ried out for each primary separately, withN = Nsim = 2000 the total number of events in the simulated dataset; the moments of each primary then receive a differe...
work page 2000
-
[5]
≃ 0.99 with the PAOD statistics, and only decreases to λ(f = 5) ≃ 0.97. Conversely, for the choice of a larger energy bin, Ei ∈ [0.65, 5] EeV, the log-likelihood ratio is already λ(f = 1) ≃ 0.29 with current statistics, and goes down to λ(f = 5) ≃ 0.1 assuming a dataset five times larger. We thus conclude, that while the ef- fect is certainly important fo...
-
[6]
Ultra-High Energy Heavy Nuclei Propagation in Extragalactic Magnetic Fields
G. Bertone, C. Isola, M. Lemoine, and G. Sigl, Phys. Rev. D 66, 103003 (2002), arXiv:astro-ph/0209192
work page Pith review arXiv 2002
- [7]
-
[8]
S. Andringa, R. Conceicao, and M. Pimenta, Astropart. Phys. 34, 360 (2011)
work page 2011
Show all 32 references
-
[9]
Aab et al
A. Aab et al. (Pierre Auger), Nucl. Instrum. Meth. A 798, 172 (2015), arXiv:1502.01323 [astro-ph.IM]
2015 arXiv
-
[10]
Stasielak (Pierre Auger), in 9th International Conference on New Frontiers in Physics (2021) arXiv:2110.09487 [astro-ph.HE]
J. Stasielak (Pierre Auger), in 9th International Conference on New Frontiers in Physics (2021) arXiv:2110.09487 [astro-ph.HE]
2021 arXiv
-
[11]
Kawai, S
H. Kawai, S. Yoshida, H. Yoshii, K. Tanaka, F. Co- hen, M. Fukushima, N. Hayashida, K. Hiyama, D. Ikeda, E. Kido, Y. Kondo, T. Nonaka, M. Ohnishi, H. Ohoka, S. Ozawa, H. Sagawa, N. Sakurai, T. Shibata, H. Shi- modaira, M. Takeda, A. Taketa, M. Takita, H. Tokuno, R. Torii, S. U...
2008
-
[12]
Ostapchenko, Phys
S. Ostapchenko, Phys. Rev. D 109, 094019 (2024), arXiv:2403.16106 [hep-ph]
2024 arXiv
-
[13]
A. A. Halimet al. (Pierre Auger), JCAP01, 022 (2024), arXiv:2305.16693 [astro-ph.HE]
2024 arXiv
-
[14]
Blazek, J
J. Blazek, J. Vicha, J. Ebr, R. Ulrich, T. Pierog, and P. Travnicek, PoSICRC2021, 441 (2021)
2021
- [15]
-
[16]
Cazon (EAS-MSU, IceCube, KASCADE Grande, NEVOD-DECOR, Pierre Auger, SUGAR, Telescope Ar- ray, Yakutsk EAS Array), PoSICRC2019, 214 (2020), arXiv:2001.07508 [astro-ph.HE]
L. Cazon (EAS-MSU, IceCube, KASCADE Grande, NEVOD-DECOR, Pierre Auger, SUGAR, Telescope Ar- ray, Yakutsk EAS Array), PoSICRC2019, 214 (2020), arXiv:2001.07508 [astro-ph.HE]
2020 arXiv
-
[17]
Bortolato, J
B. Bortolato, J. F. Kamenik, and M. Tammaro, Phys. Rev. D 108, 022004 (2023), arXiv:2212.04760 [astro- ph.HE]
2023 arXiv
-
[18]
Bortolato, J
B. Bortolato, J. F. Kamenik, and M. Tammaro, Phys. Rev. D 109, 043023 (2024), arXiv:2304.11197 [astro- ph.HE]
2024 arXiv
-
[19]
Bortolato, J
B. Bortolato, J. F. Kamenik, and M. Tammaro, (2024), arXiv:2409.06841 [astro-ph.HE]
2024 arXiv
-
[20]
Pierre auger observatory 2021 open data,
T. P. A. Collaboration, “Pierre auger observatory 2021 open data,” (2021)
2021
-
[21]
Aabet al
A. Aabet al. (Pierre Auger), Phys. Rev. D90, 122005 (2014), arXiv:1409.4809 [astro-ph.HE]
2014 arXiv
-
[22]
CORSIKA: A Monte Carlo code to simulate extensive air showers,
D. Heck, J. Knapp, J. N. Capdevielle, G. Schatz, and T. Thouw, “CORSIKA: A Monte Carlo code to simulate extensive air showers,” (1998)
1998
-
[23]
Pierog, I
T. Pierog, I. Karpenko, J. M. Katzy, E. Yatsenko, and K. Werner, Phys. Rev. C 92, 034906 (2015), arXiv:1306.0121 [hep-ph]
2015 arXiv
-
[24]
Riehn, A
F. Riehn, A. Fedynitch, and R. Engel, Astropart. Phys. 160, 102964 (2024), arXiv:2404.02636 [hep-ph]
2024 arXiv
-
[25]
Ostapchenko, Phys
S. Ostapchenko, Phys. Rev. D 83, 014018 (2011), arXiv:1010.1869 [hep-ph]
2011 arXiv
-
[26]
N. N. Kalmykov and S. S. Ostapchenko, Phys. Atom. Nucl. 56, 346 (1993). 8
1993
- [27]
-
[28]
Efron and R
B. Efron and R. J. Tibshirani, An Introduction to the Bootstrap, Monographs on Statistics and Applied Probability No. 57 (Chapman & Hall/CRC, Boca Raton, Florida, USA, 1993)
1993
-
[29]
Skilling, AIP Conference Proceedings735, 395 (2004), https://aip.scitation.org/doi/pdf/10.1063/1.1835238
J. Skilling, AIP Conference Proceedings735, 395 (2004), https://aip.scitation.org/doi/pdf/10.1063/1.1835238
2004 doi
-
[30]
Buchner, Statistics and Computing26, 383 (2014)
J. Buchner, Statistics and Computing26, 383 (2014)
2014
-
[31]
Collaborative nested sampling: Big data vs. complex physical models,
J. Buchner, “Collaborative nested sampling: Big data vs. complex physical models,” (2017)
2017
-
[32]
Ultranest – a robust, general purpose bayesian inference engine,
J. Buchner, “Ultranest – a robust, general purpose bayesian inference engine,” (2021). Appendix A: Results for QGSJet models 0 5 10 15 20 f 0.6 0.7 0.8 0.9 1.0 Fraction of protons QGSJetII-04 0.96 0.97 0.98 0.99 1.00 CL n = 1 n = 2 0 5 10 15 20 f 0.4 0.6 0.8 1.0 Fraction of pr...
2021
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.