Variability in Cosmological Hydrodynamical Simulations: how Stochastic Processes, Numerical Effects, and Reproducibility Limits impact Predictability
Pith reviewed 2026-06-27 00:08 UTC · model grok-4.3
The pith
Four identical galaxy-cluster simulations produce 10-25% differences in galaxy dark matter and stellar masses.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
By comparing matched galaxies across four controlled realizations of the same zoom-in simulation, the work shows that stochastic star formation and feedback produce 10-25% variations in galaxy dark matter and stellar masses. Feedback acts to regulate this scatter, reducing variability in stellar and black hole masses, while black hole physics amplifies it. The run-to-run differences indicate a noise-dominated but statistically reproducible regime at low resolution.
What carries the argument
A mixed linear model applied to properties of matched galaxies across repeated runs, which separates run-to-run variation from within-run noise under fixed compiler and hardware conditions.
If this is right
- Feedback regulation reduces scatter in both stellar and black hole masses.
- Inclusion of black hole physics increases the amplitude of run-to-run variability.
- The observed variations remain above the shot-noise floor while the overall results stay statistically reproducible.
- Low-resolution baseline estimates are established for future reproducibility studies.
Where Pith is reading between the lines
- Reproducibility tests in other codes or at higher resolution could reveal whether numerical resolution changes the balance between stochastic and deterministic effects.
- Observational comparisons of galaxy populations should incorporate an intrinsic simulation scatter floor when judging model fidelity.
- The same statistical separation of noise sources could be applied to suites of simulations that vary initial conditions rather than random seeds.
Load-bearing premise
The four realizations differ only by the intended stochastic processes and the mixed linear model cleanly isolates run-to-run variation from within-run noise.
What would settle it
Running additional realizations and finding that the measured scatter falls to the pure shot-noise floor or deviates systematically from the mixed linear model predictions.
Figures
read the original abstract
Cosmological hydrodynamical simulations are powerful tools for studying galaxy formation, yet their predictive precision is limited by stochastic variability and numerical uncertainty. We quantify this variability using four identical realizations of a zoom-in galaxy-cluster simulation evolved with \textsc{OpenGadget3} under tightly controlled compiler, library, and hardware settings. Variability is measured through the properties of matched galaxies across repeated runs, including a mixed linear model that separates run-to-run variation from within-run noise. Variations of approximately $10$-$25\%$ are found in galaxy dark matter and stellar masses for the baseline simulations. The variability trending above the shot-noise floor reflects the combined effects of stochastic star formation and feedback regulation, and is further amplified when black hole physics is included. Furthermore, our results indicate that feedback acts to regulate variability, reducing scatter in both stellar and black hole masses. Our inference from run-to-run variation indicates a noise-dominated regime that remains statistically reproducible, despite individual realization differences. These results establish baseline, noise-dominated variability estimates at low resolution, demonstrate how feedback modulates predictability, and provide a statistical framework for future studies of reproducibility in cosmological hydrodynamical simulations.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper claims that four identical realizations of a zoom-in galaxy-cluster simulation run with OpenGadget3 under tightly controlled compiler/library/hardware settings exhibit 10-25% run-to-run variability in matched galaxy dark-matter and stellar masses. A mixed linear model is used to separate stochastic run-to-run variation from within-run noise; the variability exceeds shot noise, is modulated by feedback (which reduces scatter), and is further increased when black-hole physics is included, yet the overall regime remains noise-dominated and statistically reproducible.
Significance. If the central variability estimates hold, the work supplies a much-needed quantitative baseline for reproducibility limits in cosmological hydrodynamical simulations at low resolution. The controlled experimental design and explicit statistical framework for decomposing sources of variation are genuine strengths that future studies can build upon.
major comments (2)
- [Abstract / mixed linear model section] Abstract and the mixed-linear-model description: the headline 10-25% variability figures are obtained from a mixed linear model fitted to only four realizations. With four groups the between-run variance component is estimated from three degrees of freedom; no residual diagnostics, recovery tests on synthetic data, or sensitivity checks to the within-run error structure are referenced, leaving open the possibility that modest misspecification inflates or deflates the reported percentages by amounts comparable to the claimed signal.
- [Galaxy matching and data selection] Galaxy-property section: the criteria used to match galaxies across the four runs, the rules for excluding unmatched or poorly resolved objects, and the precise definition of the within-run noise term are not numerically specified. These choices directly affect the input data to the mixed model and therefore the 10-25% claim; without them the result cannot be independently verified.
minor comments (1)
- [Abstract] The abstract states that feedback “acts to regulate variability” but does not quantify the reduction in scatter (e.g., by what factor or in which mass bins).
Simulated Author's Rebuttal
We thank the referee for the constructive comments on our manuscript. We address each major point below and indicate where revisions will be made.
read point-by-point responses
-
Referee: [Abstract / mixed linear model section] Abstract and the mixed-linear-model description: the headline 10-25% variability figures are obtained from a mixed linear model fitted to only four realizations. With four groups the between-run variance component is estimated from three degrees of freedom; no residual diagnostics, recovery tests on synthetic data, or sensitivity checks to the within-run error structure are referenced, leaving open the possibility that modest misspecification inflates or deflates the reported percentages by amounts comparable to the claimed signal.
Authors: We agree that four realizations provide only three degrees of freedom for the between-run variance component and that this is a limitation of the experimental design. The mixed linear model follows standard practice for decomposing hierarchical variance in repeated simulations, and the reported variability exceeds the shot-noise floor by a clear margin. Nevertheless, we will add residual diagnostics, a brief recovery test on synthetic data with known variance components, and sensitivity checks to the within-run error structure in a revised Methods section to quantify any potential impact of misspecification. revision: yes
-
Referee: [Galaxy matching and data selection] Galaxy-property section: the criteria used to match galaxies across the four runs, the rules for excluding unmatched or poorly resolved objects, and the precise definition of the within-run noise term are not numerically specified. These choices directly affect the input data to the mixed model and therefore the 10-25% claim; without them the result cannot be independently verified.
Authors: The manuscript describes the matching procedure and exclusion criteria in the Galaxy Matching subsection, but we accept that the numerical thresholds (e.g., maximum separation, minimum particle number, and the exact formulation of the within-run noise term) are stated only qualitatively. We will insert the precise numerical values and the explicit formula for the within-run noise term in the revised text so that the input data to the mixed model can be reproduced exactly. revision: yes
Circularity Check
No significant circularity; variability is measured directly from simulation outputs
full rationale
The paper's central result is an empirical measurement of run-to-run variability in galaxy properties across four controlled realizations, using a mixed linear model to decompose variance components. No load-bearing step reduces the reported 10-25% figures to a quantity defined by the authors' own prior choices, fitted parameters, or self-citations. The model is applied as a standard statistical tool to the simulation data rather than creating a self-referential loop, and the abstract and methods frame the outcome as a direct observation of stochastic effects rather than a derived prediction equivalent to its inputs. The derivation chain remains self-contained against external benchmarks.
Axiom & Free-Parameter Ledger
axioms (1)
- domain assumption The four realizations are identical except for stochastic star formation and feedback processes.
Reference graph
Works this paper leans on
-
[1]
On the onset of stochasticity in cold dark matter cosmological simulations , volume =. Monthly Notices of the Royal Astronomical Society , author =. 2008 , pages =. doi:10.1111/j.1365-2966.2008.13250.x , abstract =
-
[2]
The cosmological simulation code gadget-2 , volume =. Monthly Notices of the Royal Astronomical Society , author =. 2005 , keywords =. doi:10.1111/j.1365-2966.2005.09655.x , abstract =
-
[3]
Seabold, Skipper and Perktold, Josef , editor =. Statsmodels:. 2010 , pages =. doi:10.25080/Majora-92bf1922-011 , booktitle =
-
[4]
Monthly Notices of the Royal Astronomical Society , author =
The effects of stellar and. Monthly Notices of the Royal Astronomical Society , author =. 2024 , pages =. doi:10.1093/mnras/stae2098 , abstract =
-
[5]
The Astrophysical Journal , author =
Cool. The Astrophysical Journal , author =. 2015 , note =. doi:10.1088/2041-8205/813/1/L17 , abstract =
-
[6]
2018, MNRAS, 473, 4077, doi: 10.1093/mnras/stx2656 Planck Collaboration, Ade, P
Simulating galaxy formation with the. Monthly Notices of the Royal Astronomical Society , author =. 2018 , pages =. doi:10.1093/mnras/stx2656 , abstract =
work page internal anchor Pith review doi:10.1093/mnras/stx2656 2018
-
[7]
Louys, Mireille and Tody, Doug and Dowler, Patrick and Durand, Daniel and Michel, Laurent and Bonnarel, Francos and Micol, Alberto and. Observation. doi:10.5479/ADS/bib/2017ivoa.spec.0509L , language =
-
[8]
The Astrophysical Journal , author =
Divergence of. The Astrophysical Journal , author =. 1990 , note =. doi:10.1086/169425 , abstract =
-
[9]
Monthly Notices of the Royal Astronomical Society 526(1), 616–644 (2023)
The cosmological simulation code. Monthly Notices of the Royal Astronomical Society , author =. 2023 , keywords =. doi:10.1093/mnras/stad2717 , abstract =
-
[10]
The dependence of the substellar initial mass function on the initial conditions for star formation , volume =. Monthly Notices of the Royal Astronomical Society , author =. 2004 , pages =. doi:10.1111/j.1365-2966.2004.07259.x , abstract =
-
[11]
The evolution of large-scale structure in a universe dominated by cold dark matter , volume =. The Astrophysical Journal , author =. 1985 , note =. doi:10.1086/163168 , abstract =
-
[12]
Quenching and morphological evolution due to circumgalactic gas expulsion in a simulated galaxy with a controlled assembly history , volume =. Monthly Notices of the Royal Astronomical Society , author =. 2021 , pages =. doi:10.1093/mnras/staa3643 , abstract =
-
[13]
A non-ideal magnetohydrodynamic gadget: simulating massive galaxy clusters , volume =. Monthly Notices of the Royal Astronomical Society , author =. 2011 , keywords =. doi:10.1111/j.1365-2966.2011.19523.x , abstract =
-
[14]
Astronomy & Astrophysics , author =
The. Astronomy & Astrophysics , author =. 2020 , note =. doi:10.1051/0004-6361/202038396 , abstract =
-
[15]
Monthly Notices of the Royal Astronomical Society , author =
Galactic outflow and diffuse gas properties at z > 1 using different baryonic feedback models , volume =. Monthly Notices of the Royal Astronomical Society , author =. 2015 , pages =. doi:10.1093/mnras/stu2340 , abstract =
-
[16]
A subresolution multiphase interstellar medium model of star formation and supernova energy feedback , volume =. Monthly Notices of the Royal Astronomical Society , author =. 2010 , note =. doi:10.1111/j.1365-2966.2010.16567.x , abstract =
-
[17]
Simulating realistic disk galaxies with a novel sub-resolution ISM model
Murante, Giuseppe and Monaco, Pierluigi and Borgani, Stefano and Tornatore, Luca and Dolag, Klaus and Goz, David , month = nov, year =. Simulating realistic disk galaxies with a novel sub-resolution. doi:10.48550/arXiv.1411.3671 , abstract =
work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.1411.3671
-
[18]
Simulating cosmic structure formation with the gadget-4 code , volume =. Monthly Notices of the Royal Astronomical Society , author =. 2021 , pages =. doi:10.1093/mnras/stab1855 , abstract =
-
[19]
2012, MNRAS, 421, 2002, doi: 10.1111/j.1365-2966.2012.20431.x
Improving convergence in smoothed particle hydrodynamics simulations without pairing instability , volume =. Monthly Notices of the Royal Astronomical Society , author =. 2012 , note =. doi:10.1111/j.1365-2966.2012.21439.x , abstract =
-
[20]
The Annals of Applied Probability , author =
A. The Annals of Applied Probability , author =. 1991 , note =. doi:10.1214/aoap/1177005878 , abstract =
-
[21]
Monthly Notices of the Royal Astronomical Society , author =
Quantifying the intrinsic variability due to randomness of the. Monthly Notices of the Royal Astronomical Society , author =. 2025 , pages =. doi:10.1093/mnras/staf1542 , abstract =
-
[22]
Monthly Notices of the Royal Astronomical Society , author =
Dust evolution in zoom-in cosmological simulations of galaxy formation , volume =. Monthly Notices of the Royal Astronomical Society , author =. 2021 , pages =. doi:10.1093/mnras/stab362 , abstract =
-
[23]
Monthly Notices of the Royal Astronomical Society , author =
Calibration of a star formation and feedback model for cosmological simulations with enzo , volume =. Monthly Notices of the Royal Astronomical Society , author =. 2020 , pages =. doi:10.1093/mnras/staa2318 , abstract =
-
[24]
Monthly Notices of the Royal Astronomical Society , author =
Impact of. Monthly Notices of the Royal Astronomical Society , author =. 2023 , note =. doi:10.1093/mnras/stac2110 , abstract =
-
[25]
The Astrophysical Journal , author =
Discreteness. The Astrophysical Journal , author =. 1993 , note =. doi:10.1086/186706 , urldate =
-
[26]
Galactic
Binney, James and Tremaine, Scott , year =. Galactic
-
[27]
The Astrophysical Journal Supplement Series , author =
Error. The Astrophysical Journal Supplement Series , author =. 1989 , note =. doi:10.1086/191343 , abstract =
-
[28]
Dynamical friction and the evolution of black holes in cosmological simulations:. Astronomy & Astrophysics , author =. 2024 , note =. doi:10.1051/0004-6361/202450021 , abstract =
-
[29]
aragagnin/g3read , copyright =
Ragagnin, Antonio , month = jul, year =. aragagnin/g3read , copyright =
-
[30]
Introducing the. Monthly Notices of the Royal Astronomical Society , author =. 2014 , note =. doi:10.1093/mnras/stu1536 , abstract =
-
[31]
Monthly Notices of the Royal Astronomical Society , author =
Velocity dispersion of brightest cluster galaxies in cosmological simulations , volume =. Monthly Notices of the Royal Astronomical Society , author =. 2021 , pages =. doi:10.1093/mnras/stab2518 , abstract =
-
[32]
Galaxy. The Astrophysical Journal , author =. 1991 , note =. doi:10.1086/170483 , abstract =
-
[33]
Monthly Notices of the Royal Astronomical Society , author =
Effects of. Monthly Notices of the Royal Astronomical Society , author =. 1974 , pages =. doi:10.1093/mnras/169.2.229 , abstract =
-
[34]
The. The Astrophysical Journal , author =. 1986 , note =. doi:10.1086/164050 , abstract =
-
[35]
2003, PASP, 115, 763, doi: 10.1086/376392
Galactic. Publications of the Astronomical Society of the Pacific , author =. 2003 , note =. doi:10.1086/376392 , abstract =
work page internal anchor Pith review doi:10.1086/376392 2003
-
[36]
Buchi neri e formazione stellare negli ammassi di galassie: una vista dalle simulazioni , shorttitle =
Bassini, Luigi , month = feb, year =. Buchi neri e formazione stellare negli ammassi di galassie: una vista dalle simulazioni , shorttitle =
-
[37]
Reviews of Modern Physics , year = 1986, month = jan, volume =
X-ray emission from clusters of galaxies , volume =. Reviews of Modern Physics , author =. 1986 , note =. doi:10.1103/RevModPhys.58.1 , abstract =
-
[38]
Proceedings of the IEEE , author =
The. Proceedings of the IEEE , author =. 2005 , note =. doi:10.1109/JPROC.2004.840301 , abstract =
-
[39]
Frigo, M. and Johnson, S.G. , month = may, year =. Proceedings of the 1998. doi:10.1109/ICASSP.1998.681704 , abstract =
-
[40]
Thermal conduction in cosmological. Monthly Notices of the Royal Astronomical Society , author =. 2004 , note =. doi:10.1111/j.1365-2966.2004.07801.x , abstract =
-
[41]
Cosmological Hydrodynamics with Adaptive Mesh Refinement: a new high resolution code called RAMSES
Teyssier, Romain , month = nov, year =. Cosmological. doi:10.48550/arXiv.astro-ph/0111367 , abstract =
work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.astro-ph/0111367
-
[42]
New Astronomy , author =. 2001 , keywords =. doi:10.1016/S1384-1076(01)00042-2 , abstract =
-
[43]
Annual Review of Astronomy and Astrophysics , author =
Formation of. Annual Review of Astronomy and Astrophysics , author =. 2012 , note =. doi:10.1146/annurev-astro-081811-125502 , abstract =
-
[44]
Annual Review of Astronomy and Astrophysics , author =
Cosmological. Annual Review of Astronomy and Astrophysics , author =. 2011 , note =. doi:10.1146/annurev-astro-081710-102514 , abstract =
-
[45]
A parallel tree code , volume =. New Astronomy , author =. 1996 , keywords =. doi:10.1016/S1384-1076(96)00009-7 , abstract =
-
[46]
Monthly Notices of the Royal Astronomical Society , author =
Extended. Monthly Notices of the Royal Astronomical Society , author =. 2021 , pages =. doi:10.1093/mnras/stab2845 , abstract =
-
[47]
Annual Review of Astronomy and Astrophysics, vol
Cosmic. Annual Review of Astronomy and Astrophysics , author =. 2014 , note =. doi:10.1146/annurev-astro-081811-125615 , abstract =
work page internal anchor Pith review doi:10.1146/annurev-astro-081811-125615 2014
-
[48]
The massive end of the luminosity and stellar mass functions: dependence on the fit to the light profile , volume =. Monthly Notices of the Royal Astronomical Society , author =. 2013 , pages =. doi:10.1093/mnras/stt1607 , abstract =
work page internal anchor Pith review doi:10.1093/mnras/stt1607 2013
-
[49]
The Astrophysical Journal Supplement Series , author =. 2014 , note =. doi:10.1088/0067-0049/211/2/19 , abstract =
-
[50]
doi:https://doi.org/10.1038/s41592-019-0686-2 , adsurl =
Nature Methods , author =. 2020 , note =. doi:10.1038/s41592-019-0686-2 , abstract =
-
[51]
Computing in Science and Engineering , keywords =
Matplotlib:. Computing in Science & Engineering , author =. 2007 , note =. doi:10.1109/MCSE.2007.55 , abstract =
-
[52]
Array programming with. Nature , author =. 2020 , note =. doi:10.1038/s41586-020-2649-2 , abstract =
-
[53]
Gelman, Andrew , month = nov, year =. Data. Cambridge. doi:10.1017/CBO9780511790942 , note =
-
[54]
Evrard, A. E. and MacFarland, T. J. and Couchman, H. M. P. and Colberg, J. M. and Yoshida, N. and White, S. D. M. and Jenkins, A. R. and Frenk, C. S. and Pearce, F. R. and Peacock, J. A. and Thomas, P. A. , month = mar, year =. Galaxy. doi:10.48550/arXiv.astro-ph/0110246 , abstract =
work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.astro-ph/0110246
-
[55]
, volume =
On the problem of relaxation of stellar systems. , volume =. Akademiia Nauk SSSR Doklady , author =. 1984 , note =
1984
-
[56]
Simulations of the formation, evolution and clustering of galaxies and quasars , volume =. Nature , author =. 2005 , note =. doi:10.1038/nature03597 , abstract =
-
[57]
Languignon, David and Le Petit, Franck and Rodrigo, Carlos and Lemson, Gerard and Molinaro, Marco and Wozniak, Hervé and Languignon, David and Le Petit, Franck , month = mar, year =. Simulation. doi:10.5479/ADS/bib/2017ivoa.spec.0320L , language =
-
[58]
Lemson, Gerard and Wozniak, Herve and Bourges, Laurent and Cervino, Miguel and Gheller, Claudio and Gray, Norman and LePetit, Franck and Louys, Mireille and Ooghe, Benjamin and Wagner, Rick and Lemson, Gerard and Wozniak, Herve , month = may, year =. Simulation. doi:10.5479/ADS/bib/2012ivoa.spec.0503L , language =
-
[59]
Dowler, Patrick and Rixon, Guy and Tody, Doug and Demleitner, Markus , month = aug, year =. Table
-
[60]
doi:10.5479/ADS/bib/2018ivoa.spec.0625P , language =
-
[61]
and Ohishi, Masatoshi and O'Mullane, William and
Osuna, Pedro and Ortiz, Iñaki and Lusted, Jeff and Dowler, Pat and Szalay, Alexander and Shirasaki, Yuji and Nieto-Santisteban, Maria A. and Ohishi, Masatoshi and O'Mullane, William and. doi:10.5479/ADS/bib/2008ivoa.spec.1030O , language =
-
[62]
doi:10.5479/ADS/bib/2013ivoa.spec.0920O , language =
-
[63]
doi:10.5479/ADS/bib/2016ivoa.spec.1024H , language =
Universal. doi:10.5479/ADS/bib/2016ivoa.spec.1024H , language =
-
[64]
doi:10.5479/ADS/bib/2017ivoa.spec.0524G , language =
Graham, Matthew and Rixon, Guy and Dowler, Patrick and Major, Brian and. doi:10.5479/ADS/bib/2017ivoa.spec.0524G , language =
-
[65]
doi:10.5479/ADS/bib/2012ivoa.spec.0827D , language =
-
[66]
doi:10.5479/ADS/bib/2010ivoa.spec.1202P , language =
-
[67]
doi:10.5479/ADS/bib/2010ivoa.spec.0218P , language =
-
[68]
doi:10.5479/ADS/bib/2014ivoa.spec.1208D , language =
-
[69]
IVOA Recommendation: Simple Spectral Access Protocol Version 1.1
Tody, Doug and Dolensky, Markus and McDowell, Jonathan and Bonnarel, Francois and Budavari, Tamas and Busko, Ivo and Micol, Alberto and Osuna, Pedro and Salgado, Jesus and Skoda, Petr and Thompson, Randy and Valdes, Frank and group, the Data Access Layer working , month = mar, year =. doi:10.48550/arXiv.1203.5725 , abstract =
work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.1203.5725
-
[70]
doi:10.5479/ADS/bib/2015ivoa.spec.1223D , abstract =
-
[71]
Derriere, Sébastien and Gray, Norman and Mann, Robert and Martinez, Andrea Preite and McDowell, Jonathan and Glynn, Thomas Mc and Ochsenbein, François and Osuna, Pedro and Rixon, Guy and Williams, Roy , month = oct, year =
-
[72]
doi:10.5479/ADS/bib/2018ivoa.spec.0723D , language =
Registry. doi:10.5479/ADS/bib/2018ivoa.spec.0723D , language =
-
[73]
and Durand, D
Williams, Roy and Ochsenbein, Francois and Davenhall, C. and Durand, D. and Fernique, P. and Giaretta, David and Hanisch, R. and Mcglynn, T. and Szalay, Alexander and Wicenec, Andreas , month = jan, year =
-
[74]
The. Monthly Notices of the Royal Astronomical Society , author =. 2023 , pages =. doi:10.1093/mnras/stac3620 , abstract =
-
[75]
The F AIR guiding principles for scientific data management and stewardship
The. Scientific Data , author =. 2016 , note =. doi:10.1038/sdata.2016.18 , abstract =
-
[76]
Astronomy and Computing , author =
A web portal for hydrodynamical, cosmological simulations , volume =. Astronomy and Computing , author =. 2017 , keywords =. doi:10.1016/j.ascom.2017.05.001 , abstract =
-
[77]
arXiv e-prints , author =
Halo and. arXiv e-prints , author =. 2006 , note =
2006
-
[78]
Astronomische Nachrichten , keywords =
The. Astronomische Nachrichten , author =. 2013 , note =. doi:10.1002/asna.201211900 , abstract =
-
[79]
Batygin, Konstantin and Laughlin, Gregory , month = apr, year =. On the. doi:10.48550/arXiv.0804.1946 , abstract =
work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.0804.1946 1946
-
[80]
Annals of the New York Academy of Sciences , author =
Chaos in. Annals of the New York Academy of Sciences , author =. 1998 , note =. doi:10.1111/j.1749-6632.1998.tb11266.x , abstract =
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.