REVIEW 4 major objections 5 minor 63 references
Probabilistic cosmological inference on HI tomographic data
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read A two-stage deep network — 3D convolutional encoder plus Masked Autoregressive Flow — recovers ($\Omega_{\rm m}$, $\sigma_8$) from $128^3$ 21-cm boxes with $R^2 \geq 0.91$ and calibrated uncertainties.
desk verdict A credible proof-of-concept for SBI on 3D HI tomographic data, but the calibration claim rests on a metric that is not a coverage test and the abstract overstates OOD robustness. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the encoder–posterior pair. The encoder is a 3D convolutional network with four stages of 3D convolution, batch normalization, ReLU, and max pooling, ending in adaptive average pooling, which maps each $128^3$ neutral-hydrogen box to a 1024-dimensional latent vector; it is pre-trained jointly with a two-layer regressor under the loss of Eq. (1), which balances the log-mean-squared prediction error against the log-squared error of the predicted per-parameter variance. The posterior estimator is a Masked Autoregressive Flow, a normalizing flow that factors a density into Gaussian conditionals and is invertible with an analytic Jacobian, used inside the Automatic Posterior Transformation framework, which trains the flow to map parameters and data (here, latent codes) to an approximation of $p(\Omega_{\rm m}, \sigma_8 \mid \text{latent code})$. At test time, 10,000 samples are drawn from the fitted posterior; the median is the point prediction and the 68% and 95% percentiles give the epistemic credible intervals. The separability of the codes under UMAP is presented as evidence that the regression pre-training produces meaningful structure.
What would settle it
On the held-out boxes, train the same MAF posterior directly on the raw $128^3$ cubes or on a summary known to be sufficient (e.g., the full power spectrum plus bispectrum) and compare its credible intervals with those obtained from the latent codes, or run an expected-coverage test across the full $(\Omega_{\rm m}, \sigma_8)$ prior: if coverage deviates strongly from the nominal 68% in any region, or if the latent-code intervals are systematically wider than the field-level ones, the encoder's information sufficiency is falsified.
Extended reading notes
Core claim
In the paper's own terms, the claim is that the full 3D neutral-hydrogen field at $z \approx 1$ carries enough information to infer $\Omega_{\rm m}$ and $\sigma_8$ with a coefficient of determination $R^2 \geq 0.91$ for both parameters, using only the 1024-dimensional codes produced by a regression-trained encoder. For an in-distribution test set, most ground-truth parameter values fall inside the 68% credible interval of the MAF-based posterior, and the Uncertainty Calibration Error ranges from about 2% to 13% depending on parameter and simulation, so the posterior uncertainty is described as reasonably calibrated. The paper also claims that the latent codes are separable in feature space, with points of similar parameter values clustering together, and that the featurizer generalizes: when the encoder built from one simulation compresses boxes from the other simulation, the posterior estimator still achieves $R^2 \geq 0.84$, and when it compresses boxes from the same simulation at $z = 0.5$ (the encoder being built at $z = 1$), $R^2 \geq 0.83$ holds. These results are put forward as a proof of concept for analyzing data from future 21-cm tomographic surveys.
Load-bearing premise
The load-bearing premise is that the 1024-dimensional latent codes produced by an encoder trained only to predict $\Omega_{\rm m}$ and $\sigma_8$ retain all the information the density estimator needs for a calibrated posterior; if the regression objective discards features relevant to the posterior's shape or tails, the reported calibration is not guaranteed.
Editorial extensions
If this is right
- If correct, the pipeline supplies a likelihood-free route to cosmological constraints from 21-cm tomographic data, usable wherever a simulation-based training set exists.
- The cross-simulation result ($R^2 \geq 0.84$) implies that a featurizer trained on one survey's simulations may compress another survey's data with only modest loss, as long as the voxel distributions are not too far apart.
- The percent-level calibration means the $1\sigma$ intervals can be read as physical error bars on a single inference without an additional recalibration step.
- The linear test's $R^2 \geq 0.80$ shows the codes are nearly linearly informative, so the encoder's compression does not destroy the cosmological content.
- The worst transfer case ($R^2 \approx 0.6$ for $\sigma_8$ from $z = 0$ boxes encoded by a $z = 1$ featurizer) sets a practical redshift horizon for reusing one encoder.
Reading between the lines
- The paper does not test whether the 1024-dimensional codes are sufficient summaries; a direct comparison with a field-level posterior or with a summary known to be sufficient would indicate whether the regression pre-training discards posterior-relevant information.
- The cross-simulation robustness is demonstrated on two simulations of similar resolution and physics, so the transferability to real radio maps with noise and foregrounds remains open; the author explicitly flags domain adaptation as the next step.
- The degradation at $z = 0$ suggests the useful range of a single featurizer is roughly $\Delta z \lesssim 0.5$; training on a multi-redshift set would be a natural way to extend the method, and the paper leaves that unexplored.
- A survey-level application would combine the posteriors from many boxes; the paper does not address how to handle correlations between boxes when they are assembled into a single posterior.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a two-stage pipeline for cosmological inference from 21-cm HI tomographic data cubes. In the first stage, a 3D convolutional encoder (Table 1) is trained with a supervised regression loss (Eq. 1) to predict Ωm and σ8 from 128^3 CAMELS HI boxes at z=1, yielding 1024-dimensional latent codes. In the second stage, a Masked Autoregressive Flow within the Automatic Posterior Transformation framework is trained on these latent codes to approximate the posterior p(Ωm, σ8 | data). The authors report R² ≥ 0.91 for same-distribution APT results (SIMBA→SIMBA, TNG→TNG), compare against GP/RF/linear baselines, and test robustness when the encoder is applied to out-of-distribution data (cross-simulation, and z=0/0.5 from the same simulation). The paper claims that the posterior uncertainties are reasonably calibrated, based on the Uncertainty Calibration Error (UCE) of Eq. (6), and that most test truths fall within the 1σ credible intervals.
Significance. If the claims hold, this is a useful proof-of-concept for field-level, likelihood-free inference from 3D 21-cm tomographic data, with potential applications to upcoming HI surveys. The use of two CAMELS simulations, held-out test sets with clean encoder/APT separation, a UMAP-based check of latent separability, and comparison against several regression baselines are concrete strengths. The main gaps are statistical: the calibration claim is based on UCE, which is not a coverage test, and the abstract's out-of-distribution performance claim (R² ≥ 0.80 in general) is contradicted by the paper's own z=0 SIMBA σ8 result. These issues are fixable within the manuscript's scope.
major comments (4)
- [§4.2, §4.3, Eq. (6)] The central calibration claim is not established. UCE, as defined in Eq. (6), compares binned mean squared error ξ² to binned mean epistemic variance ζ². This is a variance-calibration diagnostic, not a coverage metric: a predictor with a constant bias and variance equal to its mean squared error can have UCE near zero while its 68% intervals contain almost none of the true values. The abstract and §4.2 state that 'most of the ground truth ... fall within 1σ uncertainty', but this is supported only by visual inspection of Figures 3 and 4; no empirical coverage fraction, prediction-interval coverage probability, or joint 2D coverage statistic is reported. Because calibrated posterior inference is a headline contribution, please add quantitative coverage tests, such as the fraction of test truths within the 68% and 95% marginal credible intervals and a 2D joint coverage metric, and report them for all four main setups.
- [Abstract, §4.6, Figure 9] The abstract's claim that out-of-distribution inference achieves 'R² ≥ 0.80 in general' is contradicted by the paper's own results: §4.6 states that 'in the worst case scenario at z=0, APT still boasts an R²~0.6 on σ8' for SIMBA, and the corresponding point is visible in Figure 9 (top-left panel). The claim should be qualified, for example by restricting it to z ≥ 0.5 or by explicitly noting the z=0 SIMBA σ8 exception. As written, the abstract overstates the demonstrated robustness of the method.
- [§3.1, Eq. (1), §3.2] The sufficiency of the latent codes for posterior estimation is assumed, not tested. The encoder is trained with a regression loss (Eq. 1) that explicitly targets Ωm and σ8, so the latent codes are aligned with point prediction of these parameters. This is not full circularity because the APT is evaluated on held-out boxes the encoder never saw, but it does mean that the downstream posterior is conditioned on features already selected by the regression objective. The paper does not test whether these codes retain information needed for the tails of the posterior or for the reported calibration, for example by comparing against an unsupervised compression or by using an information-preserving summary. Please either add such a comparison or explicitly discuss this limitation when claiming that the latent codes are 'meaningful representations' in a general sense.
- [§4.6, Figure 9] The redshift robustness experiments are missing key experimental details. The text does not state how many z=0 and z=0.5 boxes are used, whether these are the same simulation boxes as the z=1 set, or how the train/validation/test split is performed for the APT, GP, RF, and LIN models at these redshifts. Since the z-dependent R² values are a central part of the robustness claim, these details are needed for reproducibility and for interpreting the comparison across redshifts.
minor comments (5)
- [Abstract and §4.2] The phrase 'most of the ground truth ... fall within 1σ' is unquantified; please report the actual fraction of test points inside the 68% and 95% intervals rather than relying on visual inspection.
- [§4.3, Eq. (9)] A modified UCE is defined in Eq. (9), but only the UCE of Eq. (6) is reported in Figure 6. Please either report the modified values or clarify why they are omitted.
- [Tables and Figures] R² values are point estimates computed on 150 test examples; no uncertainty intervals are provided. Given that several method comparisons differ by only ~0.01 (Table 2), bootstrap or other interval estimates would help assess whether the differences are meaningful.
- [§5] There are typos in the conclusions, including 'theATP model' and 'esimator'; please proofread the final section and the abstract.
- [General] The paper does not include a data or code availability statement. Since the CAMELS data are public and the SBI package is named, a brief statement (or link) would improve reproducibility.
Circularity Check
No significant circularity: the encoder is supervised, but the APT posterior is evaluated on held-out boxes and benchmarked against independent regressors, so the central claims do not reduce to the training objective by construction.
full rationale
The paper's two-stage pipeline is not circular. The encoder is trained with the regression loss of Eq. (1) to predict Omega_m and sigma_8, so the latent codes are target-supervised; however, the encoder is trained only on the training/validation boxes, and the APT density estimator is then trained on latent codes from those same splits and tested on latent codes of held-out boxes (Section 4.2, Figures 3-4, Table 2). The APT does not see the test labels when producing posteriors, and its R^2 values are empirical test-set outcomes rather than quantities forced by the encoder objective. The paper further benchmarks APT against Gaussian Process, Random Forest, and a linear probe trained on the same latent codes (Table 2), showing that APT outperforms these independent regressors on the shared representation. Self-citations to Andrianomena & Hassan (2023b, 2025) are used for architecture inspiration and qualitative comparison, not as load-bearing proof of the main result. The calibration claim rests on UCE, which is a consistency metric rather than a full coverage test, but that is a statistical validity concern, not a circularity of the derivation chain.
Assumptions & free parameters
free parameters (8)
- Encoder learning rate =
0.0015
- Encoder architecture (channels per stage) =
[16, 32, 64, 128]
- Latent vector length =
1024
- APT/MAF learning rate =
0.0005
- Number of MADE layers in MAF =
4
- UCE number of bins =
10
- Number of posterior samples =
10000
- Data augmentation noise level =
not reported
assumptions (5)
- standard math Bayes' theorem (Eq. 2) and normalizing flow change-of-variable formula (Eq. 4)
- domain assumption CAMELS boxes at z=1 are representative of upcoming 21-cm tomographic surveys
- ad hoc to paper Regression loss (Eq. 1) yields latent codes sufficient for posterior estimation
- domain assumption Astrophysical parameters can be ignored when mapping HI maps to cosmology
- domain assumption Uniform CAMELS prior is the correct proposal
Cite this review
Pith. "Pith review of Probabilistic cosmological inference on HI tomographic data." pith.science (2026). https://pith.science/paper/JS7TJSSH
@misc{pith2026250721682,
author = {Pith},
title = {Pith review of: Probabilistic cosmological inference on HI tomographic data},
year = {2026},
howpublished = {\url{https://pith.science/paper/JS7TJSSH}},
note = {Machine review of arXiv:2507.21682}
}
abstract
We explore the possibility of retrieving cosmological information from 21-cm tomographic data at intermediate redshift. The first step in our approach consists of training an encoder, composed of several three dimensional convolutional layers, to cast the neutral hydrogen 3D data into a lower dimension latent space. Once pre-trained, the featurizer is able to generate 3D grid representations which, in turn, will be mapped onto cosmology ($\Omega_{\rm m}$, $\sigma_{8}$) via likelihood-free inference. For the latter, which is framed as a density estimation problem, we consider a Bayesian approximation method which exploits the capacity of Masked Autoregressive Flow to estimate the posterior. It is found that the representations learned by the deep encoder are separable in latent space. Results show that the neural density estimator, trained on the latent codes, is able to constrain cosmology with a precision of $R^2 \ge 0.91$ on all parameters and that most of the ground truth of the instances in the test set fall within $1\sigma$ uncertainty. It is established that the posterior uncertainty from the density estimator is reasonably calibrated. We also investigate the robustness of the feature extractor by using it to compress out-of-distribution dataset, that is either from a different simulation or from the same simulation but at different redshift. We find that, while trained on the latent codes corresponding to different types of out-of-distribution dataset, the probabilistic model is still reasonably capable of constraining cosmology, with $R^2 \ge 0.80$ in general. This highlights both the predictive power of the density estimator considered in this work and the meaningfulness of the latent codes retrieved by the encoder. We believe that the approach prescribed in this proof of concept will be of great use when analyzing 21-cm data from various surveys in the near future.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
A., et al., 2014, Astronomy & Astrophysics, 571, A1
Ade P. A., et al., 2014, Astronomy & Astrophysics, 571, A1
work page 2014
-
[2]
Aghanim N., et al., 2020, Astronomy & Astrophysics, 641, A1
work page 2020
-
[3]
Akeret J., Refregier A., Amara A., Seehars S., Hasner C., 2015, Journal of Cosmology and Astroparticle Physics, 2015, 043
work page 2015
-
[4]
Alsing J., Charnock T., Feeney S., Wandelt B., 2019, Monthly Notices of the Royal Astronomical Society, 488, 4440
work page 2019
-
[5]
Amiri M., et al., 2022, The Astrophysical Journal Supplement Series, 261, 29
work page 2022
-
[6]
Amiri M., et al., 2023, The Astrophysical Journal, 947, 16
work page 2023
-
[7]
Andrianomena S., Hassan S., 2023a, arXiv:2311.00799
-
[8]
Andrianomena S., Hassan S., 2023b, Journal of Cosmology and Astroparticle Physics, 2023, 051
work page 2023
Show all 63 references
-
[9]
Andrianomena S., Hassan S., 2025, Astrophysics and Space Science, 370, 14
2025
-
[10]
Andrianomena S., Hassan S., Villaescusa-Navarro F., 2025, Astrophysics and Space Science, 370, 1
2025
-
[11]
Ansari R., et al., 2012, Astronomy & Astrophysics, 540, A129
2012
-
[12]
pp 738--757
Bandura K., et al., 2014, in Ground-based and Airborne Telescopes V. pp 738--757
2014
-
[13]
A., Davies R
Battye R. A., Davies R. D., Weller J., 2004, Monthly Notices of the Royal Astronomical Society, 355, 1339
2004
-
[14]
Battye R., Browne I., Dickinson C., Heron G., Maffei B., Pourtsidou A., 2013, Monthly Notices of the Royal Astronomical Society, 434, 1239
2013
-
[15]
K., 2005, Monthly Notices of the Royal Astronomical Society, 358, 968
Bharadwaj S., Pandey S. K., 2005, Monthly Notices of the Royal Astronomical Society, 358, 968
2005
-
[16]
Binnie T., Zhao X., Pritchard J., Mao Y., 2025, arXiv:2502.08152
2025 arXiv
-
[17]
M., 1994, Aston University
Bishop C. M., 1994, Aston University
1994
-
[18]
G., 2010, Journal of the American Statistical Association, 105, 1178
Blum M. G., 2010, Journal of the American Statistical Association, 105, 1178
2010
-
[19]
G., Patel P., Santos M
Bull P., Ferreira P. G., Patel P., Santos M. G., 2015, The Astrophysical Journal, 803, 21
2015
-
[20]
Child R., 2020, arXiv preprint arXiv:2011.10650
2020 arXiv
-
[21]
Courty N., Flamary R., Tuia D., Rakotomamonjy A., 2016, IEEE transactions on pattern analysis and machine intelligence, 39, 1853
2016
-
[22]
Crichton D., et al., 2022, Journal of Astronomical Telescopes, Instruments, and Systems, 8, 011019
2022
-
[23]
H., Appleby S., 2019, Monthly Notices of the Royal Astronomical Society, 486, 2827
Dav \'e R., Angl \'e s-Alc \'a zar D., Narayanan D., Li Q., Rafieferantsoa M. H., Appleby S., 2019, Monthly Notices of the Royal Astronomical Society, 486, 2827
2019
-
[24]
S., et al., 2016, The Astronomical Journal, 151, 44
Dawson K. S., et al., 2016, The Astronomical Journal, 151, 44
2016
-
[25]
Eickenberg M., Exarchakis G., Hirn M., Mallat S., 2017, Advances in Neural Information Processing Systems, 30
2017
-
[26]
Eickenberg M., Exarchakis G., Hirn M., Mallat S., Thiry L., 2018, The Journal of chemical physics, 148
2018
-
[27]
pp 1050--1059
Gal Y., Ghahramani Z., 2016, in international conference on machine learning. pp 1050--1059
2016
-
[28]
pp 881--889
Germain M., Gregor K., Murray I., Larochelle H., 2015, in International conference on machine learning. pp 881--889
2015
-
[29]
Gillet N., Mesinger A., Greig B., Liu A., Ucci G., 2019, Monthly Notices of the Royal Astronomical Society, 484, 282
2019
-
[30]
pp 2404--2414
Greenberg D., Nonnenmacher M., Macke J., 2019, in International conference on machine learning. pp 2404--2414
2019
-
[31]
Greig B., Mesinger A., 2015, Monthly Notices of the Royal Astronomical Society, 449, 4246
2015
-
[32]
A., 2023, Monthly Notices of the Royal Astronomical Society, 519, 5288
Greig B., Ting Y.-S., Kaurov A. A., 2023, Monthly Notices of the Royal Astronomical Society, 519, 5288
2023
-
[33]
Hassan S., Andrianomena S., Doughty C., 2020, Monthly Notices of the Royal Astronomical Society, 494, 5761
2020
-
[34]
K., 1995, in Proceedings of 3rd international conference on document analysis and recognition
Ho T. K., 1995, in Proceedings of 3rd international conference on document analysis and recognition. pp 278--282
1995
-
[35]
Jennings E., Madigan M., 2017, Astronomy and computing, 19, 16
2017
-
[36]
L., 2009, Proceedings of the IEEE, 97, 1522
Jonas J. L., 2009, Proceedings of the IEEE, 97, 1522
2009
-
[37]
F., Kahrs L
Laves M.-H., Ihler S., Fast J. F., Kahrs L. A., Ortmaier T., 2020, in Medical imaging with deep learning. pp 393--412
2020
-
[38]
J., Bassetto G., \"O cal K., Nonnenmacher M., Macke J
Lueckmann J.-M., Goncalves P. J., Bassetto G., \"O cal K., Nonnenmacher M., Macke J. H., 2017, Advances in neural information processing systems, 30
2017
-
[39]
R., Mondal R., Watkinson C
Majumdar S., Pritchard J. R., Mondal R., Watkinson C. A., Bharadwaj S., Mellema G., 2018, Monthly Notices of the Royal Astronomical Society, 476, 4007
2018
-
[40]
pp 410--420
Martini P., et al., 2018, in Ground-based and Airborne Instrumentation for Astronomy VII. pp 410--420
2018
-
[41]
McInnes L., Healy J., Melville J., 2018, arXiv:1802.03426
2018 arXiv
-
[42]
Mellema G., et al., 2013, Experimental Astronomy, 36, 235
2013
-
[43]
K., Mesinger A., Majumdar S., 2015, arXiv preprint arXiv:1501.04203
Mellema G., Koopmans L., Shukla H., Datta K. K., Mesinger A., Majumdar S., 2015, arXiv preprint arXiv:1501.04203
2015 arXiv
-
[44]
Nelson D., et al., 2019, Computational Astrophysics and Cosmology, 6, 1
2019
-
[45]
P., et al., 2024, Monthly Notices of the Royal Astronomical Society, 528, 2078
Novaes C. P., et al., 2024, Monthly Notices of the Royal Astronomical Society, 528, 2078
2024
-
[46]
Papamakarios G., Murray I., 2016, Advances in neural information processing systems, 29
2016
-
[47]
Papamakarios G., Pavlakou T., Murray I., 2017, Advances in neural information processing systems, 30
2017
-
[48]
pp 837--848
Papamakarios G., Sterratt D., Murray I., 2019, in The 22nd international conference on artificial intelligence and statistics. pp 837--848
2019
-
[49]
Pourtsidou A., Bacon D., Crittenden R., 2017, Monthly Notices of the Royal Astronomical Society, 470, 4251
2017
-
[50]
Adaptive Computation and Machine Learning, MIT Press, Cambridge, MA, USA
Rasmussen C., Williams C., 2006, Gaussian Processes for Machine Learning. Adaptive Computation and Machine Learning, MIT Press, Cambridge, MA, USA
2006
-
[51]
pp 1530--1538
Rezende D., Mohamed S., 2015, in International conference on machine learning. pp 1530--1538
2015
-
[52]
F., Debono I., 2017, Astronomy & Astrophysics, 597, A136
Smoot G. F., Debono I., 2017, Astronomy & Astrophysics, 597, A136
2017
-
[53]
Switzer E., et al., 2013, Monthly Notices of the Royal Astronomical Society: Letters, 434, L46
2013
-
[54]
J., Greenberg D
Tejero-Cantero A., Boelts J., Deistler M., Lueckmann J.-M., Durkan C., Gon c alves P. J., Greenberg D. S., Macke J. H., 2020a, arXiv:2007.09114
2007 arXiv
-
[55]
J., Greenberg D
Tejero-Cantero A., Boelts J., Deistler M., Lueckmann J.-M., Durkan C., Gonçalves P. J., Greenberg D. S., Macke J. H., 2020b, @doi [Journal of Open Source Software] 10.21105/joss.02505 , 5, 2505
-
[56]
pp 7167--7176
Tzeng E., Hoffman J., Saenko K., Darrell T., 2017, in Proceedings of the IEEE conference on computer vision and pattern recognition. pp 7167--7176
2017
-
[57]
Villaescusa-Navarro F., et al., 2021a, arXiv:2109.09747
-
[58]
Villaescusa-Navarro F., et al., 2021b, The Astrophysical Journal, 915, 71
-
[59]
Villaescusa-Navarro F., et al., 2022a, arXiv:2201.01300
-
[60]
Villaescusa-Navarro F., et al., 2022b, The Astrophysical Journal Supplement Series, 259, 61
-
[61]
Wagg J., Bourke T., Green J., Braun R., et al., 2014, SKA1 scientific use cases
2014
-
[62]
D., 2022, The Astrophysical Journal, 926, 151
Zhao X., Mao Y., Cheng C., Wandelt B. D., 2022, The Astrophysical Journal, 926, 151
2022
-
[63]
D., 2024, The Astrophysical Journal, 973, 41
Zhao X., Mao Y., Zuo S., Wandelt B. D., 2024, The Astrophysical Journal, 973, 41
2024
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.